Date field by Atomic Update

[Noriyuki TAKEI]

Hi,All.

I update some fields by Solj Atomic Update.But in 
particular case, an error occurred.

When I try to set  the value "2017-01-01" to date filed
by Solrj Atomic Update,the following error message appears.

org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from
server at http://XXX.XXX.XXX.XXX:/solr/test_shard1_replica2: Invalid
Date String:'2017-01-01'
at
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:765)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1173)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1062)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1004)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)


In order to solve this problem,I applied Atomic Update to
only date field,and applied not Atomic Update to
all the other fields.

The code is as follows.

SolrInputDocument doc = new SolrInputDocument();

doc.addField("title", new HashMap().put("set","title_test")); // Atomic
Update
doc.addField("body", new HashMap().put("set","body_test")); // Atomic
Update
doc.addField("pub_date", "2017-01-01"); // not Atomic Update to date
field

solr.add(doc);
solr.commit();

In short,Atomic Update and not Atomic Update are mixed.

The code shown above seems to work properly.I think this solution is
good.But would you like to tell me the any other problems?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Date-field-by-Atomic-Update-tp4335226.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: does suggester's contextField support TrieDate data type?

[Erick Erickson]

WARNING: I've never really tried this, but I don't see why it wouldn't work.

What does the full query look like?
suggest.q=*:*=[2017-05-15T0:0:0.0Z TO *] maybe? And I'm
assuming that the contextField here is a tdate type and you're using
one of the Document*Dictionary implementations for the dictionary.

Need to see the relevant bits of your solrconfig.xml and schema file
as well as the actual query you send to the suggester.

And you've also built the suggester, correct? Can you get suggestions
from it if you don't specify cfq?

Best,
Erick

On Mon, May 15, 2017 at 7:37 PM, arik  wrote:
> I'd like to use a date (TrieDate) for the contextField in my SuggestComponent
> with an AnalyzingInfixLookupFactory.  Basically am trying to narrow my
> suggestions by a relevant date range, something like
> suggest.cfq=[2017-05-15T0:0:0.0Z TO *]
>
> Doesn't seem to work, so before trying further I wondered if contextField
> even supports dates?  All the examples seem to use strings.
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/does-suggester-s-contextField-support-TrieDate-data-type-tp4335208.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr /export handler - behavior during close()

[Susmit Shukla]

Hi Joel,

Regarding the implementation, I am wrapping the topmost TupleStream in a
ParallelStream and execute it on the worker cluster (one of the joined
cluster doubles up as worker cluster). ParallelStream does submit the query
to /stream handler.
for #2, for e.g. I am creating 2 CloudSolrStreams , wrapping them in
IntersectStream and wrapping that in ParallelStream and reading out the
tuples from parallel stream. close() is called on parallelStream. I do have
custom streams but that is similar to intersectStream.
I am on solr 6.3.1
The 2 solr clusters serving the join queries are having many shards. Worker
collection is also multi sharded and is one from the main clusters, so do
you imply I should be using a single sharded "worker" collection? Would the
joins execute faster?
On a side note, increasing the workers beyond 1 was not improving the
execution times but was degrading if number was 3 and above. That is
counter intuitive since the joins are huge and putting more workers should
have improved the performance.

Thanks,
Susmit


On Mon, May 15, 2017 at 6:47 AM, Joel Bernstein  wrote:

> Ok please do report any issues you run into. This is quite a good bug
> report.
>
> I reviewed the code and I believe I see the problem. The problem seems to
> be that output code from the /stream handler is not properly accounting for
> client disconnects and closing the underlying stream. What I see in the
> code is that exceptions coming from read() in the stream do automatically
> close the underlying stream. But exceptions from the writing of the stream
> do not close the stream. This needs to be fixed.
>
> A few questions about your streaming implementation:
>
> 1) Are you sending requests to the /stream handler? Or are you embedding
> CloudSolrStream in your application and bypassing the /stream handler?
>
> 2) If you're sending Streaming Expressions to the stream handler are you
> using SolrStream or CloudSolrStream to send the expression?
>
> 3) What version of Solr are you using.
>
> 4) Have you implemented any custom streams?
>
>
> #2 is an important question. If you're sending expressions to the /stream
> handler using CloudSolrStream the collection running the expression would
> have to be setup a specific way. The collection running the expression will
> have to be a* single shard collection*. You can have as many replicas as
> you want but only one shard. That's because CloudSolrStream picks one
> replica in each shard to forward the request to then merges the results
> from the shards. So if you send in an expression using CloudSolrStream that
> expression will be sent to each shard to be run and each shard will be
> duplicating the work and return duplicate results.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Sat, May 13, 2017 at 7:03 PM, Susmit Shukla 
> wrote:
>
> > Thanks Joel
> > Streaming is awesome, just had a huge implementation in my project. I
> found
> > out a couple more issues with streaming and did local hacks for them,
> would
> > raise them too.
> >
> > On Sat, May 13, 2017 at 2:09 PM, Joel Bernstein 
> > wrote:
> >
> > > Ah, then this is unexpected behavior. Can you open a ticket for this?
> > >
> > > Joel Bernstein
> > > http://joelsolr.blogspot.com/
> > >
> > > On Sat, May 13, 2017 at 2:51 PM, Susmit Shukla <
> shukla.sus...@gmail.com>
> > > wrote:
> > >
> > > > Hi Joel,
> > > >
> > > > I was using CloudSolrStream for the above test. Below is the call
> > stack.
> > > >
> > > > at
> > > > org.apache.http.impl.io.ChunkedInputStream.read(
> > > > ChunkedInputStream.java:215)
> > > > at
> > > > org.apache.http.impl.io.ChunkedInputStream.close(
> > > > ChunkedInputStream.java:316)
> > > > at
> > > > org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(
> > > > ResponseEntityProxy.java:128)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.checkClose(
> > > > EofSensorInputStream.java:228)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.close(
> > > > EofSensorInputStream.java:174)
> > > > at sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:378)
> > > > at sun.nio.cs.StreamDecoder.close(StreamDecoder.java:193)
> > > > at java.io.InputStreamReader.close(InputStreamReader.java:199)
> > > > at
> > > > org.apache.solr.client.solrj.io.stream.JSONTupleStream.
> > > > close(JSONTupleStream.java:91)
> > > > at
> > > > org.apache.solr.client.solrj.io.stream.SolrStream.close(
> > > > SolrStream.java:186)
> > > >
> > > > Thanks,
> > > > Susmit
> > > >
> > > > On Sat, May 13, 2017 at 10:48 AM, Joel Bernstein  >
> > > > wrote:
> > > >
> > > > > I was just reading the Java docs on the ChunkedInputStream.
> > > > >
> > > > > "Note that this class NEVER closes the underlying stream"
> > > > >
> > > > > In that scenario the /export would indeed continue to send data. I
> > > think
> > > > we
> > > > > can consider this an 

does suggester's contextField support TrieDate data type?

[arik]

I'd like to use a date (TrieDate) for the contextField in my SuggestComponent
with an AnalyzingInfixLookupFactory.  Basically am trying to narrow my
suggestions by a relevant date range, something like
suggest.cfq=[2017-05-15T0:0:0.0Z TO *]

Doesn't seem to work, so before trying further I wondered if contextField
even supports dates?  All the examples seem to use strings.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/does-suggester-s-contextField-support-TrieDate-data-type-tp4335208.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrJ - How to add a blocked document without child documents

[Jeffery Yuan]

Hi, Damien Kamerman

  Thanks for your reply. The problem is when we add a parent documents which
doesn't contain child info yet. 
  Later we will add same parent documents with child documents.

  But this would cause 2 parent documents with same id in the solr index.

  I workaround this issue by always deleting first, but I am wondering
whether there is better approach.

Thanks
Jeffery Yuan




--
View this message in context: 
http://lucene.472066.n3.nabble.com/SolrJ-How-to-add-a-blocked-document-without-child-documents-tp4335006p4335195.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrJ - How to add a blocked document without child documents

[Damien Kamerman]

Does this fl help?

fl=*,[child childFilter="docType:child" parentFilter=docType:parent]

On 14 May 2017 at 16:16, Jeffery Yuan  wrote:

> Nested documents is quite useful to model structural hierarchy data.
>
> Sometimes, we only have parent document which doesn't have child documents
> yet, we want to add it first, and then later update it: re-add the whole
> document including the parent documents and its all child documents.
>
> But we found out that in the server, there would be two parent documents
> with same id: one without child document, the other one which contains
> child
> documents.
>
> http://localhost:8983/solr/thecollection_shard1_replica2/
> select?q=id:*=*,[docid]=false
> 
>   
> parent
> 9816c0f3-f3ae-4a7c-a5fe-89a2c481467a
> 0
>   
>   
> child
> e27d2709-2dc0-439d-b017-4d95212bf05f
> 
>   9816c0f3-f3ae-4a7c-a5fe-89a2c481467a
> 
> 1
>   
>   
> parent
> 9816c0f3-f3ae-4a7c-a5fe-89a2c481467a
> 
>   9816c0f3-f3ae-4a7c-a5fe-89a2c481467a
> 
> 2
>   
> 
>
> How I can avoid the duplicate parent documents?
> How could I add a blocked document without child documents?
>
> - I can workaround this by delete first before add new documents but the
> performance would suffer
>
> Thanks a lot for your help and response.
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/SolrJ-How-to-add-a-blocked-document-without-
> child-documents-tp4335006.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Terms not being indexed; not sure why

[Erick Erickson]

Most likely you're searching against your default field, often "text".
A frequent problem is that you enter a search like

q=content:University of Wisconsin

and the search is actually

q=content:university text:of text:wisconsin

Try your debug=query with the original maybe?

In fact, somehow you're getting lucky, I'm not sure you you're even
getting a hit when you search by title since the parsed query is:

"parsedquery": "_text_:21610003",
"parsedquery_toString": "_text_:21610003",

i.e you're searching against your _text_ field not your filename field.

So my guess is that you're throwing everything in a _text_ field and
always searching against that. Since it's such a "bag of words", it's
just happening to score your query below the top 10.

You'll also want to be boosting the title field, perhaps use edismax.

Best,
Erick

On Mon, May 15, 2017 at 1:17 PM, Susheel Kumar  wrote:
> Can you upload your schema to some site like dropbox etc. to look and send
> the query which you are using and returning no results?
>
> Thanks,
> Susheel
>
> On Mon, May 15, 2017 at 1:46 PM, Chip Calhoun  wrote:
>
>> I'm creating a new Solr core to crawl a local site. We have a page on
>> "University of Wisconsin--Madison", but a search for that name in any form
>> won't appear within the first 10 results. the page is indexed, and I can
>> search for it by filename. Termfreq(title) shows 0s for search terms which
>> are very clearly in the title. What would cause this?
>>
>> In case it's useful, I'm pasting my results for a search on the filename,
>> with termfreq arguments for the terms I'd actually like to search on.
>>
>> {
>>   "responseHeader": {
>> "status": 0,
>> "QTime": 33,
>> "params": {
>>   "debugQuery": "true",
>>   "fl": "content, id, title, url, termfreq(content,\"university\"),
>> termfreq(content,\"wisconsin\"), termfreq(content,\"university of
>> wisconsin\"), termfreq(content,\"university of wisconsin--madison\"),
>> termfreq(title,\"university\"), termfreq(title,\"wisconsin\"),
>> termfreq(title,\"university of wisconsin\"), termfreq(title,\"university of
>> wisconsin--madison\"), score",
>>   "indent": "true",
>>   "q": "21610003",
>>   "_": "1494864119360",
>>   "wt": "json"
>> }
>>   },
>>   "response": {
>> "numFound": 1,
>> "start": 0,
>> "maxScore": 0.26968884,
>> "docs": [
>>   {
>> "content": [
>>   "University of Wisconsin--Madison. Department of Physics |
>> Scientific Biographies Menu ☰ Back to Top Home History Programs Niels Bohr
>> Library & Archive Physics History Network Institutions Physics History
>> Network Over 850 biographies of physicists and histories of institutions
>> with information pertaining to their lives, careers, and research. Tip:
>> Search within this page by using Ctrl + F or ⌘ + F Search Our Catalogs
>> Archives Books Collections Emilio Segré Visual Archives Digital Collections
>> Oral Histories Archival Finding Aids Physics History Network Preservation
>> and Support Donating Materials Saving Archival Collections Grants to
>> Archives Documentation Projects History Newsletters Center for History of
>> Physics Scholarship & Outreach Main Navigation Home About Topic Guides
>> Feedback Table of Contents Institutional History Abstract Department chairs
>> Important Dates Places Subjects Citations Relationships People Employees &
>> Officers PhD Students Associates & Members Institutions Institutional
>> Hierarchy Associates Resources Archival as Author Archival as Subject
>> Published as Author University of Wisconsin--Madison. Department of Physics
>> Dates 1868 – present Authorized Form of Name University of
>> Wisconsin--Madison. Department of Physics Additional Forms of Names
>> University of Wisconsin--Madison. Dept. of Physics Institutional History
>> Abstract The University of Wisconsin's Department of Physics began in 1868
>> and now has research groups in areas such as astrophysics; atomic,
>> molecular & optical physics; biophysics; condensed matter physics;
>> cosmology; high energy physics; nuclear physics; and plasma physics.
>> Department chairs 1868-93, John E. Davies. 1893-1925, Benjamin Snow.
>> 1925-35, Charles Mendenhall. 1936-37, John Ransom Roebuck. 1938-47, Leonard
>> R. Ingersoll. 1947-51, Ragnar Oswald Rollefson. 1951-52, Heinz H.
>> Barschall. 1952-56, Ragnar Oswald Rollefson. 1956-57, Heinz H. Barschall.
>> 1957-60, Ragnar Oswald Rollefson. 1960-63, Hugh Richards. 1963-64, Heinz H.
>> Barschall. 1964-66, William D. Walker. 1966-69, Hugh Richards. 1969-71,
>> Loyal \"Randy\" Durand. 1971-73, Maurice Barney Webb. 1973-75, Marv Ebel.
>> 1975-77, William L. Kraushaar. 1977-79, Keith Symon. 1979-81, Dave Huber.
>> 1981-84, Don D. Reeder. 1984-85, Dave Huber. 1985-88, Hugh Richards.
>> 1988-91, Martin Olsson. 1991-94, Ludwig Bruch. 1994-97, James Lawler.
>> 1997-2000, Lee G. Pondrom. 2000-05, Don D. Reeder. 2005-08, Susan
>> 

Re: Terms not being indexed; not sure why

[Susheel Kumar]

Can you upload your schema to some site like dropbox etc. to look and send
the query which you are using and returning no results?

Thanks,
Susheel

On Mon, May 15, 2017 at 1:46 PM, Chip Calhoun  wrote:

> I'm creating a new Solr core to crawl a local site. We have a page on
> "University of Wisconsin--Madison", but a search for that name in any form
> won't appear within the first 10 results. the page is indexed, and I can
> search for it by filename. Termfreq(title) shows 0s for search terms which
> are very clearly in the title. What would cause this?
>
> In case it's useful, I'm pasting my results for a search on the filename,
> with termfreq arguments for the terms I'd actually like to search on.
>
> {
>   "responseHeader": {
> "status": 0,
> "QTime": 33,
> "params": {
>   "debugQuery": "true",
>   "fl": "content, id, title, url, termfreq(content,\"university\"),
> termfreq(content,\"wisconsin\"), termfreq(content,\"university of
> wisconsin\"), termfreq(content,\"university of wisconsin--madison\"),
> termfreq(title,\"university\"), termfreq(title,\"wisconsin\"),
> termfreq(title,\"university of wisconsin\"), termfreq(title,\"university of
> wisconsin--madison\"), score",
>   "indent": "true",
>   "q": "21610003",
>   "_": "1494864119360",
>   "wt": "json"
> }
>   },
>   "response": {
> "numFound": 1,
> "start": 0,
> "maxScore": 0.26968884,
> "docs": [
>   {
> "content": [
>   "University of Wisconsin--Madison. Department of Physics |
> Scientific Biographies Menu ☰ Back to Top Home History Programs Niels Bohr
> Library & Archive Physics History Network Institutions Physics History
> Network Over 850 biographies of physicists and histories of institutions
> with information pertaining to their lives, careers, and research. Tip:
> Search within this page by using Ctrl + F or ⌘ + F Search Our Catalogs
> Archives Books Collections Emilio Segré Visual Archives Digital Collections
> Oral Histories Archival Finding Aids Physics History Network Preservation
> and Support Donating Materials Saving Archival Collections Grants to
> Archives Documentation Projects History Newsletters Center for History of
> Physics Scholarship & Outreach Main Navigation Home About Topic Guides
> Feedback Table of Contents Institutional History Abstract Department chairs
> Important Dates Places Subjects Citations Relationships People Employees &
> Officers PhD Students Associates & Members Institutions Institutional
> Hierarchy Associates Resources Archival as Author Archival as Subject
> Published as Author University of Wisconsin--Madison. Department of Physics
> Dates 1868 – present Authorized Form of Name University of
> Wisconsin--Madison. Department of Physics Additional Forms of Names
> University of Wisconsin--Madison. Dept. of Physics Institutional History
> Abstract The University of Wisconsin's Department of Physics began in 1868
> and now has research groups in areas such as astrophysics; atomic,
> molecular & optical physics; biophysics; condensed matter physics;
> cosmology; high energy physics; nuclear physics; and plasma physics.
> Department chairs 1868-93, John E. Davies. 1893-1925, Benjamin Snow.
> 1925-35, Charles Mendenhall. 1936-37, John Ransom Roebuck. 1938-47, Leonard
> R. Ingersoll. 1947-51, Ragnar Oswald Rollefson. 1951-52, Heinz H.
> Barschall. 1952-56, Ragnar Oswald Rollefson. 1956-57, Heinz H. Barschall.
> 1957-60, Ragnar Oswald Rollefson. 1960-63, Hugh Richards. 1963-64, Heinz H.
> Barschall. 1964-66, William D. Walker. 1966-69, Hugh Richards. 1969-71,
> Loyal \"Randy\" Durand. 1971-73, Maurice Barney Webb. 1973-75, Marv Ebel.
> 1975-77, William L. Kraushaar. 1977-79, Keith Symon. 1979-81, Dave Huber.
> 1981-84, Don D. Reeder. 1984-85, Dave Huber. 1985-88, Hugh Richards.
> 1988-91, Martin Olsson. 1991-94, Ludwig Bruch. 1994-97, James Lawler.
> 1997-2000, Lee G. Pondrom. 2000-05, Don D. Reeder. 2005-08, Susan
> Coppersmith. 2008-11, Akif Baha Balentekin. 2011-14, Robert Joynt.
> 2014-present, Albrecht Karle. Important Dates 1868 Department of Physics
> was founded, Madison (Wis.). 1897 – 1901 Robert Wood was on faculty. 1928 –
> 1934 John Van Vleck was on faculty. 1930 – 1937 Leland Haworth was on
> faculty. 1934 – 1947 Gregory Breit was on faculty. 1935 – 1936 Eugene
> Feenberg was on faculty. 1935 – 1972 Raymond Herb was on faculty. 1937 –
> 1938 Eugene Wigner was on faculty. 1941 – 1942 Léon Brillouin was on
> faculty. 1946 – 1997 Henry \"Heinz\" Barschall was on faculty. 1947 – 1964
> Robert Sachs was on faculty. 1950 – 1953 Joaquin \"Quin\" Luttinger was on
> faculty. 1951 – 1953 Robert Adair was on faculty. 1952 – 1954 Willy
> Haeberli was on faculty. 1954 – 1957 Kenneth Watson was on faculty. 1956 –
> present Willy Haeberli has been on faculty. 1961 – present Maurice Webb has
> been on faculty. 1962 – 1993 Donald Kerst was on faculty. 1965 – 2008
> William Kraushaar was on faculty. 1970 – present 

Terms not being indexed; not sure why

[Chip Calhoun]

I'm creating a new Solr core to crawl a local site. We have a page on 
"University of Wisconsin--Madison", but a search for that name in any form 
won't appear within the first 10 results. the page is indexed, and I can search 
for it by filename. Termfreq(title) shows 0s for search terms which are very 
clearly in the title. What would cause this?

In case it's useful, I'm pasting my results for a search on the filename, with 
termfreq arguments for the terms I'd actually like to search on.

{
  "responseHeader": {
"status": 0,
"QTime": 33,
"params": {
  "debugQuery": "true",
  "fl": "content, id, title, url, termfreq(content,\"university\"), 
termfreq(content,\"wisconsin\"), termfreq(content,\"university of wisconsin\"), 
termfreq(content,\"university of wisconsin--madison\"), 
termfreq(title,\"university\"), termfreq(title,\"wisconsin\"), 
termfreq(title,\"university of wisconsin\"), termfreq(title,\"university of 
wisconsin--madison\"), score",
  "indent": "true",
  "q": "21610003",
  "_": "1494864119360",
  "wt": "json"
}
  },
  "response": {
"numFound": 1,
"start": 0,
"maxScore": 0.26968884,
"docs": [
  {
"content": [
  "University of Wisconsin--Madison. Department of Physics | Scientific 
Biographies Menu ☰ Back to Top Home History Programs Niels Bohr Library & 
Archive Physics History Network Institutions Physics History Network Over 850 
biographies of physicists and histories of institutions with information 
pertaining to their lives, careers, and research. Tip: Search within this page 
by using Ctrl + F or ⌘ + F Search Our Catalogs Archives Books Collections 
Emilio Segré Visual Archives Digital Collections Oral Histories Archival 
Finding Aids Physics History Network Preservation and Support Donating 
Materials Saving Archival Collections Grants to Archives Documentation Projects 
History Newsletters Center for History of Physics Scholarship & Outreach Main 
Navigation Home About Topic Guides Feedback Table of Contents Institutional 
History Abstract Department chairs Important Dates Places Subjects Citations 
Relationships People Employees & Officers PhD Students Associates & Members 
Institutions Institutional Hierarchy Associates Resources Archival as Author 
Archival as Subject Published as Author University of Wisconsin--Madison. 
Department of Physics Dates 1868 – present Authorized Form of Name University 
of Wisconsin--Madison. Department of Physics Additional Forms of Names 
University of Wisconsin--Madison. Dept. of Physics Institutional History 
Abstract The University of Wisconsin's Department of Physics began in 1868 and 
now has research groups in areas such as astrophysics; atomic, molecular & 
optical physics; biophysics; condensed matter physics; cosmology; high energy 
physics; nuclear physics; and plasma physics. Department chairs 1868-93, John 
E. Davies. 1893-1925, Benjamin Snow. 1925-35, Charles Mendenhall. 1936-37, John 
Ransom Roebuck. 1938-47, Leonard R. Ingersoll. 1947-51, Ragnar Oswald 
Rollefson. 1951-52, Heinz H. Barschall. 1952-56, Ragnar Oswald Rollefson. 
1956-57, Heinz H. Barschall. 1957-60, Ragnar Oswald Rollefson. 1960-63, Hugh 
Richards. 1963-64, Heinz H. Barschall. 1964-66, William D. Walker. 1966-69, 
Hugh Richards. 1969-71, Loyal \"Randy\" Durand. 1971-73, Maurice Barney Webb. 
1973-75, Marv Ebel. 1975-77, William L. Kraushaar. 1977-79, Keith Symon. 
1979-81, Dave Huber. 1981-84, Don D. Reeder. 1984-85, Dave Huber. 1985-88, Hugh 
Richards. 1988-91, Martin Olsson. 1991-94, Ludwig Bruch. 1994-97, James Lawler. 
1997-2000, Lee G. Pondrom. 2000-05, Don D. Reeder. 2005-08, Susan Coppersmith. 
2008-11, Akif Baha Balentekin. 2011-14, Robert Joynt. 2014-present, Albrecht 
Karle. Important Dates 1868 Department of Physics was founded, Madison (Wis.). 
1897 – 1901 Robert Wood was on faculty. 1928 – 1934 John Van Vleck was on 
faculty. 1930 – 1937 Leland Haworth was on faculty. 1934 – 1947 Gregory Breit 
was on faculty. 1935 – 1936 Eugene Feenberg was on faculty. 1935 – 1972 Raymond 
Herb was on faculty. 1937 – 1938 Eugene Wigner was on faculty. 1941 – 1942 Léon 
Brillouin was on faculty. 1946 – 1997 Henry \"Heinz\" Barschall was on faculty. 
1947 – 1964 Robert Sachs was on faculty. 1950 – 1953 Joaquin \"Quin\" Luttinger 
was on faculty. 1951 – 1953 Robert Adair was on faculty. 1952 – 1954 Willy 
Haeberli was on faculty. 1954 – 1957 Kenneth Watson was on faculty. 1956 – 
present Willy Haeberli has been on faculty. 1961 – present Maurice Webb has 
been on faculty. 1962 – 1993 Donald Kerst was on faculty. 1965 – 2008 William 
Kraushaar was on faculty. 1970 – present Max Lagally has been on faculty. 1972 
– 1978 Samuel Aronson was on faculty. 1981 – present Noah Hershkowitz has been 
on faculty. 1995 – present Franz Himpsel has been on faculty. 2001 – present 
Susan Coppersmith has been on faculty. Places Headquarters Madison (Wis.) 
Subjects Astrophysics. Atomic and molecular physics. 

Re: Configure query parser to handle field name case-insensitive

[Rick Leir]

Björn
Yes, at query time you could downcase the names. Not in Solr, but in the 
front-end web app you have in front of Solr. It needs to be a bit smart, so it 
can downcase the field names but not the query terms.

I assume you do not expose Solr directly to the web.

This downcasing might be easier to do in Javascript in the browser. 
Particularly if the user never has to enter a field name.

Another solution, this time inside Solr, is to provide copyfields for ID, Id, 
and maybe iD. And for other fields that you mention in queries. This will 
consume some memory, particularly for saved fields, so I hesitate to even 
suggest it. Cheers - Rick


On May 15, 2017 9:16:59 AM EDT, "Peemöller, Björn" 
 wrote:
>Hi Rick,
>
>thank you for your reply! I really meant field *names*, since our
>values are already processed by a lower case filter (both index and
>query). However, our users are confused because they can search for
>"id:1" but not for "ID:1". Furthermore, we employ the EDisMax query
>parser, so then even get no error message.
>
>Therefore, I thought it may be sufficient to map all field names to
>lower case at the query level so that I do not have to introduce
>additional fields.
>
>Regards,
>Björn
>
>-Ursprüngliche Nachricht-
>Von: Rick Leir [mailto:rl...@leirtech.com] 
>Gesendet: Montag, 15. Mai 2017 13:48
>An: solr-user@lucene.apache.org
>Betreff: Re: Configure query parser to handle field name
>case-insensitive
>
>Björn
>Field names or values? I assume values. Your analysis chain in
>schema.xml probably downcases chars, if not then that could be your
>problem.
>
>Field _name_? Then you might have to copyfield the field to a new field
>with the desired case. Avoid doing that if you can. Cheers -- Rick
>
>On May 15, 2017 5:48:09 AM EDT, "Peemöller, Björn"
> wrote:
>>Hi all,
>>
>>I'm fairly new at using Solr and I need to configure our instance to 
>>accept field names in both uppercase and lowercase (they are defined
>as 
>>lowercase in our configuration). Is there a simple way to achieve
>this?
>>
>>Thanks in advance,
>>Björn
>>
>>Björn Peemöller
>>IT & IT Operations
>>
>>BERENBERG
>>Joh. Berenberg, Gossler & Co. KG
>>Neuer Jungfernstieg 20
>>20354 Hamburg
>>
>>Telefon +49 40 350 60-8548
>>Telefax +49 40 350 60-900
>>E-Mail
>>bjoern.peemoel...@berenberg.de
>>www.berenberg.de
>>
>>Sitz: Hamburg - Amtsgericht Hamburg HRA 42659
>>
>>
>>Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist 
>>vertraulich und kann dem Bank- und Datengeheimnis unterliegen oder 
>>sonst rechtlich geschuetzte Daten und Informationen enthalten. Wenn
>Sie 
>>nicht der richtige Adressat sind oder diese Nachricht irrtuemlich 
>>erhalten haben, informieren Sie bitte sofort den Absender über die 
>>Antwortfunktion. Anschliessend moechten Sie bitte diese Nachricht 
>>einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig 
>>loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht 
>>und/oder der ihr etwa beigefuegten Anhaenge sowie die unbefugte 
>>Weitergabe der darin enthaltenen Daten und Informationen sind nicht 
>>gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen
>
>>namens unseres Hauses grundsaetzlich der Unterschriften zweier 
>>ausreichend bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir 
>>verschicken daher keine rechtsverbindlichen Erklaerungen per E-Mail an
>
>>Dritte. Demgemaess nehmen wir per E-Mail auch keine
>rechtsverbindlichen 
>>Erklaerungen oder Auftraege von Dritten entgegen.
>>Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden 
>>Sie sich bitte an den Absender oder an i...@berenberg.de. Please refer
>
>>to http://www.berenberg.de/my_berenberg/disclaimer_e.html for our 
>>confidentiality notice.
>
>--
>Sorry for being brief. Alternate email is rickleir at yahoo dot com 
>
>Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist
>vertraulich und kann dem Bank- und Datengeheimnis unterliegen oder
>sonst rechtlich geschuetzte Daten und Informationen enthalten. Wenn Sie
>nicht der richtige Adressat sind oder diese Nachricht irrtuemlich
>erhalten haben, informieren Sie bitte sofort den Absender über die
>Antwortfunktion. Anschliessend moechten Sie bitte diese Nachricht
>einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig
>loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht
>und/oder der ihr etwa beigefuegten Anhaenge sowie die unbefugte
>Weitergabe der darin enthaltenen Daten und Informationen sind nicht
>gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen
>namens unseres Hauses grundsaetzlich der Unterschriften zweier
>ausreichend bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir
>verschicken daher keine rechtsverbindlichen Erklaerungen per E-Mail an
>Dritte. Demgemaess nehmen wir per E-Mail auch keine rechtsverbindlichen
>Erklaerungen 

Backing up indexes to an HDFS filesystem

[S G]

Hi,

I have a few doubts regarding the documentation at https://cwiki.apache.org/
confluence/display/solr/Making+and+Restoring+Backups for backing up the
indexes to a HDFS filesystem

1) How frequently are the indexes backed up?
2) Is there a possibility of data-loss if Solr crashes between two backups?
3) Is it production ready?
4) What is the performance impact of backup?
5) How quick are the restores? (i.e some benchmarking of time vs index size)


Thanks
SG

Re: Configure query parser to handle field name case-insensitive

[Erick Erickson]

So do you have _users_ directly entering Solr queries? And are the
totally trusted to be
1> not malicious
2> already know your schema?

Because direct access to the Solr URL allows me to delete all your
data. Usually there are drop-downs or other UI "stuff" that allows you
to programmatically assign the field name.

Trying to get in there and parse an arbitrary query in component is
do-able but difficult.

As Geraint says, field aliasing will work, but you'd need to cover all
the possibilities. All uppercase to lowercase is easy, but camel case
etc. would lead to a lot of aliases.

Best,
Erick

2017-05-15 6:16 GMT-07:00 Peemöller, Björn :
> Hi Rick,
>
> thank you for your reply! I really meant field *names*, since our values are 
> already processed by a lower case filter (both index and query). However, our 
> users are confused because they can search for "id:1" but not for "ID:1". 
> Furthermore, we employ the EDisMax query parser, so then even get no error 
> message.
>
> Therefore, I thought it may be sufficient to map all field names to lower 
> case at the query level so that I do not have to introduce additional fields.
>
> Regards,
> Björn
>
> -Ursprüngliche Nachricht-
> Von: Rick Leir [mailto:rl...@leirtech.com]
> Gesendet: Montag, 15. Mai 2017 13:48
> An: solr-user@lucene.apache.org
> Betreff: Re: Configure query parser to handle field name case-insensitive
>
> Björn
> Field names or values? I assume values. Your analysis chain in schema.xml 
> probably downcases chars, if not then that could be your problem.
>
> Field _name_? Then you might have to copyfield the field to a new field with 
> the desired case. Avoid doing that if you can. Cheers -- Rick
>
> On May 15, 2017 5:48:09 AM EDT, "Peemöller, Björn" 
>  wrote:
>>Hi all,
>>
>>I'm fairly new at using Solr and I need to configure our instance to
>>accept field names in both uppercase and lowercase (they are defined as
>>lowercase in our configuration). Is there a simple way to achieve this?
>>
>>Thanks in advance,
>>Björn
>>
>>Björn Peemöller
>>IT & IT Operations
>>
>>BERENBERG
>>Joh. Berenberg, Gossler & Co. KG
>>Neuer Jungfernstieg 20
>>20354 Hamburg
>>
>>Telefon +49 40 350 60-8548
>>Telefax +49 40 350 60-900
>>E-Mail
>>bjoern.peemoel...@berenberg.de
>>www.berenberg.de
>>
>>Sitz: Hamburg - Amtsgericht Hamburg HRA 42659
>>
>>
>>Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist
>>vertraulich und kann dem Bank- und Datengeheimnis unterliegen oder
>>sonst rechtlich geschuetzte Daten und Informationen enthalten. Wenn Sie
>>nicht der richtige Adressat sind oder diese Nachricht irrtuemlich
>>erhalten haben, informieren Sie bitte sofort den Absender über die
>>Antwortfunktion. Anschliessend moechten Sie bitte diese Nachricht
>>einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig
>>loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht
>>und/oder der ihr etwa beigefuegten Anhaenge sowie die unbefugte
>>Weitergabe der darin enthaltenen Daten und Informationen sind nicht
>>gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen
>>namens unseres Hauses grundsaetzlich der Unterschriften zweier
>>ausreichend bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir
>>verschicken daher keine rechtsverbindlichen Erklaerungen per E-Mail an
>>Dritte. Demgemaess nehmen wir per E-Mail auch keine rechtsverbindlichen
>>Erklaerungen oder Auftraege von Dritten entgegen.
>>Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden
>>Sie sich bitte an den Absender oder an i...@berenberg.de. Please refer
>>to http://www.berenberg.de/my_berenberg/disclaimer_e.html for our
>>confidentiality notice.
>
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>
> Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist vertraulich 
> und kann dem Bank- und Datengeheimnis unterliegen oder sonst rechtlich 
> geschuetzte Daten und Informationen enthalten. Wenn Sie nicht der richtige 
> Adressat sind oder diese Nachricht irrtuemlich erhalten haben, informieren 
> Sie bitte sofort den Absender über die Antwortfunktion. Anschliessend 
> moechten Sie bitte diese Nachricht einschliesslich etwa beigefuegter Anhaenge 
> unverzueglich vollstaendig loeschen. Das unerlaubte Kopieren oder Speichern 
> dieser Nachricht und/oder der ihr etwa beigefuegten Anhaenge sowie die 
> unbefugte Weitergabe der darin enthaltenen Daten und Informationen sind nicht 
> gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen namens 
> unseres Hauses grundsaetzlich der Unterschriften zweier ausreichend 
> bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir verschicken daher 
> keine rechtsverbindlichen Erklaerungen per E-Mail an Dritte. Demgemaess 
> nehmen wir per E-Mail auch keine rechtsverbindlichen Erklaerungen oder 
> Auftraege von Dritten entgegen.

RE: Configure query parser to handle field name case-insensitive

[Duck Geraint (ext) GBJH]

As you're using the extended dismax parser, it has an option to include per 
field aliasing:
https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser

You could include this in your solr requesthandler config, e.g.
id

Which would direct ID:1 to instead search id:1

Geraint



-Original Message-
From: Peemöller, Björn [mailto:bjoern.peemoel...@berenberg.de]
Sent: 15 May 2017 14:17
To: 'solr-user@lucene.apache.org' 
Subject: AW: Configure query parser to handle field name case-insensitive

Hi Rick,

thank you for your reply! I really meant field *names*, since our values are 
already processed by a lower case filter (both index and query). However, our 
users are confused because they can search for "id:1" but not for "ID:1". 
Furthermore, we employ the EDisMax query parser, so then even get no error 
message.

Therefore, I thought it may be sufficient to map all field names to lower case 
at the query level so that I do not have to introduce additional fields.

Regards,
Björn

-Ursprüngliche Nachricht-
Von: Rick Leir [mailto:rl...@leirtech.com]
Gesendet: Montag, 15. Mai 2017 13:48
An: solr-user@lucene.apache.org
Betreff: Re: Configure query parser to handle field name case-insensitive

Björn
Field names or values? I assume values. Your analysis chain in schema.xml 
probably downcases chars, if not then that could be your problem.

Field _name_? Then you might have to copyfield the field to a new field with 
the desired case. Avoid doing that if you can. Cheers -- Rick

On May 15, 2017 5:48:09 AM EDT, "Peemöller, Björn" 
 wrote:
>Hi all,
>
>I'm fairly new at using Solr and I need to configure our instance to
>accept field names in both uppercase and lowercase (they are defined as
>lowercase in our configuration). Is there a simple way to achieve this?
>
>Thanks in advance,
>Björn
>
>Björn Peemöller
>IT & IT Operations
>
>BERENBERG
>Joh. Berenberg, Gossler & Co. KG
>Neuer Jungfernstieg 20
>20354 Hamburg
>
>Telefon +49 40 350 60-8548
>Telefax +49 40 350 60-900
>E-Mail
>bjoern.peemoel...@berenberg.de
>www.berenberg.de
>
>Sitz: Hamburg - Amtsgericht Hamburg HRA 42659
>
>
>Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist
>vertraulich und kann dem Bank- und Datengeheimnis unterliegen oder
>sonst rechtlich geschuetzte Daten und Informationen enthalten. Wenn Sie
>nicht der richtige Adressat sind oder diese Nachricht irrtuemlich
>erhalten haben, informieren Sie bitte sofort den Absender über die
>Antwortfunktion. Anschliessend moechten Sie bitte diese Nachricht
>einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig
>loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht
>und/oder der ihr etwa beigefuegten Anhaenge sowie die unbefugte
>Weitergabe der darin enthaltenen Daten und Informationen sind nicht
>gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen
>namens unseres Hauses grundsaetzlich der Unterschriften zweier
>ausreichend bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir
>verschicken daher keine rechtsverbindlichen Erklaerungen per E-Mail an
>Dritte. Demgemaess nehmen wir per E-Mail auch keine rechtsverbindlichen
>Erklaerungen oder Auftraege von Dritten entgegen.
>Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden
>Sie sich bitte an den Absender oder an i...@berenberg.de. Please refer
>to http://www.berenberg.de/my_berenberg/disclaimer_e.html for our
>confidentiality notice.

--
Sorry for being brief. Alternate email is rickleir at yahoo dot com

Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist vertraulich und 
kann dem Bank- und Datengeheimnis unterliegen oder sonst rechtlich geschuetzte 
Daten und Informationen enthalten. Wenn Sie nicht der richtige Adressat sind 
oder diese Nachricht irrtuemlich erhalten haben, informieren Sie bitte sofort 
den Absender über die Antwortfunktion. Anschliessend moechten Sie bitte diese 
Nachricht einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig 
loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht und/oder der 
ihr etwa beigefuegten Anhaenge sowie die unbefugte Weitergabe der darin 
enthaltenen Daten und Informationen sind nicht gestattet. Wir weisen darauf 
hin, dass rechtsverbindliche Erklaerungen namens unseres Hauses grundsaetzlich 
der Unterschriften zweier ausreichend bevollmaechtigter Vertreter unseres 
Hauses beduerfen. Wir verschicken daher keine rechtsverbindlichen Erklaerungen 
per E-Mail an Dritte. Demgemaess nehmen wir per E-Mail auch keine 
rechtsverbindlichen Erklaerungen oder Auftraege von Dritten entgegen.
Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden Sie sich 
bitte an den Absender oder an i...@berenberg.de. Please refer to 
http://www.berenberg.de/my_berenberg/disclaimer_e.html for our confidentiality 
notice.

Re: Stop solr instance

[Erick Erickson]

I'd pick the one I was most comfortable with and just try it.

Best,
Erick

On Mon, May 15, 2017 at 6:26 AM, Mithu Tokder  wrote:
> Hi,
> I have one question regarding stopping Solr instance.
> Solr is deployed in three machines(cluster deployment). I have configured
> STOP.PORT and STOP.KEY in start script, accordingly configured STOP.PORT
> and STOP.KEY in stop script.
> There are three sets of start & stop script for each machine.
>
> Now my question is that as the solr instances are running in three machines
> so is it required to configure same value for STOP.PORT and STOP.KEY in
> start and stop script of three machines or i can use separate value for
> them.
>
> *Configuration 1*
> *Machine 1:*
> STOP.PORT=8071
> STOP.KEY=mysecret1
>
> *Machine 2:*
> STOP.PORT=8071
> STOP.KEY=mysecret1
>
> *Machine 3:*
> STOP.PORT=8071
> STOP.KEY=mysecret1
>
> *Configuration 2*
> *Machine 1:*
> STOP.PORT=8071
> STOP.KEY=mysecret1
>
> *Machine 2:*
> STOP.PORT=8072
> STOP.KEY=mysecret2
>
> *Machine 3:*
> STOP.PORT=8073
> STOP.KEY=mysecret3
>
> Which configuration should i use?
>
> Thanks in advance.
>
> With Regards,
> Mithu Tokder

Stop solr instance

[Mithu Tokder]

Hi,
I have one question regarding stopping Solr instance.
Solr is deployed in three machines(cluster deployment). I have configured
STOP.PORT and STOP.KEY in start script, accordingly configured STOP.PORT
and STOP.KEY in stop script.
There are three sets of start & stop script for each machine.

Now my question is that as the solr instances are running in three machines
so is it required to configure same value for STOP.PORT and STOP.KEY in
start and stop script of three machines or i can use separate value for
them.

*Configuration 1*
*Machine 1:*
STOP.PORT=8071
STOP.KEY=mysecret1

*Machine 2:*
STOP.PORT=8071
STOP.KEY=mysecret1

*Machine 3:*
STOP.PORT=8071
STOP.KEY=mysecret1

*Configuration 2*
*Machine 1:*
STOP.PORT=8071
STOP.KEY=mysecret1

*Machine 2:*
STOP.PORT=8072
STOP.KEY=mysecret2

*Machine 3:*
STOP.PORT=8073
STOP.KEY=mysecret3

Which configuration should i use?

Thanks in advance.

With Regards,
Mithu Tokder

Re: SolrCloud ... Unable to create core ... Caused by: Lock held by this virtual machine:...

[Susheel Kumar]

Sorry, Thomas.  I am unable to download right now the solr 6.5.1 version.
What i would suggest is try to create shard on two different machines/run
two solr instances on different ports. There may be some issue with this
version.

Anyone insight with 6.5.1?

Thanks,
Susheel

On Mon, May 15, 2017 at 9:46 AM, Susheel Kumar 
wrote:

> Good test.  That tells something.  Let me also run same on my end.
>
> On Mon, May 15, 2017 at 9:33 AM, Thomas Porschberg  > wrote:
>
>> Hi,
>>
>> I get no error message and the shard is created when I use
>> numShards=1
>> in the url.
>>
>> http://localhost:8983/solr/admin/collections?action=CREATE;
>> name=karpfen=1=1
>> erNode=1=karpfen
>>  --> success
>>
>> http://localhost:8983/solr/admin/collections?action=CREATE;
>> name=karpfen=2=1
>> erNode=2=karpfen
>> --> error
>>
>> Thomas
>>
>>
>> > Susheel Kumar  hat am 15. Mai 2017 um 14:36
>> geschrieben:
>> >
>> >
>> > what happens if you create just one shard.  Just use this command
>> directly
>> > on browser or thru curl.  Empty the contents from
>> >  /home/pberg/solr_new2/solr-6.5.1/server/data before running
>> >
>> > http://localhost:8983/solr/admin/collections?action=
>> > CREATE=karpfen=1=1&
>> > maxShardsPerNode=1=karpfen
>> > > name=karpfen=2=1
>> erNode=2=karpfen>
>> >
>> > On Mon, May 15, 2017 at 2:14 AM, Thomas Porschberg <
>> tho...@randspringer.de>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I have problems to setup solrcloud on one node with 2 shards. What I
>> did:
>> > >
>> > > 1. Started a external zookeeper
>> > > 2. Ensured that no solr process is running with 'bin/solr status'
>> > > 3. Posted a working conf directory from a non-cloud solr to zookeeper
>> > >with
>> > >'bin/solr zk upconfig -n karpfen -d /home/pberg/solr_new/solr-6.5.
>> 1/server/solr/tommy/conf
>> > > -z localhost:2181'
>> > >--> no errors
>> > > 4. Started solr in cloud mode with
>> > >   'bin/solr -c -z localhost:2181'
>> > > 5. Tried to create a new collection with 2 shards with
>> > >'bin/solr create -c karpfen -shards 2'
>> > >
>> > > The output is:
>> > >
>> > > Connecting to ZooKeeper at localhost:2181 ...
>> > > INFO  - 2017-05-12 18:52:22.807; org.apache.solr.client.solrj.i
>> mpl.ZkClientClusterStateProvider;
>> > > Cluster at localhost:2181 ready
>> > > Re-using existing configuration directory karpfen
>> > >
>> > > Creating new collection 'karpfen' using command:
>> > > http://localhost:8983/solr/admin/collections?action=
>> > > CREATE=karpfen=2=1&
>> > > maxShardsPerNode=2=karpfen
>> > >
>> > >
>> > > ERROR: Failed to create collection 'karpfen' due to: {127.0.1.1:8983
>> > > _solr=org.apache.solr.client.solrj.impl.HttpSolrClient$Remot
>> eSolrException:Error
>> > > from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore
>> > > 'karpfen_shard2_replica1': Unable to create core
>> [karpfen_shard2_replica1]
>> > > Caused by: Lock held by this virtual machine:
>> /home/pberg/solr_new2/solr-6.
>> > > 5.1/server/data/ohrdruf_bestand/index/write.lock}
>> > >
>> > >
>> > > The conf directory I copied contains the following files:
>> > > currency.xml elevate.xml  protwords.txt
>>  stopwords.txt
>> > > dataimport-cobt2.properties  lang schema.xml  synonyms.txt
>> > > dataimport.xml   params.json  solrconfig.xml
>> > >
>> > > "lang" is a directory.
>> > >
>> > > Are my steps wrong? Did I miss something important?
>> > >
>> > > Any help is really welcome.
>> > >
>> > > Thomas
>> > >
>>
>
>

Re: solr /export handler - behavior during close()

[Joel Bernstein]

Ok please do report any issues you run into. This is quite a good bug
report.

I reviewed the code and I believe I see the problem. The problem seems to
be that output code from the /stream handler is not properly accounting for
client disconnects and closing the underlying stream. What I see in the
code is that exceptions coming from read() in the stream do automatically
close the underlying stream. But exceptions from the writing of the stream
do not close the stream. This needs to be fixed.

A few questions about your streaming implementation:

1) Are you sending requests to the /stream handler? Or are you embedding
CloudSolrStream in your application and bypassing the /stream handler?

2) If you're sending Streaming Expressions to the stream handler are you
using SolrStream or CloudSolrStream to send the expression?

3) What version of Solr are you using.

4) Have you implemented any custom streams?


#2 is an important question. If you're sending expressions to the /stream
handler using CloudSolrStream the collection running the expression would
have to be setup a specific way. The collection running the expression will
have to be a* single shard collection*. You can have as many replicas as
you want but only one shard. That's because CloudSolrStream picks one
replica in each shard to forward the request to then merges the results
from the shards. So if you send in an expression using CloudSolrStream that
expression will be sent to each shard to be run and each shard will be
duplicating the work and return duplicate results.




















Joel Bernstein
http://joelsolr.blogspot.com/

On Sat, May 13, 2017 at 7:03 PM, Susmit Shukla 
wrote:

> Thanks Joel
> Streaming is awesome, just had a huge implementation in my project. I found
> out a couple more issues with streaming and did local hacks for them, would
> raise them too.
>
> On Sat, May 13, 2017 at 2:09 PM, Joel Bernstein 
> wrote:
>
> > Ah, then this is unexpected behavior. Can you open a ticket for this?
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Sat, May 13, 2017 at 2:51 PM, Susmit Shukla 
> > wrote:
> >
> > > Hi Joel,
> > >
> > > I was using CloudSolrStream for the above test. Below is the call
> stack.
> > >
> > > at
> > > org.apache.http.impl.io.ChunkedInputStream.read(
> > > ChunkedInputStream.java:215)
> > > at
> > > org.apache.http.impl.io.ChunkedInputStream.close(
> > > ChunkedInputStream.java:316)
> > > at
> > > org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(
> > > ResponseEntityProxy.java:128)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.checkClose(
> > > EofSensorInputStream.java:228)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.close(
> > > EofSensorInputStream.java:174)
> > > at sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:378)
> > > at sun.nio.cs.StreamDecoder.close(StreamDecoder.java:193)
> > > at java.io.InputStreamReader.close(InputStreamReader.java:199)
> > > at
> > > org.apache.solr.client.solrj.io.stream.JSONTupleStream.
> > > close(JSONTupleStream.java:91)
> > > at
> > > org.apache.solr.client.solrj.io.stream.SolrStream.close(
> > > SolrStream.java:186)
> > >
> > > Thanks,
> > > Susmit
> > >
> > > On Sat, May 13, 2017 at 10:48 AM, Joel Bernstein 
> > > wrote:
> > >
> > > > I was just reading the Java docs on the ChunkedInputStream.
> > > >
> > > > "Note that this class NEVER closes the underlying stream"
> > > >
> > > > In that scenario the /export would indeed continue to send data. I
> > think
> > > we
> > > > can consider this an anti-pattern for the /export handler currently.
> > > >
> > > > I would suggest using one of the Streaming Clients to connect to the
> > > export
> > > > handler. Either CloudSolrStream or SolrStream will both interact with
> > the
> > > > /export handler in a the way that it expects.
> > > >
> > > >
> > > > Joel Bernstein
> > > > http://joelsolr.blogspot.com/
> > > >
> > > > On Sat, May 13, 2017 at 12:28 PM, Susmit Shukla <
> > shukla.sus...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Joel,
> > > > >
> > > > > I did not observe that. On calling close() on stream, it cycled
> > through
> > > > all
> > > > > the hits that /export handler calculated.
> > > > > e.g. with a *:* query and export handler on a 100k document index,
> I
> > > > could
> > > > > see the 100kth record printed on the http wire debug log although
> > close
> > > > was
> > > > > called after reading 1st tuple. The time taken for the operation
> with
> > > > > close() call was same as that if I had read all the 100k tuples.
> > > > > As I have pointed out, close() on underlying ChunkedInputStream
> calls
> > > > > read() and solr server has probably no way to distinguish it from
> > > read()
> > > > > happening from regular tuple reads..
> > > > > I think there should be an abort() API for solr streams that hooks
> > into
> > > > > httpmethod.abort() . 

Re: SolrCloud ... Unable to create core ... Caused by: Lock held by this virtual machine:...

[Susheel Kumar]

Good test.  That tells something.  Let me also run same on my end.

On Mon, May 15, 2017 at 9:33 AM, Thomas Porschberg 
wrote:

> Hi,
>
> I get no error message and the shard is created when I use
> numShards=1
> in the url.
>
> http://localhost:8983/solr/admin/collections?action=
> CREATE=karpfen=1=1&
> maxShardsPerNode=1=karpfen
>  --> success
>
> http://localhost:8983/solr/admin/collections?action=
> CREATE=karpfen=2=1&
> maxShardsPerNode=2=karpfen
> --> error
>
> Thomas
>
>
> > Susheel Kumar  hat am 15. Mai 2017 um 14:36
> geschrieben:
> >
> >
> > what happens if you create just one shard.  Just use this command
> directly
> > on browser or thru curl.  Empty the contents from
> >  /home/pberg/solr_new2/solr-6.5.1/server/data before running
> >
> > http://localhost:8983/solr/admin/collections?action=
> > CREATE=karpfen=1=1&
> > maxShardsPerNode=1=karpfen
> >  CREATE=karpfen=2=1&
> maxShardsPerNode=2=karpfen>
> >
> > On Mon, May 15, 2017 at 2:14 AM, Thomas Porschberg <
> tho...@randspringer.de>
> > wrote:
> >
> > > Hi,
> > >
> > > I have problems to setup solrcloud on one node with 2 shards. What I
> did:
> > >
> > > 1. Started a external zookeeper
> > > 2. Ensured that no solr process is running with 'bin/solr status'
> > > 3. Posted a working conf directory from a non-cloud solr to zookeeper
> > >with
> > >'bin/solr zk upconfig -n karpfen -d /home/pberg/solr_new/solr-6.5.
> 1/server/solr/tommy/conf
> > > -z localhost:2181'
> > >--> no errors
> > > 4. Started solr in cloud mode with
> > >   'bin/solr -c -z localhost:2181'
> > > 5. Tried to create a new collection with 2 shards with
> > >'bin/solr create -c karpfen -shards 2'
> > >
> > > The output is:
> > >
> > > Connecting to ZooKeeper at localhost:2181 ...
> > > INFO  - 2017-05-12 18:52:22.807; org.apache.solr.client.solrj.impl.
> ZkClientClusterStateProvider;
> > > Cluster at localhost:2181 ready
> > > Re-using existing configuration directory karpfen
> > >
> > > Creating new collection 'karpfen' using command:
> > > http://localhost:8983/solr/admin/collections?action=
> > > CREATE=karpfen=2=1&
> > > maxShardsPerNode=2=karpfen
> > >
> > >
> > > ERROR: Failed to create collection 'karpfen' due to: {127.0.1.1:8983
> > > _solr=org.apache.solr.client.solrj.impl.HttpSolrClient$
> RemoteSolrException:Error
> > > from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore
> > > 'karpfen_shard2_replica1': Unable to create core
> [karpfen_shard2_replica1]
> > > Caused by: Lock held by this virtual machine:
> /home/pberg/solr_new2/solr-6.
> > > 5.1/server/data/ohrdruf_bestand/index/write.lock}
> > >
> > >
> > > The conf directory I copied contains the following files:
> > > currency.xml elevate.xml  protwords.txt   stopwords.txt
> > > dataimport-cobt2.properties  lang schema.xml  synonyms.txt
> > > dataimport.xml   params.json  solrconfig.xml
> > >
> > > "lang" is a directory.
> > >
> > > Are my steps wrong? Did I miss something important?
> > >
> > > Any help is really welcome.
> > >
> > > Thomas
> > >
>

Re: SolrCloud ... Unable to create core ... Caused by: Lock held by this virtual machine:...

[Thomas Porschberg]

Hi,

I get no error message and the shard is created when I use 
numShards=1
in the url.

http://localhost:8983/solr/admin/collections?action=CREATE=karpfen=1=1=1=karpfen
 --> success

http://localhost:8983/solr/admin/collections?action=CREATE=karpfen=2=1=2=karpfen
--> error

Thomas


> Susheel Kumar  hat am 15. Mai 2017 um 14:36 
> geschrieben:
> 
> 
> what happens if you create just one shard.  Just use this command directly
> on browser or thru curl.  Empty the contents from
>  /home/pberg/solr_new2/solr-6.5.1/server/data before running
> 
> http://localhost:8983/solr/admin/collections?action=
> CREATE=karpfen=1=1&
> maxShardsPerNode=1=karpfen
> 
> 
> On Mon, May 15, 2017 at 2:14 AM, Thomas Porschberg 
> wrote:
> 
> > Hi,
> >
> > I have problems to setup solrcloud on one node with 2 shards. What I did:
> >
> > 1. Started a external zookeeper
> > 2. Ensured that no solr process is running with 'bin/solr status'
> > 3. Posted a working conf directory from a non-cloud solr to zookeeper
> >with
> >'bin/solr zk upconfig -n karpfen -d 
> > /home/pberg/solr_new/solr-6.5.1/server/solr/tommy/conf
> > -z localhost:2181'
> >--> no errors
> > 4. Started solr in cloud mode with
> >   'bin/solr -c -z localhost:2181'
> > 5. Tried to create a new collection with 2 shards with
> >'bin/solr create -c karpfen -shards 2'
> >
> > The output is:
> >
> > Connecting to ZooKeeper at localhost:2181 ...
> > INFO  - 2017-05-12 18:52:22.807; 
> > org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider;
> > Cluster at localhost:2181 ready
> > Re-using existing configuration directory karpfen
> >
> > Creating new collection 'karpfen' using command:
> > http://localhost:8983/solr/admin/collections?action=
> > CREATE=karpfen=2=1&
> > maxShardsPerNode=2=karpfen
> >
> >
> > ERROR: Failed to create collection 'karpfen' due to: {127.0.1.1:8983
> > _solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> > from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore
> > 'karpfen_shard2_replica1': Unable to create core [karpfen_shard2_replica1]
> > Caused by: Lock held by this virtual machine: /home/pberg/solr_new2/solr-6.
> > 5.1/server/data/ohrdruf_bestand/index/write.lock}
> >
> >
> > The conf directory I copied contains the following files:
> > currency.xml elevate.xml  protwords.txt   stopwords.txt
> > dataimport-cobt2.properties  lang schema.xml  synonyms.txt
> > dataimport.xml   params.json  solrconfig.xml
> >
> > "lang" is a directory.
> >
> > Are my steps wrong? Did I miss something important?
> >
> > Any help is really welcome.
> >
> > Thomas
> >

AW: Configure query parser to handle field name case-insensitive

[Peemöller , Björn]

Hi Rick,

thank you for your reply! I really meant field *names*, since our values are 
already processed by a lower case filter (both index and query). However, our 
users are confused because they can search for "id:1" but not for "ID:1". 
Furthermore, we employ the EDisMax query parser, so then even get no error 
message.

Therefore, I thought it may be sufficient to map all field names to lower case 
at the query level so that I do not have to introduce additional fields.

Regards,
Björn

-Ursprüngliche Nachricht-
Von: Rick Leir [mailto:rl...@leirtech.com] 
Gesendet: Montag, 15. Mai 2017 13:48
An: solr-user@lucene.apache.org
Betreff: Re: Configure query parser to handle field name case-insensitive

Björn
Field names or values? I assume values. Your analysis chain in schema.xml 
probably downcases chars, if not then that could be your problem.

Field _name_? Then you might have to copyfield the field to a new field with 
the desired case. Avoid doing that if you can. Cheers -- Rick

On May 15, 2017 5:48:09 AM EDT, "Peemöller, Björn" 
 wrote:
>Hi all,
>
>I'm fairly new at using Solr and I need to configure our instance to 
>accept field names in both uppercase and lowercase (they are defined as 
>lowercase in our configuration). Is there a simple way to achieve this?
>
>Thanks in advance,
>Björn
>
>Björn Peemöller
>IT & IT Operations
>
>BERENBERG
>Joh. Berenberg, Gossler & Co. KG
>Neuer Jungfernstieg 20
>20354 Hamburg
>
>Telefon +49 40 350 60-8548
>Telefax +49 40 350 60-900
>E-Mail
>bjoern.peemoel...@berenberg.de
>www.berenberg.de
>
>Sitz: Hamburg - Amtsgericht Hamburg HRA 42659
>
>
>Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist 
>vertraulich und kann dem Bank- und Datengeheimnis unterliegen oder 
>sonst rechtlich geschuetzte Daten und Informationen enthalten. Wenn Sie 
>nicht der richtige Adressat sind oder diese Nachricht irrtuemlich 
>erhalten haben, informieren Sie bitte sofort den Absender über die 
>Antwortfunktion. Anschliessend moechten Sie bitte diese Nachricht 
>einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig 
>loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht 
>und/oder der ihr etwa beigefuegten Anhaenge sowie die unbefugte 
>Weitergabe der darin enthaltenen Daten und Informationen sind nicht 
>gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen 
>namens unseres Hauses grundsaetzlich der Unterschriften zweier 
>ausreichend bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir 
>verschicken daher keine rechtsverbindlichen Erklaerungen per E-Mail an 
>Dritte. Demgemaess nehmen wir per E-Mail auch keine rechtsverbindlichen 
>Erklaerungen oder Auftraege von Dritten entgegen.
>Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden 
>Sie sich bitte an den Absender oder an i...@berenberg.de. Please refer 
>to http://www.berenberg.de/my_berenberg/disclaimer_e.html for our 
>confidentiality notice.

--
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist vertraulich und 
kann dem Bank- und Datengeheimnis unterliegen oder sonst rechtlich geschuetzte 
Daten und Informationen enthalten. Wenn Sie nicht der richtige Adressat sind 
oder diese Nachricht irrtuemlich erhalten haben, informieren Sie bitte sofort 
den Absender über die Antwortfunktion. Anschliessend moechten Sie bitte diese 
Nachricht einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig 
loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht und/oder der 
ihr etwa beigefuegten Anhaenge sowie die unbefugte Weitergabe der darin 
enthaltenen Daten und Informationen sind nicht gestattet. Wir weisen darauf 
hin, dass rechtsverbindliche Erklaerungen namens unseres Hauses grundsaetzlich 
der Unterschriften zweier ausreichend bevollmaechtigter Vertreter unseres 
Hauses beduerfen. Wir verschicken daher keine rechtsverbindlichen Erklaerungen 
per E-Mail an Dritte. Demgemaess nehmen wir per E-Mail auch keine 
rechtsverbindlichen Erklaerungen oder Auftraege von Dritten entgegen. 
Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden Sie sich 
bitte an den Absender oder an i...@berenberg.de. Please refer to 
http://www.berenberg.de/my_berenberg/disclaimer_e.html for our confidentiality 
notice.

Re: SolrCloud ... Unable to create core ... Caused by: Lock held by this virtual machine:...

[Susheel Kumar]

what happens if you create just one shard.  Just use this command directly
on browser or thru curl.  Empty the contents from
 /home/pberg/solr_new2/solr-6.5.1/server/data before running

http://localhost:8983/solr/admin/collections?action=
CREATE=karpfen=1=1&
maxShardsPerNode=1=karpfen


On Mon, May 15, 2017 at 2:14 AM, Thomas Porschberg 
wrote:

> Hi,
>
> I have problems to setup solrcloud on one node with 2 shards. What I did:
>
> 1. Started a external zookeeper
> 2. Ensured that no solr process is running with 'bin/solr status'
> 3. Posted a working conf directory from a non-cloud solr to zookeeper
>with
>'bin/solr zk upconfig -n karpfen -d 
> /home/pberg/solr_new/solr-6.5.1/server/solr/tommy/conf
> -z localhost:2181'
>--> no errors
> 4. Started solr in cloud mode with
>   'bin/solr -c -z localhost:2181'
> 5. Tried to create a new collection with 2 shards with
>'bin/solr create -c karpfen -shards 2'
>
> The output is:
>
> Connecting to ZooKeeper at localhost:2181 ...
> INFO  - 2017-05-12 18:52:22.807; 
> org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider;
> Cluster at localhost:2181 ready
> Re-using existing configuration directory karpfen
>
> Creating new collection 'karpfen' using command:
> http://localhost:8983/solr/admin/collections?action=
> CREATE=karpfen=2=1&
> maxShardsPerNode=2=karpfen
>
>
> ERROR: Failed to create collection 'karpfen' due to: {127.0.1.1:8983
> _solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore
> 'karpfen_shard2_replica1': Unable to create core [karpfen_shard2_replica1]
> Caused by: Lock held by this virtual machine: /home/pberg/solr_new2/solr-6.
> 5.1/server/data/ohrdruf_bestand/index/write.lock}
>
>
> The conf directory I copied contains the following files:
> currency.xml elevate.xml  protwords.txt   stopwords.txt
> dataimport-cobt2.properties  lang schema.xml  synonyms.txt
> dataimport.xml   params.json  solrconfig.xml
>
> "lang" is a directory.
>
> Are my steps wrong? Did I miss something important?
>
> Any help is really welcome.
>
> Thomas
>

Re: Learn To Rank Questions

[Joël Trigalo]

1.
So I think it is a spark problem first (https://issues.apache.org/jir
a/browse/SPARK-10413). What we can do is to create our own model (cf
https://github.com/apache/lucene-solr/tree/master/solr/contr
ib/ltr/src/java/org/apache/solr/ltr/model) that applies the prediction, it
should be easy to do for a simple model, like logistic regression.
For PMML, the idea would also be to implement a Model that reuse a java lib
able to apply PMML.

2.
This function query gives you TF IDF of textField vs userQuery for the doc

 {!edismax qf='textField' mm=100% v=${userQuery} tie=0.1}

Also it seems to me LTR only allows float features which is a limitation.


3.
If the boost value is an index time boost I don't think it is possible. You
could put the feature you want in a field at index time and then use
FieldValueFeature
to extract it.

On Thu, May 11, 2017 at 8:16 PM, Grant Ingersoll 
wrote:

> Hi,
>
> Just getting up to speed on LTR and have a few questions (most of which are
> speculative at this point and exploratory, as I have a couple of talks
> coming up on this and other relevance features):
>
> 1. Has anyone looked at what's involved with supporting SparkML or other
> models (e.g. PMML)?
>
> 2. Has anyone looked at features for text?  i.e. returning TF-IDF vectors
> or similar.  FieldValueFeature is kind of like this, but I might want
> weights for the terms, not just the actual values.  I could get this via
> term vectors, but then it doesn't fit the framework.
>
> 3. How about payloads and/or things like boost values for documents as
> features?
>
> 4. Are there example docs of training and using the
> MultipleAdditiveTreesModel?  I see unit tests for them, but looking for
> something similar to the python script in the example dir.
>
> On 2 and 3, I imagine some of this can be done creatively via the
> SolrFeature and function queries.
>
> Thanks,
> Grant
>

Re: Configure query parser to handle field name case-insensitive

[Rick Leir]

Björn
Field names or values? I assume values. Your analysis chain in schema.xml 
probably downcases chars, if not then that could be your problem.

Field _name_? Then you might have to copyfield the field to a new field with 
the desired case. Avoid doing that if you can. Cheers -- Rick

On May 15, 2017 5:48:09 AM EDT, "Peemöller, Björn" 
 wrote:
>Hi all,
>
>I'm fairly new at using Solr and I need to configure our instance to
>accept field names in both uppercase and lowercase (they are defined as
>lowercase in our configuration). Is there a simple way to achieve this?
>
>Thanks in advance,
>Björn
>
>Björn Peemöller
>IT & IT Operations
>
>BERENBERG
>Joh. Berenberg, Gossler & Co. KG
>Neuer Jungfernstieg 20
>20354 Hamburg
>
>Telefon +49 40 350 60-8548
>Telefax +49 40 350 60-900
>E-Mail
>bjoern.peemoel...@berenberg.de
>www.berenberg.de
>
>Sitz: Hamburg - Amtsgericht Hamburg HRA 42659
>
>
>Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist
>vertraulich und kann dem Bank- und Datengeheimnis unterliegen oder
>sonst rechtlich geschuetzte Daten und Informationen enthalten. Wenn Sie
>nicht der richtige Adressat sind oder diese Nachricht irrtuemlich
>erhalten haben, informieren Sie bitte sofort den Absender über die
>Antwortfunktion. Anschliessend moechten Sie bitte diese Nachricht
>einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig
>loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht
>und/oder der ihr etwa beigefuegten Anhaenge sowie die unbefugte
>Weitergabe der darin enthaltenen Daten und Informationen sind nicht
>gestattet. Wir weisen darauf hin, dass rechtsverbindliche Erklaerungen
>namens unseres Hauses grundsaetzlich der Unterschriften zweier
>ausreichend bevollmaechtigter Vertreter unseres Hauses beduerfen. Wir
>verschicken daher keine rechtsverbindlichen Erklaerungen per E-Mail an
>Dritte. Demgemaess nehmen wir per E-Mail auch keine rechtsverbindlichen
>Erklaerungen oder Auftraege von Dritten entgegen. 
>Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden
>Sie sich bitte an den Absender oder an i...@berenberg.de. Please refer
>to http://www.berenberg.de/my_berenberg/disclaimer_e.html for our
>confidentiality notice.

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Configure query parser to handle field name case-insensitive

[Peemöller , Björn]

Hi all,

I'm fairly new at using Solr and I need to configure our instance to accept 
field names in both uppercase and lowercase (they are defined as lowercase in 
our configuration). Is there a simple way to achieve this?

Thanks in advance,
Björn

Björn Peemöller
IT & IT Operations

BERENBERG
Joh. Berenberg, Gossler & Co. KG
Neuer Jungfernstieg 20
20354 Hamburg

Telefon +49 40 350 60-8548
Telefax +49 40 350 60-900
E-Mail bjoern.peemoel...@berenberg.de
www.berenberg.de

Sitz: Hamburg - Amtsgericht Hamburg HRA 42659


Diese Nachricht einschliesslich etwa beigefuegter Anhaenge ist vertraulich und 
kann dem Bank- und Datengeheimnis unterliegen oder sonst rechtlich geschuetzte 
Daten und Informationen enthalten. Wenn Sie nicht der richtige Adressat sind 
oder diese Nachricht irrtuemlich erhalten haben, informieren Sie bitte sofort 
den Absender über die Antwortfunktion. Anschliessend moechten Sie bitte diese 
Nachricht einschliesslich etwa beigefuegter Anhaenge unverzueglich vollstaendig 
loeschen. Das unerlaubte Kopieren oder Speichern dieser Nachricht und/oder der 
ihr etwa beigefuegten Anhaenge sowie die unbefugte Weitergabe der darin 
enthaltenen Daten und Informationen sind nicht gestattet. Wir weisen darauf 
hin, dass rechtsverbindliche Erklaerungen namens unseres Hauses grundsaetzlich 
der Unterschriften zweier ausreichend bevollmaechtigter Vertreter unseres 
Hauses beduerfen. Wir verschicken daher keine rechtsverbindlichen Erklaerungen 
per E-Mail an Dritte. Demgemaess nehmen wir per E-Mail auch keine 
rechtsverbindlichen Erklaerungen oder Auftraege von Dritten entgegen. 
Sollten Sie Schwierigkeiten beim Oeffnen dieser E-Mail haben, wenden Sie sich 
bitte an den Absender oder an i...@berenberg.de. Please refer to 
http://www.berenberg.de/my_berenberg/disclaimer_e.html for our confidentiality 
notice.

Re: Seeing odd behavior with implicit routing

[Shalin Shekhar Mangar]

On Sun, May 14, 2017 at 7:40 PM, Chris Troullis  wrote:
> Hi,
>
> I've been experimenting with various sharding strategies with Solr cloud
> (6.5.1), and am seeing some odd behavior when using the implicit router. I
> am probably either doing something wrong or misinterpreting what I am
> seeing in the logs, but if someone could help clarify that would be awesome.
>
> I created a collection using the implicit router, created 10 shards, named
> shard1, shard2, etc. I indexed 3000 documents to each shard, routed by
> setting the _route_ field on the documents in my schema. All works fine, I
> verified there are 3000 documents in each shard.
>
> The odd behavior I am seeing is when I try to route a query to a specific
> shard. I submitted a simple query to shard1 using the request parameter
> _route_=shard1. The query comes back fine, but when I looked in the logs,
> it looked like it was issuing 3 separate requests:
>
> 1. The original query to shard1
> 2. A 2nd query to shard1 with the parameter ids=a bunch of document ids
> 3. The original query to a random shard (changes every time I run the query)
>
> It looks like the first query is getting back a list of ids, and the 2nd
> query is retrieving the documents for those ids? I assume this is some solr
> cloud implementation detail.
>
> What I don't understand is the 3rd query. Why is it issuing the original
> query to a random shard every time, when I am specifying the _route_? The
> _route_ parameter is definitely doing something, because if I remove it, it
> is querying all shards (which I would expect).
>
> Any ideas? I can provide the actual queries from the logs if required.

How many nodes is this collection distributed across? I suspect that
you are using a single node for experimentation?

What happens with _route_=shard1 parameter and implicit routing is
that the _route_ parameter is resolved to a list of replicas of
shard1. But, SolrJ uses only the node name of the replica along with
the collection name to make the request (this is important, we'll come
back to this later). So, ordinarily, that node hosts a single shard
(shard1) and when it receives the request, it will optimize the search
to go the non-distributed code path (since the replica has all the
data needed to satisfy the search).

But interesting things happen when the node hosts more than one shard
(say shard1 and shard3 both). When we query such a node using just the
collection name, the collection name can be resolved to either shard1
or shard3 -- this is picked randomly without looking at _route_
parameter at all. If shard3 is picked, it looks at the request, sees
that it doesn't have all the necessary data and decides to follow the
two-phase distributed search path where phase 1 is to get the ids and
score of the documents matching the query from all participating
shards (the list of such shards is limited by _route_ parameter, which
in our case will be only shard1) and a second phase where we get the
actual stored fields to be returned to the user. So you get three
queries in the log, 1) phase 1 of distributed search hitting shard1,
2) phase two of distributed search hitting shard1 and 3) the
distributed scatter-gather search run by shard3.

So to recap, this is happening because you have more than one shard1
hosted on a node. Easy workaround is to have each shard hosted on a
unique node. But we can improve things on the solr side as well by 1)
having SolrJ resolve requests down to node name and core name, 2)
having the collection name to core name resolution take _route_ param
into account. Both 1 and 2 can solve the problem. Can you please open
a Jira issue?

>
> Thanks,
>
> Chris



-- 
Regards,
Shalin Shekhar Mangar.

SolrCloud ... Unable to create core ... Caused by: Lock held by this virtual machine:...

[Thomas Porschberg]

Hi,

I have problems to setup solrcloud on one node with 2 shards. What I did:

1. Started a external zookeeper
2. Ensured that no solr process is running with 'bin/solr status'
3. Posted a working conf directory from a non-cloud solr to zookeeper
   with
   'bin/solr zk upconfig -n karpfen -d 
/home/pberg/solr_new/solr-6.5.1/server/solr/tommy/conf -z localhost:2181'
   --> no errors
4. Started solr in cloud mode with
  'bin/solr -c -z localhost:2181'
5. Tried to create a new collection with 2 shards with
   'bin/solr create -c karpfen -shards 2'

The output is:

Connecting to ZooKeeper at localhost:2181 ...
INFO  - 2017-05-12 18:52:22.807; 
org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at 
localhost:2181 ready
Re-using existing configuration directory karpfen

Creating new collection 'karpfen' using command:
http://localhost:8983/solr/admin/collections?action=CREATE=karpfen=2=1=2=karpfen


ERROR: Failed to create collection 'karpfen' due to: 
{127.0.1.1:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
 from server at http://127.0.1.1:8983/solr: Error CREATEing SolrCore 
'karpfen_shard2_replica1': Unable to create core [karpfen_shard2_replica1] 
Caused by: Lock held by this virtual machine: 
/home/pberg/solr_new2/solr-6.5.1/server/data/ohrdruf_bestand/index/write.lock}

   
The conf directory I copied contains the following files:
currency.xml elevate.xml  protwords.txt   stopwords.txt
dataimport-cobt2.properties  lang schema.xml  synonyms.txt
dataimport.xml   params.json  solrconfig.xml

"lang" is a directory.

Are my steps wrong? Did I miss something important? 

Any help is really welcome.

Thomas

solr LTR edismax features very slow and cause timeouts

[lekha_dipika]

I am trying to use the SOLR LTR plugin to train a machine learning model for
my application. One of the features I require for my model is edismax score
of my query over a set of index fields. 

The way I have created the feature is as follows:

{  "store":"sampleFeatureStore",
   "name" : "edismax_feature",
   "class" : "org.apache.solr.ltr.feature.SolrFeature",
   "params" : { "q" : "{!edismax qf=\"field1 field2 field3 field4\"
}${user_query}" }
 }


field1 ... field4 are text fields with some analyzers performing
tokenization, lowercasing etc. 

While this feature gives me the edismax scores I desire, however, on load
testing my entire setup, presence of these features increased latency
conderably and I could see timeouts in the logs for many queries.  (The
request took too long to iterate over terms) I also saw my CPU usage and JVM
memory use increase.

Without this feature things were stable and load testing was fine. 

I have 2 collections in my solr setup with 25M and 2M docs respectively.
Load testing involved sending 200 queries per minute.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-LTR-edismax-features-very-slow-and-cause-timeouts-tp4335059.html
Sent from the Solr - User mailing list archive at Nabble.com.