Re: Arabic analyser

[Alexandre Rafalovitch]

If this is for a significant project and you are ready to pay for it,
BasisTech has commercial solutions in this area I believe.

Regards,
   Alex.

Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/


On 10 November 2015 at 08:46, Mahmoud Almokadem  wrote:
> Thanks Pual,
>
> Arabic analyser applying filters of normalisation and stemming only for
> single terms out of standard tokenzier.
> Gathering all synonyms will be hard work. Should I customise my Tokenizer
> to handle this case?
>
> Sincerely,
> Mahmoud
>
>
> On Tue, Nov 10, 2015 at 3:06 PM, Paul Libbrecht  wrote:
>
>> Mahmoud,
>>
>> there is an arabic analyzer:
>>   https://wiki.apache.org/solr/LanguageAnalysis#Arabic
>> doesn't it do what you describe?
>> Synonyms probably work there too.
>>
>> Paul
>>
>> > Mahmoud Almokadem 
>> > 9 novembre 2015 17:47
>> > Thanks Jack,
>> >
>> > This is a good solution, but we have more combinations that I think
>> > can’t be handled as synonyms like every word starts with ‘عبد’ ‘Abd’
>> > and ‘أبو’ ‘Abo’. When using Standard tokenizer on ‘أبو بكر’ ‘Abo
>> > Bakr’, It’ll be tokenised to ‘أبو’ and ‘بكر’ and the filters will be
>> > applied for each separate term.
>> >
>> > Is there available tokeniser to tokenise ‘أبو *’ or ‘عبد *' as a
>> > single term?
>> >
>> > Thanks,
>> > Mahmoud
>> >
>> >
>> >
>> > Jack Krupansky 
>> > 9 novembre 2015 16:47
>> > Use an index-time (but not query time) synonym filter with a rule like:
>> >
>> > Abd Allah,Abdallah
>> >
>> > This will index the combined word in addition to the separate words.
>> >
>> > -- Jack Krupansky
>> >
>> > On Mon, Nov 9, 2015 at 4:48 AM, Mahmoud Almokadem <
>> prog.mahm...@gmail.com>
>> >
>> > Mahmoud Almokadem 
>> > 9 novembre 2015 10:48
>> > Hello,
>> >
>> > We are indexing Arabic content and facing a problem for tokenizing multi
>> > terms phrases like 'عبد الله' 'Abd Allah', so users will search for
>> > 'عبدالله' 'Abdallah' without space and need to get the results of 'عبد
>> > الله' with space. We are using StandardTokenizer.
>> >
>> >
>> > Is there any configurations to handle this case?
>> >
>> > Thank you,
>> > Mahmoud
>> >
>>
>>

Re: Simple web interface for queries

[Alexandre Rafalovitch]

Solr is not actually designed to be directly exposed to the end-users.
It is possible to delete the whole collection,etc. It is supposed to
be treated as a database behind firewall, etc.

Just thought I'll mention that in case you did not know it.

Regards,
   Alex.

Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/


On 10 November 2015 at 10:14, Nigel Robbins
 wrote:
> Hi,
>
>
>
> Just wondering if there is a simplified web user interface for users to
> issue just queries.
>
>
>
> The http://localhost:8984/solr/#/core/query URL has too many options for my
> end-users.
>
>
>
> For example, can the web interface be configured for queries so that only
> the “query string” and output can be seen ?
>
>
>
> Thanks,
>
> Nigel

Re: The time that init.d script waits before shutdown should be configurable

[Yago Riveiro]

Patch attached to https://issues.apache.org/jira/browse/SOLR-8065





The windows script is voodo for me :D, I haven’t the knowledge to port this to 
cmd script.


—/Yago Riveiro

On Mon, Nov 9, 2015 at 3:23 PM, Upayavira  wrote:

> Yago,
> I think a JIRA has been raised for this. I'd encourage you to hunt it
> down and make a patch.
> Upayavira
> On Mon, Nov 9, 2015, at 03:09 PM, Yago Riveiro wrote:
>> The time that init.d script waits before shutdown should be configurable
>> 
>> The 5 seconds is not enough to all my shards notify the shutdown and the
>> process ends with a kill command
>> 
>> I think that in solr.in.sh should exists a entry to configure the time to
>> wait before use a kill command
>> 
>> 
>> 
>> -
>> Best regards
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/The-time-that-init-d-script-waits-before-shutdown-should-be-configurable-tp4239143.html
>> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Simple web interface for queries

[davidphilip cherian]

How about velocity search UI? You can access that using /browse

https://cwiki.apache.org/confluence/display/solr/Velocity+Search+UI




On Tue, Nov 10, 2015 at 10:14 AM, Nigel Robbins <
nigel.robb...@wallstreetsystems.com> wrote:

> Hi,
>
>
>
> Just wondering if there is a simplified web user interface for users to
> issue just queries.
>
>
>
> The http://localhost:8984/solr/#/core/query URL has too many options for
> my
> end-users.
>
>
>
> For example, can the web interface be configured for queries so that only
> the “query string” and output can be seen ?
>
>
>
> Thanks,
>
> Nigel
>

Re: Security Problems

[Shawn Heisey]

On 11/9/2015 6:38 PM, 马柏樟 wrote:
> After I configure Authentication with Basic Authentication Plugin and 
> Authorization with Rule-Based Authorization Plugin, How can I prevent the 
> strangers from visiting my solr by browser? For example, if the stranger 
> visit the http://(my host):8983, the browser will pop up a window and says 
> "the server http://(my host):8983 requires a username and password"

Don't expose Solr directly to the Internet or other end users, even with
authentication.  It's just a bad idea.  The best security you can
provide for Solr is to make sure it is only accessible by your client
applications and trusted administrators.  If you do this, you won't need
any more security.

With proper configuration, a proxy server can harden Solr so it is less
open to attack, but it would likely still be vulnerable to denial of
service queries.  A proper proxy configuration would not be a trivial task.

The authentication that Solr 5.3 provides will only protect the API
endpoints that Solr exposes.  It will *not* block access to the actual
admin UI itself -- that is nothing but nearly static HTML, images, CSS,
and javascript.  The authentication does protect Solr, because the admin
UI cannot do anything to Solr without accessing the API endpoints, which
would need authentication.

Thanks,
Shawn

Simple web interface for queries

[Nigel Robbins]

Hi,



Just wondering if there is a simplified web user interface for users to
issue just queries.



The http://localhost:8984/solr/#/core/query URL has too many options for my
end-users.



For example, can the web interface be configured for queries so that only
the “query string” and output can be seen ?



Thanks,

Nigel

Re: Stopping Solr on Linux when run as a service

[Scott Stults]

Steve,

In short, don't worry: it all gets taken care of.

The way services work on Linux is, when the system shuts down it will
basically call "service (servicname) stop" on each service. That calls the
bin/init.d/solr script with a "stop" argument, which in turn calls the
bin/solr script with a "stop" argument (I'm referring to where the files
are in the distribution, not where they get installed).

k/r,
Scott


On Tue, Nov 10, 2015 at 9:40 AM, Steven White  wrote:

> Hi folks,
>
> This question maybe more of a Linux one vs. Solr, but I have to start
> someplace.
>
> I'm reading this link
> https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production
> to get Solr on Linux (I'm more of a Windows guy).
>
> The page provides good intro on how to setup Solr to start as a service on
> Linux.  Now what I don't get is this: what happens when the system is
> shutting down?  How does Solr knows to shutdown gracefully when there is
> noting on that page talks about issuing a "stop" command on system
> shutdown?  Can someone shed some light on this?  Like I said, I'm more of a
> "Windows" guy.
>
> Thanks in advanced!!
>
> Steve
>



-- 
Scott Stults | Founder & Solutions Architect | OpenSource Connections, LLC
| 434.409.2780
http://www.opensourceconnections.com

Re: Stopping Solr on Linux when run as a service

[Shawn Heisey]

On 11/10/2015 7:40 AM, Steven White wrote:
> This question maybe more of a Linux one vs. Solr, but I have to start
> someplace.
> 
> I'm reading this link
> https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production
> to get Solr on Linux (I'm more of a Windows guy).
> 
> The page provides good intro on how to setup Solr to start as a service on
> Linux.  Now what I don't get is this: what happens when the system is
> shutting down?  How does Solr knows to shutdown gracefully when there is
> noting on that page talks about issuing a "stop" command on system
> shutdown?  Can someone shed some light on this?  Like I said, I'm more of a
> "Windows" guy.

This definitely applies more to Linux than Solr.  Understanding the
answer requires understanding basic Linux system design and administration.

Linux systems use something, usually sysvinit, systemd, or upstart, to
manage service startup and shutdown.  I will not go into detail, but one
of the things that these programs do is manage services via symlinks
into the /etc/init.d structure -- starting them at boot time and
shutting them down when the OS is shutting down.

When you use the Solr installer script, you get a script in /etc/init.d
and symlinks for startup and shutdown, so the system has the ability to
ask Solr to turn itself off.

If programs do not shut themselves down fast enough at shutdown time,
the operating system will attempt to hard kill them before it shuts
itself off ... but even if they don't die, Linux can still forcibly halt
itself and turn off the power.

Thanks,
Shawn

Re: Solr Search: Access Control / Role based security

[Anshum Gupta]

I think both of those overlap at some point but aren't really directly
related or problems that would be solved in the same manner.

Document level security, though can be implemented using custom
authentication/authorization plugins, but there are a fair amount of users
who use ManifoldCF for the same. So it's totally your pick.

I'm not 100% sure, but I think using a custom authentication/authorization
plugin + an update request processor is more work than using ManifoldCF for
that purpose.

On Tue, Nov 10, 2015 at 10:37 AM, Susheel Kumar 
wrote:

> Thanks everyone for the suggestions.
>
> Hi Noble - Were there any thoughts made on utilizing Apache ManifoldCF
> while developing Authentication/Authorization plugins or anything to add
> there.
>
> Thanks,
> Susheel
>
> On Tue, Nov 10, 2015 at 5:01 AM, Alessandro Benedetti <
> abenede...@apache.org
> > wrote:
>
> > I've been working for a while with Apache ManifoldCF and Enterprise
> Search
> > in Solr ( with Document level security) .
> > Basically you can add a couple of extra fields , for example :
> >
> > allow_token : containing all the tokens that can view the document
> > deny_token : containing all the tokens that are denied to view the
> document
> >
> > Apache ManifoldCF provides an integration that add an additional layer,
> and
> > is able to combine different data sources permission schemes.
> > The Authority Service endpoint will take in input the user name and
> return
> > all the allow_token values and deny_token.
> > At this point you can append the related filter queries to your queries
> and
> > be sure that the user will only see what is supposed to see.
> >
> > It's basically an extension of the strategy you were proposing, role
> based.
> > Of course keep protected your endpoints and avoid users to put custom fq,
> > or all your document security model would be useless :)
> >
> > Cheers
> >
> >
> > On 9 November 2015 at 21:52, Scott Stults <
> > sstu...@opensourceconnections.com
> > > wrote:
> >
> > > Susheel,
> > >
> > > This is perfectly fine for simple use-cases and has the benefit that
> the
> > > filterCache will help things stay nice and speedy. Apache ManifoldCF
> > goes a
> > > bit further and ties back to your authentication and authorization
> > > mechanism:
> > >
> > >
> > >
> >
> http://manifoldcf.apache.org/release/trunk/en_US/concepts.html#ManifoldCF+security+model
> > >
> > >
> > > k/r,
> > > Scott
> > >
> > > On Thu, Nov 5, 2015 at 2:26 PM, Susheel Kumar 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I have seen couple of use cases / need where we want to restrict
> result
> > > of
> > > > search based on role of a user.  For e.g.
> > > >
> > > > - if user role is admin, any document from the search result will be
> > > > returned
> > > > - if user role is manager, only documents intended for managers will
> be
> > > > returned
> > > > - if user role is worker, only documents intended for workers will be
> > > > returned
> > > >
> > > > Typical practise is to tag the documents with the roles (using a
> > > > multi-valued field) during indexing and then during search append
> > filter
> > > > query to restrict result based on roles.
> > > >
> > > > Wondering if there is any other better way out there and if this
> common
> > > > requirement should be added as a Solr feature/plugin.
> > > >
> > > > The current security plugins are more towards making Solr
> > apis/resources
> > > > secure not towards securing/controlling data during search.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
> > > >
> > > >
> > > > Please share your thoughts.
> > > >
> > > > Thanks,
> > > > Susheel
> > > >
> > >
> > >
> > >
> > > --
> > > Scott Stults | Founder & Solutions Architect | OpenSource Connections,
> > LLC
> > > | 434.409.2780
> > > http://www.opensourceconnections.com
> > >
> >
> >
> >
> > --
> > --
> >
> > Benedetti Alessandro
> > Visiting card : http://about.me/alessandro_benedetti
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
> >
>



-- 
Anshum Gupta

Re: Search query speed

[Gopal Patwa]

You could start adding debug=true in your request, it will show complete
query execution time

On Tue, Nov 10, 2015 at 9:39 AM John Stric  wrote:

> The speed of particular query has gone from about 42 msec to 66 msec
> without any changes.
>
> How do I go about troubleshooting what may have happened? And how do I
> improve that speed?
>
> Thanks
>

Re: Solr results relevancy / scoring

[Brian Narsi]

Yes the term was getting tokenized as 15. And Norms are not ignored.

Thank you for helping understand the issue.

On Mon, Nov 9, 2015 at 6:07 AM, Alessandro Benedetti 
wrote:

>  I quote Emir and I would like to ask if the Norms are ignored or not.
> If they are not ignored and 15 is one of the search tokens, I can expect an
> high score for a doc containing "15" because the Norm value will be quite
> high ( as the field contains basically exactly the query term).
>
> Cheers
>
> On 9 November 2015 at 10:01, Emir Arnautovic  >
> wrote:
>
> > To get answer for why 15, you can use field analysis for index/query and
> > see that "15%" is probably tokenized and as both 15 and 15%.
> >
> > Emir
> >
> >
> > On 06.11.2015 20:22, Erick Erickson wrote:
> >
> >> I'm not sure what the question your asking is. You say
> >> that you have debugged the query and the score for 15 is
> >> higher than the ones below it. What's surprising about that?
> >>
> >> Are you saying you don't understand how the score is
> >> calculated? Or the output when adding =true
> >> is inconsistent or what?
> >>
> >> Best,
> >> Erick
> >>
> >> On Fri, Nov 6, 2015 at 11:04 AM, Brian Narsi 
> wrote:
> >>
> >>> I have a situation where.
> >>>
> >>> User search query
> >>>
> >>> q=15%
> >>>
> >>> Solr results contain several documents that are
> >>>
> >>> 15%
> >>> 15%
> >>> 15%
> >>> 15%
> >>> 15 (why?)
> >>> 15%
> >>> 15%
> >>>
> >>> I have debugged the query and can see that the score for 15 is higher
> >>> than
> >>> the ones below it.
> >>>
> >>> Why is that? Where can I read in detail about how the scoring is being
> >>> done?
> >>>
> >>> Thanks
> >>>
> >>
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
>
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

Search query speed

[John Stric]

The speed of particular query has gone from about 42 msec to 66 msec
without any changes.

How do I go about troubleshooting what may have happened? And how do I
improve that speed?

Thanks

Re: Security Problems

[Upayavira]

Noble,

I get that a UI which is open source does not benefit from ACL control -
we're not giving away anything that isn't public (other than perhaps
info that could be used to identify the version of Solr, or even the
fact that it *is* solr).

However, from a user experience point of view, requiring credentials to
see the UI would be more conventional, and therefore lead to less
confusion. Is it possible for us to protect the UI static files, only
for the sake of user experience, rather than security?

Upayavira

On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
> The admin UI is a bunch of static pages . We don't let the ACL control
> static content
> 
> you must blacklist all the core/collection apis and it is pretty much
> useless for anyone to access the admin UI (w/o the credentials , of
> course)
> 
> On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟  wrote:
> > Hi,
> >
> > After I configure Authentication with Basic Authentication Plugin and 
> > Authorization with Rule-Based Authorization Plugin, How can I prevent the 
> > strangers from visiting my solr by browser? For example, if the stranger 
> > visit the http://(my host):8983, the browser will pop up a window and says 
> > "the server http://(my host):8983 requires a username and password"
> 
> 
> 
> -- 
> -
> Noble Paul

Re: Solr Search: Access Control / Role based security

[Susheel Kumar]

Thanks everyone for the suggestions.

Hi Noble - Were there any thoughts made on utilizing Apache ManifoldCF
while developing Authentication/Authorization plugins or anything to add
there.

Thanks,
Susheel

On Tue, Nov 10, 2015 at 5:01 AM, Alessandro Benedetti  wrote:

> I've been working for a while with Apache ManifoldCF and Enterprise Search
> in Solr ( with Document level security) .
> Basically you can add a couple of extra fields , for example :
>
> allow_token : containing all the tokens that can view the document
> deny_token : containing all the tokens that are denied to view the document
>
> Apache ManifoldCF provides an integration that add an additional layer, and
> is able to combine different data sources permission schemes.
> The Authority Service endpoint will take in input the user name and return
> all the allow_token values and deny_token.
> At this point you can append the related filter queries to your queries and
> be sure that the user will only see what is supposed to see.
>
> It's basically an extension of the strategy you were proposing, role based.
> Of course keep protected your endpoints and avoid users to put custom fq,
> or all your document security model would be useless :)
>
> Cheers
>
>
> On 9 November 2015 at 21:52, Scott Stults <
> sstu...@opensourceconnections.com
> > wrote:
>
> > Susheel,
> >
> > This is perfectly fine for simple use-cases and has the benefit that the
> > filterCache will help things stay nice and speedy. Apache ManifoldCF
> goes a
> > bit further and ties back to your authentication and authorization
> > mechanism:
> >
> >
> >
> http://manifoldcf.apache.org/release/trunk/en_US/concepts.html#ManifoldCF+security+model
> >
> >
> > k/r,
> > Scott
> >
> > On Thu, Nov 5, 2015 at 2:26 PM, Susheel Kumar 
> > wrote:
> >
> > > Hi,
> > >
> > > I have seen couple of use cases / need where we want to restrict result
> > of
> > > search based on role of a user.  For e.g.
> > >
> > > - if user role is admin, any document from the search result will be
> > > returned
> > > - if user role is manager, only documents intended for managers will be
> > > returned
> > > - if user role is worker, only documents intended for workers will be
> > > returned
> > >
> > > Typical practise is to tag the documents with the roles (using a
> > > multi-valued field) during indexing and then during search append
> filter
> > > query to restrict result based on roles.
> > >
> > > Wondering if there is any other better way out there and if this common
> > > requirement should be added as a Solr feature/plugin.
> > >
> > > The current security plugins are more towards making Solr
> apis/resources
> > > secure not towards securing/controlling data during search.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
> > >
> > >
> > > Please share your thoughts.
> > >
> > > Thanks,
> > > Susheel
> > >
> >
> >
> >
> > --
> > Scott Stults | Founder & Solutions Architect | OpenSource Connections,
> LLC
> > | 434.409.2780
> > http://www.opensourceconnections.com
> >
>
>
>
> --
> --
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

Re: Security Problems

[Anshum Gupta]

The reason why we bypass that is so that we don't hit the authentication
plugin for every request that comes in for static content. I think we could
call the authentication plugin for that but that'd be an overkill. Better
experience ? yes

On Tue, Nov 10, 2015 at 11:24 AM, Upayavira  wrote:

> Noble,
>
> I get that a UI which is open source does not benefit from ACL control -
> we're not giving away anything that isn't public (other than perhaps
> info that could be used to identify the version of Solr, or even the
> fact that it *is* solr).
>
> However, from a user experience point of view, requiring credentials to
> see the UI would be more conventional, and therefore lead to less
> confusion. Is it possible for us to protect the UI static files, only
> for the sake of user experience, rather than security?
>
> Upayavira
>
> On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
> > The admin UI is a bunch of static pages . We don't let the ACL control
> > static content
> >
> > you must blacklist all the core/collection apis and it is pretty much
> > useless for anyone to access the admin UI (w/o the credentials , of
> > course)
> >
> > On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟  wrote:
> > > Hi,
> > >
> > > After I configure Authentication with Basic Authentication Plugin and
> Authorization with Rule-Based Authorization Plugin, How can I prevent the
> strangers from visiting my solr by browser? For example, if the stranger
> visit the http://(my host):8983, the browser will pop up a window and
> says "the server http://(my host):8983 requires a username and
> password"
> >
> >
> >
> > --
> > -
> > Noble Paul
>



-- 
Anshum Gupta

Re: Security Problems

[Anshum Gupta]

It has a cost :)

I think it'd make sense to restrict access to /admin and not really bother
about .css/js etc. So if a user tries to access an image from the image
from the admin UI directly, the request would go through but that should be
fine.

On Tue, Nov 10, 2015 at 12:22 PM, Upayavira  wrote:

> Is the authentication plugin that expensive?
>
> I can help by minifying the UI down to a smaller number of CSS/JS/etc
> files :-)
>
> It may be overkill, but it would also give better experience. And isn't
> that what most applications do? Check authentication tokens on every
> request?
>
> Upayavira
>
> On Tue, Nov 10, 2015, at 07:33 PM, Anshum Gupta wrote:
> > The reason why we bypass that is so that we don't hit the authentication
> > plugin for every request that comes in for static content. I think we
> > could
> > call the authentication plugin for that but that'd be an overkill. Better
> > experience ? yes
> >
> > On Tue, Nov 10, 2015 at 11:24 AM, Upayavira  wrote:
> >
> > > Noble,
> > >
> > > I get that a UI which is open source does not benefit from ACL control
> -
> > > we're not giving away anything that isn't public (other than perhaps
> > > info that could be used to identify the version of Solr, or even the
> > > fact that it *is* solr).
> > >
> > > However, from a user experience point of view, requiring credentials to
> > > see the UI would be more conventional, and therefore lead to less
> > > confusion. Is it possible for us to protect the UI static files, only
> > > for the sake of user experience, rather than security?
> > >
> > > Upayavira
> > >
> > > On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
> > > > The admin UI is a bunch of static pages . We don't let the ACL
> control
> > > > static content
> > > >
> > > > you must blacklist all the core/collection apis and it is pretty much
> > > > useless for anyone to access the admin UI (w/o the credentials , of
> > > > course)
> > > >
> > > > On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟  wrote:
> > > > > Hi,
> > > > >
> > > > > After I configure Authentication with Basic Authentication Plugin
> and
> > > Authorization with Rule-Based Authorization Plugin, How can I prevent
> the
> > > strangers from visiting my solr by browser? For example, if the
> stranger
> > > visit the http://(my host):8983, the browser will pop up a window and
> > > says "the server http://(my host):8983 requires a username and
> > > password"
> > > >
> > > >
> > > >
> > > > --
> > > > -
> > > > Noble Paul
> > >
> >
> >
> >
> > --
> > Anshum Gupta
>



-- 
Anshum Gupta

Re: Security Problems

[Erik Hatcher]

Front it with request handler(s) and get security for free :) (top level not 
cite specific it'd have to be)

> On Nov 10, 2015, at 14:24, Upayavira  wrote:
> 
> Noble,
> 
> I get that a UI which is open source does not benefit from ACL control -
> we're not giving away anything that isn't public (other than perhaps
> info that could be used to identify the version of Solr, or even the
> fact that it *is* solr).
> 
> However, from a user experience point of view, requiring credentials to
> see the UI would be more conventional, and therefore lead to less
> confusion. Is it possible for us to protect the UI static files, only
> for the sake of user experience, rather than security?
> 
> Upayavira
> 
>> On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
>> The admin UI is a bunch of static pages . We don't let the ACL control
>> static content
>> 
>> you must blacklist all the core/collection apis and it is pretty much
>> useless for anyone to access the admin UI (w/o the credentials , of
>> course)
>> 
>>> On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟  wrote:
>>> Hi,
>>> 
>>> After I configure Authentication with Basic Authentication Plugin and 
>>> Authorization with Rule-Based Authorization Plugin, How can I prevent the 
>>> strangers from visiting my solr by browser? For example, if the stranger 
>>> visit the http://(my host):8983, the browser will pop up a window and says 
>>> "the server http://(my host):8983 requires a username and password"
>> 
>> 
>> 
>> -- 
>> -
>> Noble Paul

Re: Search query speed

[Toke Eskildsen]

John Stric  wrote:
> The speed of particular query has gone from about 42 msec to 66 msec
> without any changes.

- Your JVM has allocated a bit more RAM, leaving less for disk cache.
- Your index has grown.
- The amount of concurrent requests has increased, pushing objects from Eden 
space to main heap, causing heavier garbage collection.
- A backup job is running.
- Another program has started, eating free mem from disk cache.
- Swap activity has increased.

The list goes on. Java performance is a fickle beast and Solr performance 
doubly so.

> How do I go about troubleshooting what may have happened?

Try to restore to a previous state and see if that changes response time: Shut 
down all running programs on the machine, start Solr again, warm it and test?

- Toke Eskildsen

Re: Security Problems

[Upayavira]

Is the authentication plugin that expensive?

I can help by minifying the UI down to a smaller number of CSS/JS/etc
files :-)

It may be overkill, but it would also give better experience. And isn't
that what most applications do? Check authentication tokens on every
request?

Upayavira

On Tue, Nov 10, 2015, at 07:33 PM, Anshum Gupta wrote:
> The reason why we bypass that is so that we don't hit the authentication
> plugin for every request that comes in for static content. I think we
> could
> call the authentication plugin for that but that'd be an overkill. Better
> experience ? yes
> 
> On Tue, Nov 10, 2015 at 11:24 AM, Upayavira  wrote:
> 
> > Noble,
> >
> > I get that a UI which is open source does not benefit from ACL control -
> > we're not giving away anything that isn't public (other than perhaps
> > info that could be used to identify the version of Solr, or even the
> > fact that it *is* solr).
> >
> > However, from a user experience point of view, requiring credentials to
> > see the UI would be more conventional, and therefore lead to less
> > confusion. Is it possible for us to protect the UI static files, only
> > for the sake of user experience, rather than security?
> >
> > Upayavira
> >
> > On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
> > > The admin UI is a bunch of static pages . We don't let the ACL control
> > > static content
> > >
> > > you must blacklist all the core/collection apis and it is pretty much
> > > useless for anyone to access the admin UI (w/o the credentials , of
> > > course)
> > >
> > > On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟  wrote:
> > > > Hi,
> > > >
> > > > After I configure Authentication with Basic Authentication Plugin and
> > Authorization with Rule-Based Authorization Plugin, How can I prevent the
> > strangers from visiting my solr by browser? For example, if the stranger
> > visit the http://(my host):8983, the browser will pop up a window and
> > says "the server http://(my host):8983 requires a username and
> > password"
> > >
> > >
> > >
> > > --
> > > -
> > > Noble Paul
> >
> 
> 
> 
> -- 
> Anshum Gupta

Re: fl=value equals?

[Jan Høydahl]

Where is your “b” parameter used? I think that instead of trying to set a new 
“b” http param (which solr will not evaluate as a function), you should instead 
try to insert your function or switch qParser directly where the “b” param is 
used, e.g. in a bq or similar.

A bit heavy weight, but you could of course write a custom SearchComponent to 
construct your “b” parameter...

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 10. nov. 2015 kl. 23.52 skrev William Bell :
> 
> We are trying to look at a value, and change another value based on that.
> 
> For example, for national search we want to pass in radius=national, and
> then set another variable equal to 0, else set the other variable = to
> geodist() calculation.
> 
> We tried {!switch} but this only appears to work on fq/q. There is no
> function for constants for equals
> 
> http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,'national'),0,geodist())
> 
> This does not work:
> 
> http://localhost:8983/solr/select?q=*:*=national={!switch
> case.national=0 default=geodist() v=$radius}
> 
> Ideas?
> 
> 
> 
> -- 
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076

Re: Convert output response xml into input xml format using xslt

[Jan Høydahl]

It already ships with Solr. 
http://localhost:8983/solr/select?q=*:*=xslt=updateXml

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 9. nov. 2015 kl. 19.27 skrev davidphilip cherian 
> :
> 
> Has anyone written a sample xslt (and would like to share) that converts
> output response xml of solr into its  input format, to
> repost/reindex it back?
> 
> Thanks

Re: Security Problems

[Upayavira]

Or, rather than touch Jetty, you could simply use iptables or such
firewall as is provided by your operating system.

Upayavira

On Tue, Nov 10, 2015, at 08:08 AM, Vijay Mhaskar - 2 wrote:
> Or you can try configuring IP based access control mechanism using
> IPAccessHandler in jetty. I have configured this for our SolrCloud setup
> and
> it works very well. While configuring it for SolrCloud we need to be more
> careful because Solr has lot of inter node communication happening during
> distributed searching or indexing. 
> 
> 
> 
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Security-Problems-tp4239302p4239318.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Security Problems

[Vijay Mhaskar - 2]

Or you can try configuring IP based access control mechanism using
IPAccessHandler in jetty. I have configured this for our SolrCloud setup and
it works very well. While configuring it for SolrCloud we need to be more
careful because Solr has lot of inter node communication happening during
distributed searching or indexing. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Security-Problems-tp4239302p4239318.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Suggester with Geo?

[Mikhail Khludnev]

Hello,

It seems it's nearly possible, but needs to be done
https://github.com/apache/lucene-solr/blob/trunk/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java#L576
(this is for AnalyzingInfixLookupFactory)

And for FuzzySuggester such implementation is much more complicated.


On Mon, Nov 9, 2015 at 10:18 PM, William Bell  wrote:

> http://lucidworks.com/blog/solr-suggester/
>
>
> Wondering if anyone has uses these new techniques with a boost on
> geodist() inverted? So the rows that get returned that are closest
> need to come back first.
>
>
> We are still using Edge Grams since we have not figured out how to
> boost the results on geo spatial.
>
>
> Anyone have thoughts?
>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

Re: Solr Suggester with Geo?

[Alessandro Benedetti]

Out of the box the Suggester componenent provides a set of Lookup
implementation to look for the suggestions from the dictionary selected.
Most of the suggestion lookups are FST based ( except the Analysing Infix
one , that relies on an external auxiliary Lucene Index) .

Depending on your requirement you should customise the lookUp
implementation to score the suggestions taking in consideration the
distance.
Customising the "Auxiliary Lucene Index " based look up strategies would be
easier.
Playing with the FST look up would be much more complicated.

Cheers

On 9 November 2015 at 20:16, Sameer Maggon 
wrote:

> Looking through the code and some example Suggesters, it seems that
> theoretically, one can write a GeoSuggester and provide that as the Lookup
> implementation (lookupimpl) that would factor in the geo score or extend
> the SolrSuggestor to support spatial extensions in the same spirit as
> "Filters" are supported today.
>
> Sameer.
>
> On Mon, Nov 9, 2015 at 11:47 AM, William Bell  wrote:
>
> > Yeah we have that working today. But the issue is we want to use
> > http://lucidworks.com/blog/solr-suggester/
> >
> > And you cannot do a boost with that right?
> >
> >
> >
> > On Mon, Nov 9, 2015 at 12:41 PM, Sameer Maggon <
> sam...@measuredsearch.com>
> > wrote:
> >
> > > Have you looked at the Spatial extensions for Solr? If you are indexing
> > > Lat/Lon along with your documents, you can compute the distance from
> the
> > > origin & use that distance as one of the boost factors to affect the
> > score.
> > > Typically, use cases around that combine the geo score with other
> factors
> > > as a pure sort by geo score might not give you the relevant results.
> > >
> > > e.g. typing to search for "sushi restaurants" near Santa Monica, CA -
> you
> > > might not want "thai restaurants" that are closest to you. (Local
> Search
> > > use case)
> > >
> > > https://cwiki.apache.org/confluence/display/solr/Spatial+Search
> > >
> > > Thanks,
> > > --
> > > *Sameer Maggon*
> > > www.measuredsearch.com 
> > > Fully Managed Solr-as-a-Service | Solr Consulting | Solr Support
> > >
> > >
> > >
> > > On Mon, Nov 9, 2015 at 11:18 AM, William Bell 
> > wrote:
> > >
> > > > http://lucidworks.com/blog/solr-suggester/
> > > >
> > > >
> > > > Wondering if anyone has uses these new techniques with a boost on
> > > > geodist() inverted? So the rows that get returned that are closest
> > > > need to come back first.
> > > >
> > > >
> > > > We are still using Edge Grams since we have not figured out how to
> > > > boost the results on geo spatial.
> > > >
> > > >
> > > > Anyone have thoughts?
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Bill Bell
> > > > billnb...@gmail.com
> > > > cell 720-256-8076
> > > >
> > >
> >
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Solr Search: Access Control / Role based security

[Alessandro Benedetti]

I've been working for a while with Apache ManifoldCF and Enterprise Search
in Solr ( with Document level security) .
Basically you can add a couple of extra fields , for example :

allow_token : containing all the tokens that can view the document
deny_token : containing all the tokens that are denied to view the document

Apache ManifoldCF provides an integration that add an additional layer, and
is able to combine different data sources permission schemes.
The Authority Service endpoint will take in input the user name and return
all the allow_token values and deny_token.
At this point you can append the related filter queries to your queries and
be sure that the user will only see what is supposed to see.

It's basically an extension of the strategy you were proposing, role based.
Of course keep protected your endpoints and avoid users to put custom fq,
or all your document security model would be useless :)

Cheers


On 9 November 2015 at 21:52, Scott Stults  wrote:

> Susheel,
>
> This is perfectly fine for simple use-cases and has the benefit that the
> filterCache will help things stay nice and speedy. Apache ManifoldCF goes a
> bit further and ties back to your authentication and authorization
> mechanism:
>
>
> http://manifoldcf.apache.org/release/trunk/en_US/concepts.html#ManifoldCF+security+model
>
>
> k/r,
> Scott
>
> On Thu, Nov 5, 2015 at 2:26 PM, Susheel Kumar 
> wrote:
>
> > Hi,
> >
> > I have seen couple of use cases / need where we want to restrict result
> of
> > search based on role of a user.  For e.g.
> >
> > - if user role is admin, any document from the search result will be
> > returned
> > - if user role is manager, only documents intended for managers will be
> > returned
> > - if user role is worker, only documents intended for workers will be
> > returned
> >
> > Typical practise is to tag the documents with the roles (using a
> > multi-valued field) during indexing and then during search append filter
> > query to restrict result based on roles.
> >
> > Wondering if there is any other better way out there and if this common
> > requirement should be added as a Solr feature/plugin.
> >
> > The current security plugins are more towards making Solr apis/resources
> > secure not towards securing/controlling data during search.
> >
> >
> https://cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
> >
> >
> > Please share your thoughts.
> >
> > Thanks,
> > Susheel
> >
>
>
>
> --
> Scott Stults | Founder & Solutions Architect | OpenSource Connections, LLC
> | 434.409.2780
> http://www.opensourceconnections.com
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Security Problems

[Noble Paul]

The admin UI is a bunch of static pages . We don't let the ACL control
static content

you must blacklist all the core/collection apis and it is pretty much
useless for anyone to access the admin UI (w/o the credentials , of
course)

On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟  wrote:
> Hi,
>
> After I configure Authentication with Basic Authentication Plugin and 
> Authorization with Rule-Based Authorization Plugin, How can I prevent the 
> strangers from visiting my solr by browser? For example, if the stranger 
> visit the http://(my host):8983, the browser will pop up a window and says 
> "the server http://(my host):8983 requires a username and password"



-- 
-
Noble Paul

Re: The time that init.d script waits before shutdown should be configurable

[Ere Maijala]

If you're still going the shell script way, I'd suggest incorporating my 
changes (patch attached to the issue). It allows waiting for a longer 
time but only as long as necessary (like it already does during startup).


--Ere

11.11.2015, 0.01, Upayavira kirjoitti:



On Tue, Nov 10, 2015, at 04:22 PM, Yago Riveiro wrote:

Patch attached to https://issues.apache.org/jira/browse/SOLR-8065





The windows script is voodo for me :D, I haven’t the knowledge to port
this to cmd script.


Great! I saw this!

Two things - firstly, when making a patch, please try to avoid
whitespace changes. The only changes that should show in the diff should
be material changes.

Secondly - I think there was a suggestion that this change could be
ported *inside* the SolrCLI - i.e. into Java code. Do you reckon you
could handle that change? Harder than just updating a shell script, I
know, but could be very useful.

Upayavira



--
Ere Maijala
Kansalliskirjasto / The National Library of Finland

Need help to create Solr committer account / email address.

[ANUP DIXIT]

Hi Solr Team,

Please help me or let me know process, how can I create an Solr committer
account / email address.
I did read instruction at https://reference.apache.org/committer/email, but
not able to get this right or configured it correctly.

Have a good day !!!


*Thanks & Regards, **Anup Dixit*

*"Print this mail only if absolutely necessary. Save Paper. Save Trees."*

Stopping Solr on Linux when run as a service

[Steven White]

Hi folks,

This question maybe more of a Linux one vs. Solr, but I have to start
someplace.

I'm reading this link
https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production
to get Solr on Linux (I'm more of a Windows guy).

The page provides good intro on how to setup Solr to start as a service on
Linux.  Now what I don't get is this: what happens when the system is
shutting down?  How does Solr knows to shutdown gracefully when there is
noting on that page talks about issuing a "stop" command on system
shutdown?  Can someone shed some light on this?  Like I said, I'm more of a
"Windows" guy.

Thanks in advanced!!

Steve

Re: The time that init.d script waits before shutdown should be configurable

[Upayavira]

On Tue, Nov 10, 2015, at 04:22 PM, Yago Riveiro wrote:
> Patch attached to https://issues.apache.org/jira/browse/SOLR-8065
> 
> 
> 
> 
> 
> The windows script is voodo for me :D, I haven’t the knowledge to port
> this to cmd script.

Great! I saw this!

Two things - firstly, when making a patch, please try to avoid
whitespace changes. The only changes that should show in the diff should
be material changes.

Secondly - I think there was a suggestion that this change could be
ported *inside* the SolrCLI - i.e. into Java code. Do you reckon you
could handle that change? Harder than just updating a shell script, I
know, but could be very useful.

Upayavira

fl=value equals?

[William Bell]

We are trying to look at a value, and change another value based on that.

For example, for national search we want to pass in radius=national, and
then set another variable equal to 0, else set the other variable = to
geodist() calculation.

We tried {!switch} but this only appears to work on fq/q. There is no
function for constants for equals

http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,'national'),0,geodist())

This does not work:

http://localhost:8983/solr/select?q=*:*=national={!switch
case.national=0 default=geodist() v=$radius}

Ideas?



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Arabic analyser

[Mahmoud Almokadem]

Thanks Pual,

Arabic analyser applying filters of normalisation and stemming only for
single terms out of standard tokenzier.
Gathering all synonyms will be hard work. Should I customise my Tokenizer
to handle this case?

Sincerely,
Mahmoud


On Tue, Nov 10, 2015 at 3:06 PM, Paul Libbrecht  wrote:

> Mahmoud,
>
> there is an arabic analyzer:
>   https://wiki.apache.org/solr/LanguageAnalysis#Arabic
> doesn't it do what you describe?
> Synonyms probably work there too.
>
> Paul
>
> > Mahmoud Almokadem 
> > 9 novembre 2015 17:47
> > Thanks Jack,
> >
> > This is a good solution, but we have more combinations that I think
> > can’t be handled as synonyms like every word starts with ‘عبد’ ‘Abd’
> > and ‘أبو’ ‘Abo’. When using Standard tokenizer on ‘أبو بكر’ ‘Abo
> > Bakr’, It’ll be tokenised to ‘أبو’ and ‘بكر’ and the filters will be
> > applied for each separate term.
> >
> > Is there available tokeniser to tokenise ‘أبو *’ or ‘عبد *' as a
> > single term?
> >
> > Thanks,
> > Mahmoud
> >
> >
> >
> > Jack Krupansky 
> > 9 novembre 2015 16:47
> > Use an index-time (but not query time) synonym filter with a rule like:
> >
> > Abd Allah,Abdallah
> >
> > This will index the combined word in addition to the separate words.
> >
> > -- Jack Krupansky
> >
> > On Mon, Nov 9, 2015 at 4:48 AM, Mahmoud Almokadem <
> prog.mahm...@gmail.com>
> >
> > Mahmoud Almokadem 
> > 9 novembre 2015 10:48
> > Hello,
> >
> > We are indexing Arabic content and facing a problem for tokenizing multi
> > terms phrases like 'عبد الله' 'Abd Allah', so users will search for
> > 'عبدالله' 'Abdallah' without space and need to get the results of 'عبد
> > الله' with space. We are using StandardTokenizer.
> >
> >
> > Is there any configurations to handle this case?
> >
> > Thank you,
> > Mahmoud
> >
>
>

Re: Arabic analyser

[Paul Libbrecht]

Mahmoud,

there is an arabic analyzer:
  https://wiki.apache.org/solr/LanguageAnalysis#Arabic
doesn't it do what you describe?
Synonyms probably work there too.

Paul

> Mahmoud Almokadem 
> 9 novembre 2015 17:47
> Thanks Jack,
>
> This is a good solution, but we have more combinations that I think
> can’t be handled as synonyms like every word starts with ‘عبد’ ‘Abd’
> and ‘أبو’ ‘Abo’. When using Standard tokenizer on ‘أبو بكر’ ‘Abo
> Bakr’, It’ll be tokenised to ‘أبو’ and ‘بكر’ and the filters will be
> applied for each separate term.
>
> Is there available tokeniser to tokenise ‘أبو *’ or ‘عبد *' as a
> single term?
>
> Thanks,
> Mahmoud
>
>
>
> Jack Krupansky 
> 9 novembre 2015 16:47
> Use an index-time (but not query time) synonym filter with a rule like:
>
> Abd Allah,Abdallah
>
> This will index the combined word in addition to the separate words.
>
> -- Jack Krupansky
>
> On Mon, Nov 9, 2015 at 4:48 AM, Mahmoud Almokadem 
>
> Mahmoud Almokadem 
> 9 novembre 2015 10:48
> Hello,
>
> We are indexing Arabic content and facing a problem for tokenizing multi
> terms phrases like 'عبد الله' 'Abd Allah', so users will search for
> 'عبدالله' 'Abdallah' without space and need to get the results of 'عبد
> الله' with space. We are using StandardTokenizer.
>
>
> Is there any configurations to handle this case?
>
> Thank you,
> Mahmoud
>

Re: fl=value equals?

[William Bell]

I was able to get it to work kinda with a map().

http://localhost:8983/solr/select?q=*:*=1=

map($radius,1,1,0,geodist())

Where 1= National

Do you have an example of a SearchComponent? It would be pretty easy to
copy map() and develop an equals() right?

if(equals($radius, 'national'), 0, geodist())

This would probably be useful for everyone.

On Tue, Nov 10, 2015 at 4:05 PM, Jan Høydahl  wrote:

> Where is your “b” parameter used? I think that instead of trying to set a
> new “b” http param (which solr will not evaluate as a function), you should
> instead try to insert your function or switch qParser directly where the
> “b” param is used, e.g. in a bq or similar.
>
> A bit heavy weight, but you could of course write a custom SearchComponent
> to construct your “b” parameter...
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 10. nov. 2015 kl. 23.52 skrev William Bell :
> >
> > We are trying to look at a value, and change another value based on that.
> >
> > For example, for national search we want to pass in radius=national, and
> > then set another variable equal to 0, else set the other variable = to
> > geodist() calculation.
> >
> > We tried {!switch} but this only appears to work on fq/q. There is no
> > function for constants for equals
> >
> >
> http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,'national'),0,geodist())
> >
> > This does not work:
> >
> > http://localhost:8983/solr/select?q=*:*=national={!switch
> > case.national=0 default=geodist() v=$radius}
> >
> > Ideas?
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076