I agree this is concerning. This should at least be more publicity about
this... maybe FAQ on escaping user queries. I chased through the details
and it appears macro expansion snuck in as part of the JSON Facet API,
SOLR-7214 for v5.1, even though it affects requests not related to the JSON
Facet feature.
Just now in an app I'm working on I used this macro expansion inside my
user query (parsed by edismax) with a reference to another parameter and,
sure enough, it did the substitution. I changed it to an undefined
parameter and it throw an exception (not desirable for user queries) I
suppose it's not that risky but it's something that shouldn't throw an
error if a user is perhaps searching a source code search engine, for
example, using such a pattern. It might allow a user to spy on the
internals... like try to reference ${qf} and then have the search engine
spit back what your query was so you can then learn some of the fields
names to try and do more advanced queries.
Another sneaky thing is that even edismax lets you switch to another query
parser by starting your user query with {!.... etc. :-) edismax in
particular ought not to allow that. In my search apps I commonly put a
leading space to thwart this.
On Wed, Jun 14, 2017 at 9:49 AM Karl Wright <daddy...@gmail.com> wrote:
> Hi Erik,
>
> I only have snippets of logs from the user so my details are limited. I
> could request the parameter value for you if you like. But since it's
> coming out of some large Documentum repository, pretty much anything is
> possible. ;-) Documentum may have attributes that also use similar
> escaping for macros AFAICT.
>
> Karl
>
>
> On Wed, Jun 14, 2017 at 9:38 AM, Erik Hatcher <erik.hatc...@gmail.com>
> wrote:
>
>> Karl -
>>
>> There’s expandMacros=false, as covered here:
>> https://cwiki.apache.org/confluence/display/solr/Parameter+Substitution
>>
>> But… what exactly is being sent to Solr? Is there some kind of “${…”
>> being sent as a parameter? Just curious what’s getting you into this in
>> the first place. But disabling probably is your most desired solution.
>>
>> Erik
>>
>>
>> > On Jun 14, 2017, at 9:34 AM, Karl Wright <daddy...@gmail.com> wrote:
>> >
>> > Hi all,
>> >
>> > I've got a ManifoldCF user who is posting content to Solr using the MCF
>> Solr output connector. This connector uses SolrJ under the covers -- a
>> fairly recent version -- but also has overridden some classes to insure
>> that multipart form posts will be used for most content.
>> >
>> > The problem is that, for a specific document, the user is getting an
>> ArrayIndexOutOfBounds exception in Solr, as follows:
>> >
>> > >>>>>>
>> > 2017-06-14T08:25:16,546 - ERROR [qtp862890654-69725:SolrException@148]
>> - {collection=c:documentum_manifoldcf_stg,
>> core=x:documentum_manifoldcf_stg_shard1_replica1,
>> node_name=n:**********:8983_solr, replica=r:core_node1, shard=s:shard1} -
>> java.lang.StringIndexOutOfBoundsException: String index out of range: -296
>> > at java.lang.String.substring(String.java:1911)
>> > at
>> org.apache.solr.request.macro.MacroExpander._expand(MacroExpander.java:143)
>> > at
>> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:93)
>> > at
>> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:59)
>> > at
>> org.apache.solr.request.macro.MacroExpander.expand(MacroExpander.java:45)
>> > at
>> org.apache.solr.request.json.RequestUtil.processParams(RequestUtil.java:157)
>> > at
>> org.apache.solr.util.SolrPluginUtils.setDefaults(SolrPluginUtils.java:172)
>> > at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:152)
>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2102)
>> > at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
>> > at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
>> > at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>> > at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
>> > at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> > at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> > at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> > at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> > at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> > at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> > at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> > at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> > at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> > at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> > at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> > at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> > at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> > at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> > at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> > at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> > at org.eclipse.jetty.io
>> .AbstractConnection$2.run(AbstractConnection.java:540)
>> > at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> > at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> > at java.lang.Thread.run(Thread.java:745)
>> > <<<<<<
>> >
>> > It looks worrisome to me that there's now possibly some kind of "macro
>> expansion" that is being triggered within parameters being sent to Solr.
>> Can anyone tell me either how to (a) disable this feature, or (b) how the
>> MCF Solr output connector should escape parameters being posted so that
>> Solr does not attempt any macro expansion? If the latter, I also need to
>> know when this feature appeared, since obviously whether or not to do the
>> escaping will depend on the precise version of the Solr instance involved.
>> >
>> > I'm also quite concerned that considerations of backwards compatibility
>> may have been lost at some point with Solr, since heretofore I could count
>> on older versions of SolrJ working with newer versions of Solr. Please
>> clarify what the current policy is....
>> >
>> >
>> > Thanks,
>> > Karl
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
> --
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
