Hi Jerome,
I was able to setup a configset to perform OpenNLP NER, loading the model files
from local storage.
There is a trick though[1]: the model files must be located *in a jar* or *in a
subdirectory* under ${solr.solr.home}/lib/ or under a directory specified via a
solrconfig.xml <lib> directive.
I tested with the bin/solr cloud example, and put model files under the two
solr home directories, at example/cloud/node1/solr/lib/opennlp/ and
example/cloud/node1/solr/lib/opennlp/. The “opennlp/“ subdirectory is
required, though its name can be anything else you choose.
[1] As you noted, ZkSolrResourceLoader delegates to its parent classloader when
it can’t find resources in a configset, and the parent classloader is set up to
load from subdirectories and jar files under ${solr.solr.home}/lib/ or under a
directory specified via a solrconfig.xml <lib> directive. These directories
themselves are not included in the set of directories from which resources are
loaded; only their children are.
--
Steve
www.lucidworks.com
> On Jul 9, 2018, at 10:10 PM, Jerome Yang <jey...@pivotal.io> wrote:
>
> Hi Steve,
>
> Put models under " ${solr.solr.home}/lib/ " is not working.
> I check the "ZkSolrResourceLoader" seems it will first try to find modes in
> config set.
> If not find, then it uses class loader to load from resources.
>
> Regards,
> Jerome
>
> On Tue, Jul 10, 2018 at 9:58 AM Jerome Yang <jey...@pivotal.io> wrote:
>
>> Thanks Steve!
>>
>>
>> On Tue, Jul 10, 2018 at 5:20 AM Steve Rowe <sar...@gmail.com> wrote:
>>
>>> Hi Jerome,
>>>
>>> See the ref guide[1] for a writeup of how to enable uploading files
>>> larger than 1MB into ZooKeeper.
>>>
>>> Local storage should also work - have you tried placing OpenNLP model
>>> files in ${solr.solr.home}/lib/ ? - make sure you do the same on each node.
>>>
>>> [1]
>>> https://lucene.apache.org/solr/guide/7_4/setting-up-an-external-zookeeper-ensemble.html#increasing-the-file-size-limit
>>>
>>> --
>>> Steve
>>> www.lucidworks.com
>>>
>>>> On Jul 9, 2018, at 12:50 AM, Jerome Yang <jey...@pivotal.io> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> In Solrcloud mode, where to put the OpenNLP models?
>>>> Upload to zookeeper?
>>>> As I test on solr 7.3.1, seems absolute path on local host is not
>>> working.
>>>> And can not upload into zookeeper if the model size exceed 1M.
>>>>
>>>> Regards,
>>>> Jerome
>>>>
>>>> On Wed, Apr 18, 2018 at 9:54 AM Steve Rowe <sar...@gmail.com> wrote:
>>>>
>>>>> Hi Alexey,
>>>>>
>>>>> First, thanks for moving the conversation to the mailing list.
>>> Discussion
>>>>> of usage problems should take place here rather than in JIRA.
>>>>>
>>>>> I locally set up Solr 7.3 similarly to you and was able to get things
>>> to
>>>>> work.
>>>>>
>>>>> Problems with your setup:
>>>>>
>>>>> 1. Your update chain is missing the Log and Run update processors at
>>> the
>>>>> end (I see these are missing from the example in the javadocs for the
>>>>> OpenNLP NER update processor; I’ll fix that):
>>>>>
>>>>> <processor class="solr.LogUpdateProcessorFactory" />
>>>>> <processor class="solr.RunUpdateProcessorFactory" />
>>>>>
>>>>> The Log update processor isn’t strictly necessary, but, from <
>>>>>
>>> https://lucene.apache.org/solr/guide/7_3/update-request-processors.html#custom-update-request-processor-chain
>>>>>> :
>>>>>
>>>>> Do not forget to add RunUpdateProcessorFactory at the end of any
>>>>> chains you define in solrconfig.xml. Otherwise update requests
>>>>> processed by that chain will not actually affect the indexed
>>> data.
>>>>>
>>>>> 2. Your example document is missing an “id” field.
>>>>>
>>>>> 3. For whatever reason, the pre-trained model "en-ner-person.bin"
>>> doesn’t
>>>>> extract anything from text “This is Steve Jobs 2”. It will extract
>>> “Steve
>>>>> Jobs” from text “This is Steve Jobs in white” e.g. though.
>>>>>
>>>>> 4. (Not a problem necessarily) You may want to use a multi-valued
>>> “string”
>>>>> field for the “dest” field in your update chain, e.g. “people_str”
>>> (“*_str”
>>>>> in the default configset is so configured).
>>>>>
>>>>> --
>>>>> Steve
>>>>> www.lucidworks.com
>>>>>
>>>>>> On Apr 17, 2018, at 8:23 AM, Alexey Ponomarenko <
>>> alex1989s...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi once more I am trying to implement named entities extraction using
>>>>> this
>>>>>> manual
>>>>>>
>>>>>
>>> https://lucene.apache.org/solr/7_3_0//solr-analysis-extras/org/apache/solr/update/processor/OpenNLPExtractNamedEntitiesUpdateProcessorFactory.html
>>>>>>
>>>>>> I am modified solrconfig.xml like this:
>>>>>>
>>>>>> <updateRequestProcessorChain name="multiple-extract">
>>>>>> <processor
>>>>> class="solr.OpenNLPExtractNamedEntitiesUpdateProcessorFactory">
>>>>>> <str name="modelFile">opennlp/en-ner-person.bin</str>
>>>>>> <str name="analyzerFieldType">text_opennlp</str>
>>>>>> <str name="source">description_en</str>
>>>>>> <str name="dest">content</str>
>>>>>> </processor>
>>>>>> </updateRequestProcessorChain>
>>>>>>
>>>>>> But when I was trying to add data using:
>>>>>>
>>>>>> *request:*
>>>>>>
>>>>>> POST
>>>>>>
>>>>>
>>> http://localhost:8983/solr/numberplate/update?version=2.2&wt=xml&update.chain=multiple-extract
>>>>>>
>>>>>> <add><doc><field name="description_en">This is Steve Jobs 2
>>>>>> </field><field name="content_pos">This is text 2</field><field
>>>>>> name="content">This is text for content 2</field></doc></add>
>>>>>>
>>>>>> *response*
>>>>>>
>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>> <response>
>>>>>> <lst name="responseHeader">
>>>>>> <int name="status">0</int>
>>>>>> <int name="QTime">3</int>
>>>>>> </lst>
>>>>>> </response>
>>>>>>
>>>>>> But I don't see any data inserted to *content* field and in any other
>>>>> field.
>>>>>>
>>>>>> *If you need some additional data I can provide it.*
>>>>>>
>>>>>> Can you help me? What have I done wrong?
>>>>>
>>>>>
>>>>
>>>> --
>>>> Pivotal Greenplum | Pivotal Software, Inc. <https://pivotal.io/>
>>>
>>>
>>
>> --
>> Pivotal Greenplum | Pivotal Software, Inc. <https://pivotal.io/>
>>
>>
>
> --
> Pivotal Greenplum | Pivotal Software, Inc. <https://pivotal.io/>
