Korean script conversion

[Eyal Naamati]

Hi,

We are starting to index records in Korean. Korean text can be written in two 
scripts: Han characters (Chinese) and Hangul characters (Korean).
We are looking for some solr filter or another built in solr component that 
converts between Han and Hangul characters (transliteration).
I know there is the ICUTransformFilterFactory that can convert between Japanese 
or chinese scripts, for example:
 for 
Japanese script conversions
So far I couldn't find anything readymade for Korean scripts, but perhaps 
someone knows of one?

Thanks!
Eyal Naamati
Alma Developer
Tel: +972-2-6499313
Mobile: +972-547915255
eyal.naam...@exlibrisgroup.com
[Description: Description: Description: Description: C://signature/exlibris.jpg]
www.exlibrisgroup.com

Re: How to boost documents at index time?

[Ahmet Arslan]

Hi,

Did you disable norms ( omitNorms="true" ) accidentally ?

Ahmet


On Saturday, March 28, 2015 9:49 AM, CKReddy Bhimavarapu  
wrote:
I am want to boost docs at index time, I am doing this using boost
parameter in doc field .
but I can't see direct impact on the  doc by using  debuQuery.

My question is that is there any other way to boost doc at index time and
can see the reflected changes i.e direct impact.

-- 
ckreddybh. 

Re: Can SOLR custom analyzer access another field's value?

[Erick Erickson]

I think you're really writing a custom update processor that creates its own
analysis chain to do what you want. Likely this would be somewhat expensive
unless you could batch together user's documents so you'd only have to fetch
the stopwords once.

Then you'd have to do something equivalent at query time, assuming you allowed
query-time synonyms too.

Doing this per-user sounds like an ambitions project, I'd really ask
if it's worth
the effort? If you took the approach of having one collection per
user, it would all
"just work", but depending on how many users you have it could get difficult.

Best,
Erick

On Sat, Mar 28, 2015 at 4:14 PM, sylkaalex  wrote:
> My main goal creating custom stop word filter which will connect to my db and
> get stopwords list which will be different for each user. This filter will
> be attached to deal_title field and during indexing my documents I need to
> know which user is owner of this document (I can get this info from user_id
> field) and use appropriate stop words list for this user.
>
>  required="true" multiValued="false" />
>   required="false" multiValued="false"/>
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Can-SOLR-custom-analyzer-access-another-field-s-value-tp4195851p4196110.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Can SOLR custom analyzer access another field's value?

[sylkaalex]

My main goal creating custom stop word filter which will connect to my db and
get stopwords list which will be different for each user. This filter will
be attached to deal_title field and during indexing my documents I need to
know which user is owner of this document (I can get this info from user_id
field) and use appropriate stop words list for this user. 


 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Can-SOLR-custom-analyzer-access-another-field-s-value-tp4195851p4196110.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ZFS File System for SOLR 3.6 and SOLR 4

[Bill Bell]

Is the an advantage for Xfs over ext4 for Solr ? Anyone done testing?

Bill Bell
Sent from mobile


> On Mar 27, 2015, at 8:14 AM, Shawn Heisey  wrote:
> 
>> On 3/27/2015 12:30 AM, abhi Abhishek wrote:
>> i am trying to use ZFS as filesystem for my Linux Environment. are
>> there any performance implications of using any filesystem other than
>> ext-3/ext-4 with SOLR?
> 
> That should work with no problem.
> 
> The only time Solr tends to have problems is if you try to use a network
> filesystem.  As long as it's a local filesystem and it implements
> everything a program can typically expect from a local filesystem, Solr
> should work perfectly.
> 
> Because of the compatibility problems that the license for ZFS has with
> the GPL, ZFS on Linux is probably not as well tested as other
> filesystems like ext4, xfs, or btrfs, but I have not heard about any big
> problems, so it's probably safe.
> 
> Thanks,
> Shawn
> 

Re: How to boost documents at index time?

[Bill Bell]

Issue a Jura ticket ?

Did you try debugQuery ?

Bill Bell
Sent from mobile


> On Mar 28, 2015, at 1:49 AM, CKReddy Bhimavarapu  wrote:
> 
> I am want to boost docs at index time, I am doing this using boost
> parameter in doc field .
> but I can't see direct impact on the  doc by using  debuQuery.
> 
> My question is that is there any other way to boost doc at index time and
> can see the reflected changes i.e direct impact.
> 
> -- 
> ckreddybh. 

Re: Best way to monitor Solr regarding crashes

[Otis Gospodnetic]

Hi Michael ,

SPM - http://sematext.com/spm will help. It can monitor all SOLR and JVM 
metrics and alert you when their values cross thresholds or become abnormal. In 
your case I'd first look at the JVM metrics - memory pools and their 
utilization. Heartbeat alert will notify you when your server(s) become 
unresponsive without you having to ping them. Solr logs will also likely have 
clues.

Otis
 

> On Mar 28, 2015, at 09:45, Michael Bakonyi  wrote:
> 
> Hi,
> 
> we were using Solr for about 3 months without problems until a few days ago 
> it crashed one time and we don't know why. After a restart everything was 
> fine again but we want to be better prepared the next time this could happen. 
> So I'd like to know what's the best way to monitor a single Solr-instance and 
> what logging-configuration you think is useful for this kind of monitoring. 
> Maybe there's a possibility to automatically restart Solr after it crashed + 
> to see in detail in the logs what happend right before the crash ..?
> 
> Can you give me any hints? We're using Tomcat 6.X with Solr 4.8.X
> 
> Cheers,
> Michael

Re: Solr 5.0.0 and HDFS

[Mark Miller]

Hmm...can you file a JIRA issue with this info?

- Mark

On Fri, Mar 27, 2015 at 6:09 PM Joseph Obernberger 
wrote:

> I just started up a two shard cluster on two machines using HDFS. When I
> started to index documents, the log shows errors like this. They repeat
> when I execute searches.  All seems well - searches and indexing appear
> to be working.
> Possibly a configuration issue?
> My HDFS config:
>   class="solr.HdfsDirectoryFactory">
>  true
>  160
>   name="solr.hdfs.blockcache.direct.memory.allocation">true
>  16384
>  true
>  false
>  true
>  64
>  512
>  hdfs://nameservice1:8020/solr5
>  /etc/hadoop/conf.cloudera.hdfs1 str>
>  
> Thank you!
>
> -Joe
> 
>
> java.lang.IllegalStateException: file:
> BlockDirectory(HdfsDirectory@799d5a0e
> lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@49838b82) appears
> both in delegate and in cache: cache=[_25.fnm, _2d.si, _2e.nvd, _2b.si,
> _28.tvx, _2c.tvx, _1t.si, _27.nvd, _2b.tvd, _2d_Lucene50_0.pos, _23.nvd,
> _28_Lucene50_0.doc, _28_Lucene50_0.dvd, _2d.fdt, _2c_Lucene50_0.pos,
> _23.fdx, _2b_Lucene50_0.doc, _2d.nvm, _28.nvd, _23.fnm,
> _2b_Lucene50_0.tim, _2e.fdt, _2d_Lucene50_0.doc, _2b_Lucene50_0.dvd,
> _2d_Lucene50_0.dvd, _2b.nvd, _2g.tvx, _28_Lucene50_0.dvm,
> _1v_Lucene50_0.tip, _2e_Lucene50_0.dvm, _2e_Lucene50_0.pos, _2g.fdx,
> _2e.nvm, _2f.fdx, _1s.tvd, _23.nvm, _27.nvm, _1s_Lucene50_0.tip,
> _2c.fnm, _2b.fdt, _2d.fdx, _2c.fdx, _2c.nvm, _2e.fnm,
> _2d_Lucene50_0.dvm, _28.nvm, _28.fnm, _2b_Lucene50_0.tip,
> _2e_Lucene50_0.dvd, _2c.si, _2f.fdt, _2b.fnm, _2e_Lucene50_0.tip,
> _28.si, _28_Lucene50_0.tip, _2f.tvd, _2d_Lucene50_0.tim, _2f.tvx,
> _2b_Lucene50_0.pos, _2e.fdx, _28.fdx, _2c_Lucene50_0.dvd, _2g.tvd,
> _2c_Lucene50_0.tim, _2b.nvm, _23.fdt, _1s_Lucene50_0.tim,
> _28_Lucene50_0.tim, _2c_Lucene50_0.doc, _28.tvd, _2b.tvx, _2c.nvd,
> _2b.fdx, _2c_Lucene50_0.tip, _2e_Lucene50_0.doc, _2e_Lucene50_0.tim,
> _2c.fdt, _27.tvd, _2d.tvd, _2d.tvx, _28_Lucene50_0.pos,
> _2b_Lucene50_0.dvm, _2e.si, _2e.tvd, _2d.fnm, _2c.tvd, _2g.fdt, _2e.tvx,
> _28.fdt, _2d_Lucene50_0.tip, _2c_Lucene50_0.dvm,
> _2d.nvd],delegate=[_10.fdt, _10.fdx, _10.fnm, _10.nvd, _10.nvm, _10.si,
> _10.tvd, _10.tvx, _10_Lucene50_0.doc, _10_Lucene50_0.dvd,
> _10_Lucene50_0.dvm, _10_Lucene50_0.pos, _10_Lucene50_0.tim,
> _10_Lucene50_0.tip, _11.fdt, _11.fdx, _11.fnm, _11.nvd, _11.nvm, _11.si,
> _11.tvd, _11.tvx, _11_Lucene50_0.doc, _11_Lucene50_0.dvd,
> _11_Lucene50_0.dvm, _11_Lucene50_0.pos, _11_Lucene50_0.tim,
> _11_Lucene50_0.tip, _12.fdt, _12.fdx, _12.fnm, _12.nvd, _12.nvm, _12.si,
> _12.tvd, _12.tvx, _12_Lucene50_0.doc, _12_Lucene50_0.dvd,
> _12_Lucene50_0.dvm, _12_Lucene50_0.pos, _12_Lucene50_0.tim,
> _12_Lucene50_0.tip, _13.fdt, _13.fdx, _13.fnm, _13.nvd, _13.nvm, _13.si,
> _13.tvd, _13.tvx, _13_Lucene50_0.doc, _13_Lucene50_0.dvd,
> _13_Lucene50_0.dvm, _13_Lucene50_0.pos, _13_Lucene50_0.tim,
> _13_Lucene50_0.tip, _14.fdt, _14.fdx, _14.fnm, _14.nvd, _14.nvm, _14.si,
> _14.tvd, _14.tvx, _14_Lucene50_0.doc, _14_Lucene50_0.dvd,
> _14_Lucene50_0.dvm, _14_Lucene50_0.pos, _14_Lucene50_0.tim,
> _14_Lucene50_0.tip, _15.fdt, _15.fdx, _15.fnm, _15.nvd, _15.nvm, _15.si,
> _15.tvd, _15.tvx, _15_Lucene50_0.doc, _15_Lucene50_0.dvd,
> _15_Lucene50_0.dvm, _15_Lucene50_0.pos, _15_Lucene50_0.tim,
> _15_Lucene50_0.tip, _1f.fdt, _1f.fdx, _1f.fnm, _1f.nvd, _1f.nvm, _1f.si,
> _1f.tvd, _1f.tvx, _1f_Lucene50_0.doc, _1f_Lucene50_0.dvd,
> _1f_Lucene50_0.dvm, _1f_Lucene50_0.pos, _1f_Lucene50_0.tim,
> _1f_Lucene50_0.tip, _1g.fdt, _1g.fdx, _1g.fnm, _1g.nvd, _1g.nvm, _1g.si,
> _1g.tvd, _1g.tvx, _1g_Lucene50_0.doc, _1g_Lucene50_0.dvd,
> _1g_Lucene50_0.dvm, _1g_Lucene50_0.pos, _1g_Lucene50_0.tim,
> _1g_Lucene50_0.tip, _1h.fdt, _1h.fdx, _1h.fnm, _1h.nvd, _1h.nvm, _1h.si,
> _1h.tvd, _1h.tvx, _1h_Lucene50_0.doc, _1h_Lucene50_0.dvd,
> _1h_Lucene50_0.dvm, _1h_Lucene50_0.pos, _1h_Lucene50_0.tim,
> _1h_Lucene50_0.tip, _1i.fdt, _1i.fdx, _1i.fnm, _1i.nvd, _1i.nvm, _1i.si,
> _1i.tvd, _1i.tvx, _1i_Lucene50_0.doc, _1i_Lucene50_0.dvd,
> _1i_Lucene50_0.dvm, _1i_Lucene50_0.pos, _1i_Lucene50_0.tim,
> _1i_Lucene50_0.tip, _1j.fdt, _1j.fdx, _1j.fnm, _1j.nvd, _1j.nvm, _1j.si,
> _1j.tvd, _1j.tvx, _1j_Lucene50_0.doc, _1j_Lucene50_0.dvd,
> _1j_Lucene50_0.dvm, _1j_Lucene50_0.pos, _1j_Lucene50_0.tim,
> _1j_Lucene50_0.tip, _1k.fdt, _1k.fdx, _1k.fnm, _1k.nvd, _1k.nvm, _1k.si,
> _1k.tvd, _1k.tvx, _1k_Lucene50_0.doc, _1k_Lucene50_0.dvd,
> _1k_Lucene50_0.dvm, _1k_Lucene50_0.pos, _1k_Lucene50_0.tim,
> _1k_Lucene50_0.tip, _1l.fdt, _1l.fdx, _1l.fnm, _1l.nvd, _1l.nvm, _1l.si,
> _1l.tvd, _1l.tvx, _1l_Lucene50_0.doc, _1l_Lucene50_0.dvd,
> _1l_Lucene50_0.dvm, _1l_Lucene50_0.pos, _1l_Lucene50_0.tim,
> _1l_Lucene50_0.tip, _1m.fdt, _1m.fdx, _1m.fnm, _1m.nvd, _1m.nvm, _1m.si,
> _1m.tvd, _1m.tvx, _1m_Lucene50_0.doc, _1m_Lucene50_0.dvd,
> _1m_Lucene50_0.dvm, _1m_Lucene50_0.pos, _1m_Lucene50_0.tim,
>

Re: New To Solr, getting error using the quick start guide

[Shawn Heisey]

On 3/27/2015 8:00 PM, Will ferrer wrote:
> I am new to solr and trying to run through the quick start guide (
> http://lucene.apache.org/solr/quickstart.html).
> 
> The installation seems fine but then I run:
> 
> bin/solr start -e cloud -noprompt

You are starting the cloud example with no prompts.

> http://localhost:8983/solr/#/ shows data in my web browser, but the cloud
> tab is empty under graph.
> 
> Any advice any one give me to get me started here with the product would be
> very appreciated.

The console log that you included with your message said nothing about
creating the gettingstarted collection, but when I try the command you
used on the following setups, it creates the collection every time:

*) Linux, from the tags/lucene_solr_5_0_0 source.
*) Windows 8.1, from the binary 5.0.0 download.
*) Linux, from the branch_5x source.

Here's my console log from the first item above - Solr built from the
tags/lucene_solr_5_0_0 source:

---

elyograg@sauron:~/asf/lucene_solr_5_0_0/solr$ bin/solr -e cloud -noprompt

Welcome to the SolrCloud example!


Starting up 2 Solr nodes for your example SolrCloud cluster.
Creating Solr home directory
/home/elyograg/asf/lucene_solr_5_0_0/solr/example/cloud/node1/solr
Cloning Solr home directory
/home/elyograg/asf/lucene_solr_5_0_0/solr/example/cloud/node1 into
/home/elyograg/asf/lucene_solr_5_0_0/solr/example/cloud/node2

Starting up SolrCloud node1 on port 8983 using command:

solr start -cloud -s example/cloud/node1/solr -p 8983


Waiting to see Solr listening on port 8983 [/]
Started Solr server on port 8983 (pid=13260). Happy searching!


Starting node2 on port 7574 using command:

solr start -cloud -s example/cloud/node2/solr -p 7574 -z localhost:9983


Waiting to see Solr listening on port 7574 [/]
Started Solr server on port 7574 (pid=13419). Happy searching!

  Connecting to ZooKeeper at localhost:9983
Uploading
/home/elyograg/asf/lucene_solr_5_0_0/solr/server/solr/configsets/data_driven_schema_configs/conf
for config gettingstarted to ZooKeeper at localhost:9983

Creating new collection 'gettingstarted' using command:
http://166.70.79.221:7574/solr/admin/collections?action=CREATE&name=gettingstarted&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=gettingstarted

{
  "responseHeader":{
"status":0,
"QTime":6869},
  "success":{"":{
  "responseHeader":{
"status":0,
"QTime":6387},
  "core":"gettingstarted_shard1_replica2"}}}



SolrCloud example running, please visit http://localhost:8983/solr


elyograg@sauron:~/asf/lucene_solr_5_0_0/solr$

---

I don't know why it's not creating the collection for you, unless maybe
you are running a different version built from older source code or
something.

Thanks,
Shawn

Re: Best way to monitor Solr regarding crashes

[Shawn Heisey]

On 3/28/2015 7:45 AM, Michael Bakonyi wrote:
> we were using Solr for about 3 months without problems until a few days ago 
> it crashed one time and we don't know why. After a restart everything was 
> fine again but we want to be better prepared the next time this could happen. 
> So I'd like to know what's the best way to monitor a single Solr-instance and 
> what logging-configuration you think is useful for this kind of monitoring. 
> Maybe there's a possibility to automatically restart Solr after it crashed + 
> to see in detail in the logs what happend right before the crash ..?
> 
> Can you give me any hints? We're using Tomcat 6.X with Solr 4.8.X

Detecting that the instance is down could be done in a lot of ways,
including sending an HTTP request and looking for a valid HTTP response
code.

Do you know why it crashed?  Solr normally will not completely crash
even when problems are encountered, but there could be a bug, especially
in an older release.

If it was an OutOfMemoryError exception (which quite often does NOT
cause a complete crash), then you can give the java commandline a
program/script to run on OOME that can do anything you care to write.

A script (bin/oom_solr.sh) is included in the 5.0 download which gets
automatically used by the 5.0 startup script on Linux or UNIX systems.
That script will completely kill the process.  It does not attempt to
restart it.  Killing the process is safest when an OOME is encountered,
because program operation is completely unpredictable when java runs out
of memory, and usually Solr will stop working correctly.

Thanks,
Shawn

Best way to monitor Solr regarding crashes

[Michael Bakonyi]

Hi,

we were using Solr for about 3 months without problems until a few days ago it 
crashed one time and we don't know why. After a restart everything was fine 
again but we want to be better prepared the next time this could happen. So I'd 
like to know what's the best way to monitor a single Solr-instance and what 
logging-configuration you think is useful for this kind of monitoring. Maybe 
there's a possibility to automatically restart Solr after it crashed + to see 
in detail in the logs what happend right before the crash ..?

Can you give me any hints? We're using Tomcat 6.X with Solr 4.8.X

Cheers,
Michael

How to boost documents at index time?

[CKReddy Bhimavarapu]

I am want to boost docs at index time, I am doing this using boost
parameter in doc field .
but I can't see direct impact on the  doc by using  debuQuery.

My question is that is there any other way to boost doc at index time and
can see the reflected changes i.e direct impact.

-- 
ckreddybh.