Re: Elastic Search configuration

2015-05-12 Thread Hardik Dobariya 

we have around 500 indices all sizing to approx 50gb respectively and total 
size goes around 1tb for all indices and will keep on increasing .
The reason behind setting node3 as Master=false,data=false because in 
configuration file i read this type of node will only work for searching 
,aggregation etc. Do i still need to set node3 three as mater and data

On Wednesday, May 13, 2015 at 1:49:18 AM UTC+5:30, Magnus Bäck wrote:
>
> On Tuesday, May 12, 2015 at 14:24 CEST, 
>  Hardik Dobariya > wrote: 
>
> > oh yes i forgot to mentioned that we are using SAN mounted volumes. 
>
> Um, okay. But you said you accessed them via a UNC path (\\pc1\data). 
> Then it's not a locally mounted volume. 
>
> > And according to your reply i can say my configuration will work if am 
> > using SAN mounted volumes? 
>
> Yes. SAN and NAS both work but SAN is preferred. 
>
> > one last question if my configuration is correct then do i have to 
> > drop down my shard size from 5 to 1 or to something less to 3 which 
> > might increase performance? 
>
> There are rarely any "correct" configurations, and what's optimal 
> depends on many factors. While there are rules of thumb you'd have 
> to measure for yourself what's best for you. 
>
> The optimal number of shards depends on how many indexes you have, 
> how many nodes you have (that we know; three) how big the indexes 
> are etc. 
>
> Shards have a certain overhead so you don't want to have too many 
> (you'll waste resources to shard book keeping) but you don't want 
> them to be too big either and you'll want to spread them out between 
> your nodes. 
>
> > These are the three nodes configured 
> >  1. Node 1:Master=true,data=true (30gb ram,8 cores) 
> >  2. Node 2:Master=true,data=true (30gb ram,8 cores) 
> >  3. Node 3:Master=false,data=false (30gb ram,8 cores) 
>
> Yes, and again, you should turn node 3 into master=true, data=true 
> as well. 
>
> -- 
> Magnus Bäck| Software Engineer, Development Tools 
> magnu...@sonymobile.com  | Sony Mobile Communications 
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bcc44119-1521-4ef5-8024-73eee4947537%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

percolate returns empty matches under heavy load during elasticsearch cluster re-sizing

2015-05-12 Thread ekesken 
hi,

we have a percolation miss problem described in this post 
http://stackoverflow.com/questions/30194246/percolate-returns-empty-matches-under-heavy-load-during-elasticsearch-cluster-re
 and 
probably related with this issue 
https://github.com/elastic/elasticsearch/issues/10722.

you can reproduce problem using this script 
https://gist.github.com/ekesken/de41598a1e7e54c6f33c

at least we need a workaround for now, because missing content is not 
acceptable for us. is there a way to check if every shard is really OK from 
code before sending percolation requests? obviously checking green status 
does not work.

/ekesken

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5276971a-a884-4313-a3eb-8d175e2d46e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch is not able to search for Nonnglish text present in PDF type of attachment

2015-05-12 Thread Prashant Agrawal 
Hi Robert,

I didn't get the complete understanding from the same. Can you just
elaborate like if anything can be done for the same wrt Elasticsearch or
plugin perspective or this is the inconsistent in terms of PDF type of
attachment and nothing can be done for the same.

Its all depends upon the PDF content where some can be indexed properly and
some not.

~Prashant



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-is-not-able-to-search-for-NonEnglish-text-present-in-PDF-type-of-attachmentE-tp4074717p4074732.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1431493936628-4074732.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Ping exception after elastic search restart

2015-05-12 Thread bvnrwork 
Can some one let me know why below exception comes after restart 

System.Exception: ping returned no status code
Inner exception: System.Net.WebException: The operation has timed out
   *** Exception to reporting point stack trace ***
   at System.Net.HttpWebRequest.GetResponse()
   at 
Elasticsearch.Net.Connection.HttpConnection.DoSynchronousRequest(HttpWebRequest 
request, Byte[] data, IRequestConfiguration requestSpecificConfig)
   *** Reporting point to the top stack trace ***

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/eac5a212-2e6c-4984-9c47-bd772d49e2a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

2015-05-12 Thread Magnus Bäck 
On Tuesday, May 12, 2015 at 14:24 CEST,
 Hardik Dobariya  wrote:

> oh yes i forgot to mentioned that we are using SAN mounted volumes.

Um, okay. But you said you accessed them via a UNC path (\\pc1\data).
Then it's not a locally mounted volume.

> And according to your reply i can say my configuration will work if am
> using SAN mounted volumes?

Yes. SAN and NAS both work but SAN is preferred.

> one last question if my configuration is correct then do i have to
> drop down my shard size from 5 to 1 or to something less to 3 which
> might increase performance?

There are rarely any "correct" configurations, and what's optimal
depends on many factors. While there are rules of thumb you'd have
to measure for yourself what's best for you.

The optimal number of shards depends on how many indexes you have,
how many nodes you have (that we know; three) how big the indexes
are etc.

Shards have a certain overhead so you don't want to have too many
(you'll waste resources to shard book keeping) but you don't want
them to be too big either and you'll want to spread them out between
your nodes.

> These are the three nodes configured
>  1. Node 1:Master=true,data=true (30gb ram,8 cores)
>  2. Node 2:Master=true,data=true (30gb ram,8 cores)
>  3. Node 3:Master=false,data=false (30gb ram,8 cores)

Yes, and again, you should turn node 3 into master=true, data=true
as well.

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150512201910.GB2679%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Re: Curator 3

2015-05-12 Thread Magnus Bäck 
On Tuesday, May 12, 2015 at 21:06 CEST,
 Wendel Ferreira  wrote:

> Pessoal, boa tarde.
>  Depois que atualizei o elasticsearch para a 1.5 vi que a rotinha
> no crontab do curator parou de funcionar.

This mailing list is in English. The new discussion forum that's
replacing this list has a couple of non-English categories but
currently none for Portuguese (if that indeed is your preferred
language).

https://discuss.elastic.co/c/in-your-native-tongue

[...]

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150512201102.GA2679%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Curator 3

2015-05-12 Thread Wendel Ferreira 
Pessoal, boa tarde.

 Depois que atualizei o elasticsearch para a 1.5 vi que a rotinha no 
crontab do curator parou de funcionar.
Eu estava lendo a documentação em 
https://www.elastic.co/blog/curator-3-0-released e vi que mudou algumas 
coisas.

A linha do contrab do curator 2 que eu tinha era esta (excluir logs com 
mais de 90 dias):
curator delete --older-than 90

Versão do Elasticsearch
 # rpm -qa | grep elas
elasticsearch-1.5.1-1.noarch

Temo como fazer de uma forma mais simples do que na documentação?
Exemplo:
curator delete indices --older-than 30 --newer-than 60 --time-unit days 
\
  --timestring '%Y.%m.%d' --prefix logs --suffix prod \
  --exclude logs-2015.02.01-prod --exclude 2015.01.31 \
  --index logs-2015.02.01-dev

Qual seria o comando equivalente na versão 3 para excluir logs/indices com 
mais de 90 dias?

Desde já obrigado.

Att.
Wendel Ferreira




-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5ca88993-869b-4b54-85c3-c8324afea27a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ScriptedField Time format question

2015-05-12 Thread Adam Whittaker 
Hello,

I am trying to get something like the Hour of day bar histogram that is 
demonstrated on this page working:

https://www.elastic.co/blog/kibana-4-beta-3-now-more-filtery

It says to create a scripted field based on the timestamp like this:

Integer.parseInt(new Date(doc['OpenedDate'].value).format("H"))

I added my datetime field (openedDate) but it gives me and error when i got 
to discover (an error occurred with your request).

I also tried swapping out my datetime field with the @timestamp field (as 
per the sample) but i got the same error.

Any help would be appreciated.

Thanks,

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2e5f02bc-6841-41aa-aaaf-8abc85577a78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

2015-05-12 Thread Hardik Dobariya 
oh yes i forgot to mentioned that we are using SAN mounted volumes.
And according to your reply i can say my configuration will work if am 
using SAN mounted volumes?
one last question if my configuration is correct then do i have to drop 
down my shard size from 5 to 1 or to something less to 3 which might 
increase performance?
These are the three nodes configured


   1. Node 1:Master=true,data=true (30gb ram,8 cores)
   2. Node 2:Master=true,data=true (30gb ram,8 cores)
   3. Node 3:Master=false,data=false (30gb ram,8 cores)


On Tuesday, May 12, 2015 at 5:46:54 PM UTC+5:30, Magnus Bäck wrote:
>
> On Tuesday, May 12, 2015 at 13:24 CEST, 
>  Hardik Dobariya > wrote: 
>
> > yes i can understand the use of network file system is not good. 
> > Actually we do not have any physical machines.we are using virtual 
> > environment.This is the reason we are using NAS to store data on 
> > network. 
>
> Sure, but you don't have to use NAS just because you use VMs. 
> Local disks or volumes mounted from a SAN are still preferred 
> to accessing the file system over SMB or NFS. 
>
> > Any suggestion on using virtual environment for elastic search? 
> > and yes am already using 3 nodes.2 masters with data and one child 
>
> Yes, and as Mark says that's a bad idea. You're better off allowing 
> all three nodes to be masters. Unless you have a serious query load 
> you should keep data on all three nodes. Having a third of the 
> cluster's capacity dedicated to processing queries is most likely 
> very wasteful. 
>
> -- 
> Magnus Bäck| Software Engineer, Development Tools 
> magnu...@sonymobile.com  | Sony Mobile Communications 
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8bb923dd-67e2-457d-982b-f38c8d372c0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

2015-05-12 Thread Magnus Bäck 
On Tuesday, May 12, 2015 at 13:24 CEST,
 Hardik Dobariya  wrote:

> yes i can understand the use of network file system is not good.
> Actually we do not have any physical machines.we are using virtual
> environment.This is the reason we are using NAS to store data on
> network.

Sure, but you don't have to use NAS just because you use VMs.
Local disks or volumes mounted from a SAN are still preferred
to accessing the file system over SMB or NFS.

> Any suggestion on using virtual environment for elastic search?
> and yes am already using 3 nodes.2 masters with data and one child

Yes, and as Mark says that's a bad idea. You're better off allowing
all three nodes to be masters. Unless you have a serious query load
you should keep data on all three nodes. Having a third of the
cluster's capacity dedicated to processing queries is most likely
very wasteful.

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150512121645.GA806%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

2015-05-12 Thread Hardik Dobariya 
yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual 
environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?
and yes am already using 3 nodes.2 masters with data and one child 

On Tuesday, May 12, 2015 at 4:52:25 PM UTC+5:30, Hardik Dobariya wrote:
>
> yes i can understand the use of network file system is not good.
> Actually we do not have any physical machines.we are using virtual 
> environment.This is the reason we are using NAS to store data on network.
> Any suggestion on using virtual environment for elastic search?
>
> On Tuesday, May 12, 2015 at 4:39:15 PM UTC+5:30, Mark Walkom wrote:
>>
>> We do not recommend the use of networked file systems.
>> We definitely don't recommend using those and putting both sets of data 
>> in the one location.
>>
>> Reducing shard count may help, it'd certainly reduce the overhead in 
>> maintaining all those shards. Just keep shard sizes under 50GB if you can.
>> Also having two masters can lead to potential split brain, you'd be 
>> better off making node 3 a master: true, data: false and setting 
>> minimum_masters.
>>
>> On 12 May 2015 at 20:22, Hardik Dobariya  wrote:
>>
>>> No am not facing any performance issue.Just want to confirm if my 
>>> configuration is feasible or not and what will happen if i bring down shard 
>>> to 1.
>>> My ES version is 1.5.2 and java release is 8.45
>>>
>>> On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:

 Are you having performance issues now?

 If so I'd start with replacing your disk setup, chances are it's not as 
 fast as you think, especially if you are storing the data from both nodes 
 on the same path.
 Also what ES version are you on, what Java release and version?

 On 12 May 2015 at 19:01, Hardik Dobariya  
 wrote:

> Please let me know if below elastic server configuration is feasible 
> or not.
>
> Total 1 cluster with 3 nodes having 5 shards and 1 replica with 
> multiple indices having total around 100 million data ,sizing 1tb and 
> still 
> increasing
> --
>
>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>
> --
>
> All nodes are pointing to same UNC path(\\pc1\\data) data 
> location.Data is not stored locally on nodes,all data is stored on one 
> location as mentioned above.we are using Nas to store data and seems Nas 
> is 
> very fast so we are storing data on UNC path and not locally.
>
> While googling found if shards are decreased ,it will increase search 
> performance.According to the specified configuration should i drop no of 
> shards to 1?.Will it increase the performance or does the configuration 
> need any changes?.
> Please suggest me the configuration.
> Thanks
>
> -- 
> Please update your bookmarks! We have moved to 
> https://discuss.elastic.co/
> --- 
> You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  -- 
>>> Please update your bookmarks! We have moved to 
>>> https://discuss.elastic.co/
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elas

Re: Elastic Search configuration

2015-05-12 Thread Hardik Dobariya 
yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual 
environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?

On Tuesday, May 12, 2015 at 4:39:15 PM UTC+5:30, Mark Walkom wrote:
>
> We do not recommend the use of networked file systems.
> We definitely don't recommend using those and putting both sets of data in 
> the one location.
>
> Reducing shard count may help, it'd certainly reduce the overhead in 
> maintaining all those shards. Just keep shard sizes under 50GB if you can.
> Also having two masters can lead to potential split brain, you'd be better 
> off making node 3 a master: true, data: false and setting minimum_masters.
>
> On 12 May 2015 at 20:22, Hardik Dobariya  > wrote:
>
>> No am not facing any performance issue.Just want to confirm if my 
>> configuration is feasible or not and what will happen if i bring down shard 
>> to 1.
>> My ES version is 1.5.2 and java release is 8.45
>>
>> On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
>>>
>>> Are you having performance issues now?
>>>
>>> If so I'd start with replacing your disk setup, chances are it's not as 
>>> fast as you think, especially if you are storing the data from both nodes 
>>> on the same path.
>>> Also what ES version are you on, what Java release and version?
>>>
>>> On 12 May 2015 at 19:01, Hardik Dobariya  wrote:
>>>
 Please let me know if below elastic server configuration is feasible or 
 not.

 Total 1 cluster with 3 nodes having 5 shards and 1 replica with 
 multiple indices having total around 100 million data ,sizing 1tb and 
 still 
 increasing
 --

1. Node 1:Master=true,data=true (30gb ram,8 cores)
2. Node 2:Master=true,data=true (30gb ram,8 cores)
3. Node 3:Master=false,data=false (30gb ram,8 cores)

 --

 All nodes are pointing to same UNC path(\\pc1\\data) data location.Data 
 is not stored locally on nodes,all data is stored on one location as 
 mentioned above.we are using Nas to store data and seems Nas is very fast 
 so we are storing data on UNC path and not locally.

 While googling found if shards are decreased ,it will increase search 
 performance.According to the specified configuration should i drop no of 
 shards to 1?.Will it increase the performance or does the configuration 
 need any changes?.
 Please suggest me the configuration.
 Thanks

 -- 
 Please update your bookmarks! We have moved to 
 https://discuss.elastic.co/
 --- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> Please update your bookmarks! We have moved to 
>> https://discuss.elastic.co/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/66346979-7579-41a0-9e9c-d3650bf81227%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

2015-05-12 Thread Mark Walkom 
We do not recommend the use of networked file systems.
We definitely don't recommend using those and putting both sets of data in
the one location.

Reducing shard count may help, it'd certainly reduce the overhead in
maintaining all those shards. Just keep shard sizes under 50GB if you can.
Also having two masters can lead to potential split brain, you'd be better
off making node 3 a master: true, data: false and setting minimum_masters.

On 12 May 2015 at 20:22, Hardik Dobariya 
wrote:

> No am not facing any performance issue.Just want to confirm if my
> configuration is feasible or not and what will happen if i bring down shard
> to 1.
> My ES version is 1.5.2 and java release is 8.45
>
> On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
>>
>> Are you having performance issues now?
>>
>> If so I'd start with replacing your disk setup, chances are it's not as
>> fast as you think, especially if you are storing the data from both nodes
>> on the same path.
>> Also what ES version are you on, what Java release and version?
>>
>> On 12 May 2015 at 19:01, Hardik Dobariya  wrote:
>>
>>> Please let me know if below elastic server configuration is feasible or
>>> not.
>>>
>>> Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple
>>> indices having total around 100 million data ,sizing 1tb and still
>>> increasing
>>> --
>>>
>>>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>>>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>>>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>>>
>>> --
>>>
>>> All nodes are pointing to same UNC path(\\pc1\\data) data location.Data
>>> is not stored locally on nodes,all data is stored on one location as
>>> mentioned above.we are using Nas to store data and seems Nas is very fast
>>> so we are storing data on UNC path and not locally.
>>>
>>> While googling found if shards are decreased ,it will increase search
>>> performance.According to the specified configuration should i drop no of
>>> shards to 1?.Will it increase the performance or does the configuration
>>> need any changes?.
>>> Please suggest me the configuration.
>>> Thanks
>>>
>>> --
>>> Please update your bookmarks! We have moved to
>>> https://discuss.elastic.co/
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9qjR17AOO4hinjxbvAKYO5L5tttPncz_fPL-ZQ5mGCKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch is not able to search for Nonnglish text present in PDF type of attachment

2015-05-12 Thread Robert Muir 
Its your PDF (and the font being used plays a role in this case).

PDFs encode glyphs (display order), not characters (logical order).
Usually the distinction is not important, but for complex writing systems
it matters.

Open your PDF in acrobat and highlight the word in question, and do
"copy/paste" and you will see it pastes the same way.
You can also see this bogus mapping clearly if you extract the font data
with fontforge (attached).


On Tue, May 12, 2015 at 5:17 AM, Prashant Agrawal <
prashant.agra...@paladion.net> wrote:

> Hi Team,
>
> We are facing an issue while searching the Non English text indexed as PDF
> type of document. Below are the complete details.
>
> 1) I am having a pdf document as New_Pdf_issue.pdf which is attached in
> this
> mail.
> 2) Created an indexing request alongwith mapping as well which is attached
> as pdf_index_issue.sh
> 3) Now if you will look onto pdf attachment you will find keywords such as
> "अधिकार", so if i am searching as "अधिकार" I am not able to get any
> matching
> documents for the same.
>
> Note : What we observed is like when we perform search query as
> {
>   "fields": [
> "SessionAtt.content_type",
> "SessionAtt"
>   ],
>   "query": {
> "bool": {
>   "must": [
> {
>   "query_string": {
> "fields": [
>   "Content",
>   "SessionAtt"
> ],
> "query": "*"
>   }
> }
>   ]
> }
>   }
> }
>
> We are observing as "अधिकार" words has been indexed as "अधधकार".
>
> So can anyone let me know what could be the issue for the same.
>
> ~Prashant
>
> pdf_index_issue.sh
> <
> http://elasticsearch-users.115913.n3.nabble.com/file/n4074717/pdf_index_issue.sh
> >
> New_Pdf_issue.pdf
> <
> http://elasticsearch-users.115913.n3.nabble.com/file/n4074717/New_Pdf_issue.pdf
> >
>
>
>
> --
> View this message in context:
> http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-is-not-able-to-search-for-Nonnglish-text-present-in-PDF-type-of-attachment-tp4074717.html
> Sent from the Elasticsearch Users mailing list archive at Nabble.com.
>
> --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/1431422241775-4074717.post%40n3.nabble.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAD06sYQRvGGvtROhGFKY%3DUkfgdvbM%3DAHiEftk8it4wWpgpK5hg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Forums Are Now Live at http://discuss.elastic.co

2015-05-12 Thread Leslie Hawthorn 
It is entirely possible some of these items can be fixed. Investigating.

Thank you for the candid feedback, Doug, Ivan and Jörg.

Cheers,
LH

On Mon, May 11, 2015 at 10:24 PM, joergpra...@gmail.com <
joergpra...@gmail.com> wrote:

> +1 for Doug and Ivan
>
> I'd also like to find the real names (which are available at Discuss
> because they are shown in the profile of a user) being added to the message
> view and the mail "From" header, for a more personal communication style.
> It would be easier to begin a reply with a greeting then.
>
> Hopefully this is received as a constructive feedback and not as moaning
> about Discuss software.
>
> Best,
>
> Jörg
>
>
> On Mon, May 11, 2015 at 12:03 AM, Doug Turnbull <
> dturnb...@opensourceconnections.com> wrote:
>
>> Oh! I didn't see that. Thank you. I will look again.
>>
>> Yeah I actually like many things about discourse. I also like many of the
>> low friction aspects of a mailing list. I am worried my email sounds too
>> negative. I should list positives like great markdown support and much
>> prettier and more legible emails. Discourse is phenomenal forum software. I
>> honestly hope they can really bridge the gap effortlessly to have as part
>> of their capabilities truly effortless implementation of a dev mailing list
>> with an optional forum view.
>>
>> I'm sure I'll be using discourse to great effect (and it seems to be
>> being used). So I could be completly wrong about all my points. :)
>>
>> Doug
>>
>> On Sunday, May 10, 2015, Ivan Brusic  wrote:
>>
>>> I should have added something similar to what does expressed in his last
>>> paragraph. My feedback was meant to be constructive. Despite this being a
>>> technical mailing list, I still appreciate a more personal touch.
>>>
>>> BTW, Doug, you can watch specific categories. There is a general
>>> watch-all setting that you might have turned on.
>>>
>>> Ivan
>>> On May 10, 2015 2:37 PM, "Doug Turnbull" <
>>> dturnb...@opensourceconnections.com> wrote:
>>>
 I agree.

 While I appreciate the experimentation, I hope discourse can get to a
 point where I can forget it's even a forum. Whereby everything can just
 truly feel like a native mailing list with a great deal less friction, but
 it's not there yet.

 Here's some of the missing features:

 - a signup process that's more an email subscription (just paste in an
 email address) without having to create any kind of forum account or
 profile.

 - the ability to reply to someone individually to their email address.
 The ability for others to email me directly without discourses PM feature.

 - It needs the ability to subscribe per topic. Things are a bit noisy
 right now (ie I'm not interested in log stash)

 - eliminate "noreply" in the email notifications. I don't feel like I
 should participate via email when I see this. Give it a friendly name.

 Right now use via email feels second class. But I think it's the most
 important thing. I'm likely to scan my low priority inbox where mailing
 list emails are sent. I'm going to struggle to remember to check in on and
 participate in a forum to help folks. It's another place to go and all my
 other OSS mailing lists come to my email and I can work with them
 seemlessly. So I'm likely to forget to check or possibly not want to bother
 with elastic which uses a different system.

 Yes, I do get the notifications, but it doesn't quite feel the same as
 a mailing list for the reasons above. It feels like a notification from
 another system.

 Anyway long and frank email. Forgive the bluntness. I just wanted to
 express hopefully useful feedback. I do appreciate the thoughtfulness
 here. I know elastic and discourse folks are very smart. Email truly can
 become a first class experience and keep some of the great things about
 discourse.

 Cheers!
 Doug

 On Sunday, May 10, 2015, Ivan Brusic  wrote:

> I really do not care for the new mailing list.
>
> First of all, I can no longer see real names and email addresses. All
> I see is whatever nonsensical handle someone choose on sign up. Searching
> for Adrien no longer returns his latest posts.
>
> Second, since every email comes from nore...@discuss.elastic.co, I
> can no longer see who replied to a thread. All I see is the handle of the
> original poster. I can see the reply counts, but have no idea who the
> replies are from.
>
> Add both of these issues together, and the list has now become very
> impersonal.
>
> Cheers,
>
> Ivan
>  On May 5, 2015 10:49 AM, "Leslie Hawthorn" <
> leslie.hawth...@elastic.co> wrote:
>
>> Sadly, we cannot twibble bit to allow certain types of links but not
>> others.
>>
>> However, we can adjust the forum settings to allow users to include
>> links in their

Re: Elastic Search configuration

2015-05-12 Thread Hardik Dobariya 
No am not facing any performance issue.Just want to confirm if my 
configuration is feasible or not and what will happen if i bring down shard 
to 1.
My ES version is 1.5.2 and java release is 8.45

On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
>
> Are you having performance issues now?
>
> If so I'd start with replacing your disk setup, chances are it's not as 
> fast as you think, especially if you are storing the data from both nodes 
> on the same path.
> Also what ES version are you on, what Java release and version?
>
> On 12 May 2015 at 19:01, Hardik Dobariya  > wrote:
>
>> Please let me know if below elastic server configuration is feasible or 
>> not.
>>
>> Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple 
>> indices having total around 100 million data ,sizing 1tb and still 
>> increasing
>> --
>>
>>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>>
>> --
>>
>> All nodes are pointing to same UNC path(\\pc1\\data) data location.Data 
>> is not stored locally on nodes,all data is stored on one location as 
>> mentioned above.we are using Nas to store data and seems Nas is very fast 
>> so we are storing data on UNC path and not locally.
>>
>> While googling found if shards are decreased ,it will increase search 
>> performance.According to the specified configuration should i drop no of 
>> shards to 1?.Will it increase the performance or does the configuration 
>> need any changes?.
>> Please suggest me the configuration.
>> Thanks
>>
>> -- 
>> Please update your bookmarks! We have moved to 
>> https://discuss.elastic.co/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

2015-05-12 Thread Mark Walkom 
Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as
fast as you think, especially if you are storing the data from both nodes
on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya 
wrote:

> Please let me know if below elastic server configuration is feasible or
> not.
>
> Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple
> indices having total around 100 million data ,sizing 1tb and still
> increasing
> --
>
>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>
> --
>
> All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is
> not stored locally on nodes,all data is stored on one location as mentioned
> above.we are using Nas to store data and seems Nas is very fast so we are
> storing data on UNC path and not locally.
>
> While googling found if shards are decreased ,it will increase search
> performance.According to the specified configuration should i drop no of
> shards to 1?.Will it increase the performance or does the configuration
> need any changes?.
> Please suggest me the configuration.
> Thanks
>
> --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_7DjNa%2B82_B-v95%2BW1HbSs-VsUC%3DHxa8OuRLOhJCUV-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch is not able to search for Nonnglish text present in PDF type of attachment

2015-05-12 Thread Prashant Agrawal 
Hi Team,

We are facing an issue while searching the Non English text indexed as PDF
type of document. Below are the complete details.

1) I am having a pdf document as New_Pdf_issue.pdf which is attached in this
mail.
2) Created an indexing request alongwith mapping as well which is attached
as pdf_index_issue.sh
3) Now if you will look onto pdf attachment you will find keywords such as
"अधिकार", so if i am searching as "अधिकार" I am not able to get any matching
documents for the same.

Note : What we observed is like when we perform search query as 
{
  "fields": [
"SessionAtt.content_type",
"SessionAtt"
  ],
  "query": {
"bool": {
  "must": [
{
  "query_string": {
"fields": [
  "Content",
  "SessionAtt"
],
"query": "*"
  }
}
  ]
}
  }
}

We are observing as "अधिकार" words has been indexed as "अधधकार".

So can anyone let me know what could be the issue for the same.

~Prashant

pdf_index_issue.sh

  
New_Pdf_issue.pdf

  



--
View this message in context: 
http://elasticsearch-users.115913.n3.nabble.com/Elasticsearch-is-not-able-to-search-for-Nonnglish-text-present-in-PDF-type-of-attachment-tp4074717.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1431422241775-4074717.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

Re: Missing operating systems statistics on 2 nodes out of 5

2015-05-12 Thread Mark Walkom 
Are you using containers, Docker, LXC etc?

On 12 May 2015 at 18:22, Alexander Vassilevski <
alexander.vassilev...@gmail.com> wrote:

> Hey guys,
>
> I'm sorry if this is a duplicate post - I did in fact do various searches
> in the group prior to writing this, but I'd be more than glad to use a URL
> if someone has already solved this exact problem.
>
> Here's a short description of the problem:
>
> I have a cluster with 5 data nodes ( all eligible masters, with minimum 3
> ).  I also have a no data no master node which I use for load balancing
> search and indexing requests ( its the REST endpoint ).
>
> The problem is that for data nodes 4 and 5 I am getting no OS statistics (
> the ones provided by the sigar lib ).
>
> The only difference between nodes 1-3 and 4,5 are that the last two were
> installed via puppet.
> There are no significant ( meaning nothing more nothing less, just
> different values for IP's and hostnames) elasticsearch.yml config
> differences and I haven't set anything unique on the last two nodes via
> cluster permanent/transient,  node or index settings.
> I have tried restarting both all the individual nodes one by one and the
> cluster as a whole and neither of these resolved the issue.
>
> I am willing to provide config files, program output, anything, etc, so
> that we can get this issue resolved - just let me know!
>
> The elasticsearch version I'm using on all the nodes is 1.5.0, java is
>
> -Alex V
>
> --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/8a58934a-efba-4264-a5b8-b01a187efb80%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-mBGho%2BsCQ7jkmN-PrgwxqWQvhhjfcxjkbTM8tOzHhjA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elastic Search configuration

2015-05-12 Thread Hardik Dobariya 


Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple 
indices having total around 100 million data ,sizing 1tb and still 
increasing
--

   1. Node 1:Master=true,data=true (30gb ram,8 cores)
   2. Node 2:Master=true,data=true (30gb ram,8 cores)
   3. Node 3:Master=false,data=false (30gb ram,8 cores)

--

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is 
not stored locally on nodes,all data is stored on one location as mentioned 
above.we are using Nas to store data and seems Nas is very fast so we are 
storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search 
performance.According to the specified configuration should i drop no of 
shards to 1?.Will it increase the performance or does the configuration 
need any changes?.
Please suggest me the configuration.
Thanks

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Missing operating systems statistics on 2 nodes out of 5

2015-05-12 Thread Alexander Vassilevski 
Hey guys,

I'm sorry if this is a duplicate post - I did in fact do various searches 
in the group prior to writing this, but I'd be more than glad to use a URL 
if someone has already solved this exact problem.

Here's a short description of the problem:

I have a cluster with 5 data nodes ( all eligible masters, with minimum 3 
).  I also have a no data no master node which I use for load balancing 
search and indexing requests ( its the REST endpoint ).

The problem is that for data nodes 4 and 5 I am getting no OS statistics ( 
the ones provided by the sigar lib ).

The only difference between nodes 1-3 and 4,5 are that the last two were 
installed via puppet.
There are no significant ( meaning nothing more nothing less, just 
different values for IP's and hostnames) elasticsearch.yml config 
differences and I haven't set anything unique on the last two nodes via 
cluster permanent/transient,  node or index settings.
I have tried restarting both all the individual nodes one by one and the 
cluster as a whole and neither of these resolved the issue.

I am willing to provide config files, program output, anything, etc, so 
that we can get this issue resolved - just let me know!

The elasticsearch version I'm using on all the nodes is 1.5.0, java is 

-Alex V

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8a58934a-efba-4264-a5b8-b01a187efb80%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.