Re: Elastic Search configuration

[Hardik Dobariya]

Thank you Mark and Magnus for the help provided :) 

On Wednesday, May 13, 2015 at 12:46:46 PM UTC+5:30, Magnus Bäck wrote:
>
> On Wednesday, May 13, 2015 at 08:43 CEST, 
>  Hardik Dobariya > wrote: 
>
> > we have around 500 indices all sizing to approx 50gb respectively and 
> > total size goes around 1tb for all indices and will keep on increasing. 
> > The reason behind setting node3 as Master=false,data=false because in 
> > configuration file i read this type of node will only work for 
> > searching ,aggregation etc. Do i still need to set node3 three as mater 
> > and data 
>
> You'll definitely want all three nodes to be master-eligible (as Mark 
> explained this prevents a split brain situation). Whether it makes sense 
> to dedicate one of the nodes to queries depends a bit on the types of 
> queries you make but I'd say it's unlikely that dedicating a third of 
> your cluster's capacity for queries is the best use of your money. 
>
> -- 
> Magnus Bäck| Software Engineer, Development Tools 
> magnu...@sonymobile.com  | Sony Mobile Communications 
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/17866b5d-f6df-4333-8fc8-c450f8b48954%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Magnus Bäck]

On Wednesday, May 13, 2015 at 08:43 CEST,
 Hardik Dobariya  wrote:

> we have around 500 indices all sizing to approx 50gb respectively and
> total size goes around 1tb for all indices and will keep on increasing.
> The reason behind setting node3 as Master=false,data=false because in
> configuration file i read this type of node will only work for
> searching ,aggregation etc. Do i still need to set node3 three as mater
> and data

You'll definitely want all three nodes to be master-eligible (as Mark
explained this prevents a split brain situation). Whether it makes sense
to dedicate one of the nodes to queries depends a bit on the types of
queries you make but I'd say it's unlikely that dedicating a third of
your cluster's capacity for queries is the best use of your money.

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150513071628.GA13466%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Hardik Dobariya]

we have around 500 indices all sizing to approx 50gb respectively and total 
size goes around 1tb for all indices and will keep on increasing .
The reason behind setting node3 as Master=false,data=false because in 
configuration file i read this type of node will only work for searching 
,aggregation etc. Do i still need to set node3 three as mater and data

On Wednesday, May 13, 2015 at 1:49:18 AM UTC+5:30, Magnus Bäck wrote:
>
> On Tuesday, May 12, 2015 at 14:24 CEST, 
>  Hardik Dobariya > wrote: 
>
> > oh yes i forgot to mentioned that we are using SAN mounted volumes. 
>
> Um, okay. But you said you accessed them via a UNC path (\\pc1\data). 
> Then it's not a locally mounted volume. 
>
> > And according to your reply i can say my configuration will work if am 
> > using SAN mounted volumes? 
>
> Yes. SAN and NAS both work but SAN is preferred. 
>
> > one last question if my configuration is correct then do i have to 
> > drop down my shard size from 5 to 1 or to something less to 3 which 
> > might increase performance? 
>
> There are rarely any "correct" configurations, and what's optimal 
> depends on many factors. While there are rules of thumb you'd have 
> to measure for yourself what's best for you. 
>
> The optimal number of shards depends on how many indexes you have, 
> how many nodes you have (that we know; three) how big the indexes 
> are etc. 
>
> Shards have a certain overhead so you don't want to have too many 
> (you'll waste resources to shard book keeping) but you don't want 
> them to be too big either and you'll want to spread them out between 
> your nodes. 
>
> > These are the three nodes configured 
> >  1. Node 1:Master=true,data=true (30gb ram,8 cores) 
> >  2. Node 2:Master=true,data=true (30gb ram,8 cores) 
> >  3. Node 3:Master=false,data=false (30gb ram,8 cores) 
>
> Yes, and again, you should turn node 3 into master=true, data=true 
> as well. 
>
> -- 
> Magnus Bäck| Software Engineer, Development Tools 
> magnu...@sonymobile.com  | Sony Mobile Communications 
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bcc44119-1521-4ef5-8024-73eee4947537%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Magnus Bäck]

On Tuesday, May 12, 2015 at 14:24 CEST,
 Hardik Dobariya  wrote:

> oh yes i forgot to mentioned that we are using SAN mounted volumes.

Um, okay. But you said you accessed them via a UNC path (\\pc1\data).
Then it's not a locally mounted volume.

> And according to your reply i can say my configuration will work if am
> using SAN mounted volumes?

Yes. SAN and NAS both work but SAN is preferred.

> one last question if my configuration is correct then do i have to
> drop down my shard size from 5 to 1 or to something less to 3 which
> might increase performance?

There are rarely any "correct" configurations, and what's optimal
depends on many factors. While there are rules of thumb you'd have
to measure for yourself what's best for you.

The optimal number of shards depends on how many indexes you have,
how many nodes you have (that we know; three) how big the indexes
are etc.

Shards have a certain overhead so you don't want to have too many
(you'll waste resources to shard book keeping) but you don't want
them to be too big either and you'll want to spread them out between
your nodes.

> These are the three nodes configured
>  1. Node 1:Master=true,data=true (30gb ram,8 cores)
>  2. Node 2:Master=true,data=true (30gb ram,8 cores)
>  3. Node 3:Master=false,data=false (30gb ram,8 cores)

Yes, and again, you should turn node 3 into master=true, data=true
as well.

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150512201910.GB2679%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Hardik Dobariya]

oh yes i forgot to mentioned that we are using SAN mounted volumes.
And according to your reply i can say my configuration will work if am 
using SAN mounted volumes?
one last question if my configuration is correct then do i have to drop 
down my shard size from 5 to 1 or to something less to 3 which might 
increase performance?
These are the three nodes configured


   1. Node 1:Master=true,data=true (30gb ram,8 cores)
   2. Node 2:Master=true,data=true (30gb ram,8 cores)
   3. Node 3:Master=false,data=false (30gb ram,8 cores)


On Tuesday, May 12, 2015 at 5:46:54 PM UTC+5:30, Magnus Bäck wrote:
>
> On Tuesday, May 12, 2015 at 13:24 CEST, 
>  Hardik Dobariya > wrote: 
>
> > yes i can understand the use of network file system is not good. 
> > Actually we do not have any physical machines.we are using virtual 
> > environment.This is the reason we are using NAS to store data on 
> > network. 
>
> Sure, but you don't have to use NAS just because you use VMs. 
> Local disks or volumes mounted from a SAN are still preferred 
> to accessing the file system over SMB or NFS. 
>
> > Any suggestion on using virtual environment for elastic search? 
> > and yes am already using 3 nodes.2 masters with data and one child 
>
> Yes, and as Mark says that's a bad idea. You're better off allowing 
> all three nodes to be masters. Unless you have a serious query load 
> you should keep data on all three nodes. Having a third of the 
> cluster's capacity dedicated to processing queries is most likely 
> very wasteful. 
>
> -- 
> Magnus Bäck| Software Engineer, Development Tools 
> magnu...@sonymobile.com  | Sony Mobile Communications 
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/8bb923dd-67e2-457d-982b-f38c8d372c0e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Magnus Bäck]

On Tuesday, May 12, 2015 at 13:24 CEST,
 Hardik Dobariya  wrote:

> yes i can understand the use of network file system is not good.
> Actually we do not have any physical machines.we are using virtual
> environment.This is the reason we are using NAS to store data on
> network.

Sure, but you don't have to use NAS just because you use VMs.
Local disks or volumes mounted from a SAN are still preferred
to accessing the file system over SMB or NFS.

> Any suggestion on using virtual environment for elastic search?
> and yes am already using 3 nodes.2 masters with data and one child

Yes, and as Mark says that's a bad idea. You're better off allowing
all three nodes to be masters. Unless you have a serious query load
you should keep data on all three nodes. Having a third of the
cluster's capacity dedicated to processing queries is most likely
very wasteful.

-- 
Magnus Bäck| Software Engineer, Development Tools
magnus.b...@sonymobile.com | Sony Mobile Communications

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/20150512121645.GA806%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Hardik Dobariya]

yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual 
environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?
and yes am already using 3 nodes.2 masters with data and one child 

On Tuesday, May 12, 2015 at 4:52:25 PM UTC+5:30, Hardik Dobariya wrote:
>
> yes i can understand the use of network file system is not good.
> Actually we do not have any physical machines.we are using virtual 
> environment.This is the reason we are using NAS to store data on network.
> Any suggestion on using virtual environment for elastic search?
>
> On Tuesday, May 12, 2015 at 4:39:15 PM UTC+5:30, Mark Walkom wrote:
>>
>> We do not recommend the use of networked file systems.
>> We definitely don't recommend using those and putting both sets of data 
>> in the one location.
>>
>> Reducing shard count may help, it'd certainly reduce the overhead in 
>> maintaining all those shards. Just keep shard sizes under 50GB if you can.
>> Also having two masters can lead to potential split brain, you'd be 
>> better off making node 3 a master: true, data: false and setting 
>> minimum_masters.
>>
>> On 12 May 2015 at 20:22, Hardik Dobariya  wrote:
>>
>>> No am not facing any performance issue.Just want to confirm if my 
>>> configuration is feasible or not and what will happen if i bring down shard 
>>> to 1.
>>> My ES version is 1.5.2 and java release is 8.45
>>>
>>> On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:

 Are you having performance issues now?

 If so I'd start with replacing your disk setup, chances are it's not as 
 fast as you think, especially if you are storing the data from both nodes 
 on the same path.
 Also what ES version are you on, what Java release and version?

 On 12 May 2015 at 19:01, Hardik Dobariya  
 wrote:

> Please let me know if below elastic server configuration is feasible 
> or not.
>
> Total 1 cluster with 3 nodes having 5 shards and 1 replica with 
> multiple indices having total around 100 million data ,sizing 1tb and 
> still 
> increasing
> --
>
>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>
> --
>
> All nodes are pointing to same UNC path(\\pc1\\data) data 
> location.Data is not stored locally on nodes,all data is stored on one 
> location as mentioned above.we are using Nas to store data and seems Nas 
> is 
> very fast so we are storing data on UNC path and not locally.
>
> While googling found if shards are decreased ,it will increase search 
> performance.According to the specified configuration should i drop no of 
> shards to 1?.Will it increase the performance or does the configuration 
> need any changes?.
> Please suggest me the configuration.
> Thanks
>
> -- 
> Please update your bookmarks! We have moved to 
> https://discuss.elastic.co/
> --- 
> You received this message because you are subscribed to the Google 
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to elasticsearc...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

  -- 
>>> Please update your bookmarks! We have moved to 
>>> https://discuss.elastic.co/
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elas

Re: Elastic Search configuration

[Hardik Dobariya]

yes i can understand the use of network file system is not good.
Actually we do not have any physical machines.we are using virtual 
environment.This is the reason we are using NAS to store data on network.
Any suggestion on using virtual environment for elastic search?

On Tuesday, May 12, 2015 at 4:39:15 PM UTC+5:30, Mark Walkom wrote:
>
> We do not recommend the use of networked file systems.
> We definitely don't recommend using those and putting both sets of data in 
> the one location.
>
> Reducing shard count may help, it'd certainly reduce the overhead in 
> maintaining all those shards. Just keep shard sizes under 50GB if you can.
> Also having two masters can lead to potential split brain, you'd be better 
> off making node 3 a master: true, data: false and setting minimum_masters.
>
> On 12 May 2015 at 20:22, Hardik Dobariya  > wrote:
>
>> No am not facing any performance issue.Just want to confirm if my 
>> configuration is feasible or not and what will happen if i bring down shard 
>> to 1.
>> My ES version is 1.5.2 and java release is 8.45
>>
>> On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
>>>
>>> Are you having performance issues now?
>>>
>>> If so I'd start with replacing your disk setup, chances are it's not as 
>>> fast as you think, especially if you are storing the data from both nodes 
>>> on the same path.
>>> Also what ES version are you on, what Java release and version?
>>>
>>> On 12 May 2015 at 19:01, Hardik Dobariya  wrote:
>>>
 Please let me know if below elastic server configuration is feasible or 
 not.

 Total 1 cluster with 3 nodes having 5 shards and 1 replica with 
 multiple indices having total around 100 million data ,sizing 1tb and 
 still 
 increasing
 --

1. Node 1:Master=true,data=true (30gb ram,8 cores)
2. Node 2:Master=true,data=true (30gb ram,8 cores)
3. Node 3:Master=false,data=false (30gb ram,8 cores)

 --

 All nodes are pointing to same UNC path(\\pc1\\data) data location.Data 
 is not stored locally on nodes,all data is stored on one location as 
 mentioned above.we are using Nas to store data and seems Nas is very fast 
 so we are storing data on UNC path and not locally.

 While googling found if shards are decreased ,it will increase search 
 performance.According to the specified configuration should i drop no of 
 shards to 1?.Will it increase the performance or does the configuration 
 need any changes?.
 Please suggest me the configuration.
 Thanks

 -- 
 Please update your bookmarks! We have moved to 
 https://discuss.elastic.co/
 --- 
 You received this message because you are subscribed to the Google 
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to elasticsearc...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>  -- 
>> Please update your bookmarks! We have moved to 
>> https://discuss.elastic.co/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/66346979-7579-41a0-9e9c-d3650bf81227%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Mark Walkom]

We do not recommend the use of networked file systems.
We definitely don't recommend using those and putting both sets of data in
the one location.

Reducing shard count may help, it'd certainly reduce the overhead in
maintaining all those shards. Just keep shard sizes under 50GB if you can.
Also having two masters can lead to potential split brain, you'd be better
off making node 3 a master: true, data: false and setting minimum_masters.

On 12 May 2015 at 20:22, Hardik Dobariya 
wrote:

> No am not facing any performance issue.Just want to confirm if my
> configuration is feasible or not and what will happen if i bring down shard
> to 1.
> My ES version is 1.5.2 and java release is 8.45
>
> On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
>>
>> Are you having performance issues now?
>>
>> If so I'd start with replacing your disk setup, chances are it's not as
>> fast as you think, especially if you are storing the data from both nodes
>> on the same path.
>> Also what ES version are you on, what Java release and version?
>>
>> On 12 May 2015 at 19:01, Hardik Dobariya  wrote:
>>
>>> Please let me know if below elastic server configuration is feasible or
>>> not.
>>>
>>> Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple
>>> indices having total around 100 million data ,sizing 1tb and still
>>> increasing
>>> --
>>>
>>>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>>>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>>>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>>>
>>> --
>>>
>>> All nodes are pointing to same UNC path(\\pc1\\data) data location.Data
>>> is not stored locally on nodes,all data is stored on one location as
>>> mentioned above.we are using Nas to store data and seems Nas is very fast
>>> so we are storing data on UNC path and not locally.
>>>
>>> While googling found if shards are decreased ,it will increase search
>>> performance.According to the specified configuration should i drop no of
>>> shards to 1?.Will it increase the performance or does the configuration
>>> need any changes?.
>>> Please suggest me the configuration.
>>> Thanks
>>>
>>> --
>>> Please update your bookmarks! We have moved to
>>> https://discuss.elastic.co/
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9qjR17AOO4hinjxbvAKYO5L5tttPncz_fPL-ZQ5mGCKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Hardik Dobariya]

No am not facing any performance issue.Just want to confirm if my 
configuration is feasible or not and what will happen if i bring down shard 
to 1.
My ES version is 1.5.2 and java release is 8.45

On Tuesday, May 12, 2015 at 3:38:40 PM UTC+5:30, Mark Walkom wrote:
>
> Are you having performance issues now?
>
> If so I'd start with replacing your disk setup, chances are it's not as 
> fast as you think, especially if you are storing the data from both nodes 
> on the same path.
> Also what ES version are you on, what Java release and version?
>
> On 12 May 2015 at 19:01, Hardik Dobariya  > wrote:
>
>> Please let me know if below elastic server configuration is feasible or 
>> not.
>>
>> Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple 
>> indices having total around 100 million data ,sizing 1tb and still 
>> increasing
>> --
>>
>>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>>
>> --
>>
>> All nodes are pointing to same UNC path(\\pc1\\data) data location.Data 
>> is not stored locally on nodes,all data is stored on one location as 
>> mentioned above.we are using Nas to store data and seems Nas is very fast 
>> so we are storing data on UNC path and not locally.
>>
>> While googling found if shards are decreased ,it will increase search 
>> performance.According to the specified configuration should i drop no of 
>> shards to 1?.Will it increase the performance or does the configuration 
>> need any changes?.
>> Please suggest me the configuration.
>> Thanks
>>
>> -- 
>> Please update your bookmarks! We have moved to 
>> https://discuss.elastic.co/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ecbaf990-27d2-46f0-8ae5-2aba7a012b85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elastic Search configuration

[Mark Walkom]

Are you having performance issues now?

If so I'd start with replacing your disk setup, chances are it's not as
fast as you think, especially if you are storing the data from both nodes
on the same path.
Also what ES version are you on, what Java release and version?

On 12 May 2015 at 19:01, Hardik Dobariya 
wrote:

> Please let me know if below elastic server configuration is feasible or
> not.
>
> Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple
> indices having total around 100 million data ,sizing 1tb and still
> increasing
> --
>
>1. Node 1:Master=true,data=true (30gb ram,8 cores)
>2. Node 2:Master=true,data=true (30gb ram,8 cores)
>3. Node 3:Master=false,data=false (30gb ram,8 cores)
>
> --
>
> All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is
> not stored locally on nodes,all data is stored on one location as mentioned
> above.we are using Nas to store data and seems Nas is very fast so we are
> storing data on UNC path and not locally.
>
> While googling found if shards are decreased ,it will increase search
> performance.According to the specified configuration should i drop no of
> shards to 1?.Will it increase the performance or does the configuration
> need any changes?.
> Please suggest me the configuration.
> Thanks
>
> --
> Please update your bookmarks! We have moved to https://discuss.elastic.co/
> ---
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_7DjNa%2B82_B-v95%2BW1HbSs-VsUC%3DHxa8OuRLOhJCUV-Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elastic Search configuration

[Hardik Dobariya]

Please let me know if below elastic server configuration is feasible or not.

Total 1 cluster with 3 nodes having 5 shards and 1 replica with multiple 
indices having total around 100 million data ,sizing 1tb and still 
increasing
--

   1. Node 1:Master=true,data=true (30gb ram,8 cores)
   2. Node 2:Master=true,data=true (30gb ram,8 cores)
   3. Node 3:Master=false,data=false (30gb ram,8 cores)

--

All nodes are pointing to same UNC path(\\pc1\\data) data location.Data is 
not stored locally on nodes,all data is stored on one location as mentioned 
above.we are using Nas to store data and seems Nas is very fast so we are 
storing data on UNC path and not locally.

While googling found if shards are decreased ,it will increase search 
performance.According to the specified configuration should i drop no of 
shards to 1?.Will it increase the performance or does the configuration 
need any changes?.
Please suggest me the configuration.
Thanks

-- 
Please update your bookmarks! We have moved to https://discuss.elastic.co/
--- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5d223976-a535-4ccf-a9e1-f1d0855015ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Request for elastic search configuration

[David Pilato]

Are you looking for this? 
http://www.elasticsearch.org/tutorials/deploying-elasticsearch-with-chef-solo/

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 19 déc. 2013 à 11:44, shailendra.path...@gmail.com a écrit :

Hi Srikanth,and all elasticsearch developers.

i need  your  help regarding deploying ES server using chef solo..i have made 
one ES instance ..i have 7 servers .can you please provide full procedures for 
the same..


Thanks

> On Thursday, December 19, 2013 6:23:45 AM UTC+5:30, srikanth ramineni wrote:
> Hi Team,
> 
> 
> We have below server configurations.  We are trying to load more than 500 
> million data into elastic. When data reaches 40 millions, shards are going 
> down and data is deleting . Can you please help us with best configuration 
> for above requirement.
> 
> Below are the server details
> 
> we have  4  nodes.
> 
> Each node contains 
> CPU: 4 , RAM: 16G ,  Storage: 122G
> ES_HEAP_SIZE 8 G
> 
> 
> 
> Thanks,
> Srikanth.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4e65841e-3edc-4a1a-b16a-dbf2c265a27c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/BA543D32-8882-4978-86AA-0FBC16EF77C1%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Request for elastic search configuration

[shailendra . pathak9]

Hi Srikanth,and all elasticsearch developers.

i need  your  help regarding deploying ES server using chef solo..i have 
made one ES instance ..i have 7 servers .can you please provide full 
procedures for the same..


Thanks

On Thursday, December 19, 2013 6:23:45 AM UTC+5:30, srikanth ramineni wrote:
>
> Hi Team,
>
>
> We have below server configurations.  We are trying to load more than 500 
> million data into elastic. When data reaches 40 millions, shards are going 
> down and data is deleting . Can you please help us with best configuration 
> for above requirement.
>
> Below are the server details
>
> we have  4  nodes.
>
> Each node contains 
> CPU: 4 , RAM: 16G ,  Storage: 122G
> ES_HEAP_SIZE 8 G
>
>
>
> Thanks,
> Srikanth.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4e65841e-3edc-4a1a-b16a-dbf2c265a27c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Request for elastic search configuration

[David Pilato]

Anything in logs?

--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 19 déc. 2013 à 05:30, srikanth ramineni  a écrit :

Hi,

We installed head plugin for monitoring the cluster. once it loaded  27GB,  
cluster is going to unavailable. We are not able to understand exact problem..

Can you please suggest  elasticsearch.yml file configurations for 4 nodes for 
single cluster. we are loading data from oracle to elastic by using java 
threads..


Thanks,
Srikanth.


> On Wed, Dec 18, 2013 at 8:01 PM, Mark Walkom  
> wrote:
> You will want to use Oracle java rather than OpenJDK, you will get a lot 
> better performance.
> 
> Now next question, what are you using for monitoring your cluster? Are you 
> using plugins like ElasticHQ, kopf, bighead, head, paramedic etc etc. If not 
> install one of those and it will give you insight into what is happening with 
> your cluster and what possible causes for the problems might be.
> 
> Regards,
> Mark Walkom
> 
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
> 
> 
>> On 19 December 2013 14:49, srikanth ramineni  wrote:
>> Hi Mark Walkom,
>> 
>> I have updated the detail in  pastebin . Pleae find below url for details.
>> 
>> http://pastebin.com/62aPBkPc
>> 
>> 
>> Please get back to me for further quries.
>> 
>> thanks,
>> Srikanth.
>> 
>> 
>>> On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom  
>>> wrote:
>>> http://pastebin.com/ - Please don't send attachments to the list, it goes 
>>> to hundreds/thousands of people.
>>> 
>>> What JDK is it, oracle or open?
>>> What sort of data is it - logs? Web pages? Text?
>>> 
>>> Regards,
>>> Mark Walkom
>>> 
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>> 
>>> 
 On 19 December 2013 14:18, srikanth ramineni  wrote:
 Hi Mark Walkom,
 
 
 Thanks for your quick response...  
 
 Sorry , I am not sure how to post from gist/pastebin/etc.   Below are your 
 requested details
 
 Operating system : Linux
 ES version  : elastic search-0.90.7
 JAVA Version :JDK 1.7
 Data : we didn't have any parent child relations. Just we are loading with 
 auto generated primary key.(29,666,482 documents=  21.9 GB).
 
 Data Loading : we are loading data into elastic by writing 20 threads. 
 
 Client creation :
 
  Settings settings = 
 ImmutableSettings.settingsBuilder().put("cluster.name", 
 "UCS_DEV_ES_CLUSTER").build();
 
 elasticClient = new 
 TransportClient(settings).addTransportAddress(new 
 InetSocketTransportAddress("server1", 9300))

  .addTransportAddress(new 
 InetSocketTransportAddress("server2", 9300));

 Below are the server details
 
 we have  4  nodes.
 
 Each node contains 
 CPU: 4 , RAM: 16G ,  Storage: 122G
 ES_HEAP_SIZE 8 G   
 
 
 Please find enclosed  elasticsearch.yml file which we are using.
 
 
 
 
 
 
 
 
 
 
 
 Thanks,
 Srikanth.  
  
 
 
 
> On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom  
> wrote:
> It will help if you post (via a gist/pastebin/etc) what you have now.
> 
> Also what sort of data is it, how are you loading it, what versions of 
> things are you running on (OS, java, ES, etc etc)
> 
> Regards,
> Mark Walkom
> 
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
> 
> 
>> On 19 December 2013 11:53, srikanth ramineni  
>> wrote:
>> Hi Team,
>> 
>> 
>> We have below server configurations.  We are trying to load more than 
>> 500 million data into elastic. When data reaches 40 millions, shards are 
>> going down and data is deleting . Can you please help us with best 
>> configuration for above requirement.
>> 
>> Below are the server details
>> 
>> we have  4  nodes.
>> 
>> Each node contains 
>> CPU: 4 , RAM: 16G ,  Storage: 122G
>> ES_HEAP_SIZE 8 G
>> 
>> 
>> 
>> Thanks,
>> Srikanth.
>> 
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to elasticsearch+unsubscr...@googlegroups.com.
>> 
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
> 
> -- 

Re: Request for elastic search configuration

[srikanth ramineni]

Hi,

We installed head plugin for monitoring the cluster. once it loaded  27GB,
 cluster is going to unavailable. We are not able to understand exact
problem..

Can you please suggest  elasticsearch.yml file configurations for 4 nodes
for single cluster. we are loading data from oracle to elastic by using
java threads..


Thanks,
Srikanth.


On Wed, Dec 18, 2013 at 8:01 PM, Mark Walkom wrote:

> You will want to use Oracle java rather than OpenJDK, you will get a lot
> better performance.
>
> Now next question, what are you using for monitoring your cluster? Are you
> using plugins like ElasticHQ, kopf, bighead, head, paramedic etc etc. If
> not install one of those and it will give you insight into what is
> happening with your cluster and what possible causes for the problems might
> be.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 19 December 2013 14:49, srikanth ramineni wrote:
>
>> Hi Mark Walkom,
>>
>> I have updated the detail in  pastebin . Pleae find below url for details.
>>
>> http://pastebin.com/62aPBkPc
>>
>>
>> Please get back to me for further quries.
>>
>> thanks,
>> Srikanth.
>>
>>
>> On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom 
>> wrote:
>>
>>> http://pastebin.com/ - Please don't send attachments to the list, it
>>> goes to hundreds/thousands of people.
>>>
>>> What JDK is it, oracle or open?
>>> What sort of data is it - logs? Web pages? Text?
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 19 December 2013 14:18, srikanth ramineni wrote:
>>>
 Hi Mark Walkom,


 Thanks for your quick response...

 Sorry , I am not sure how to post from gist/pastebin/etc.   Below are
 your requested details

 Operating system : Linux
 ES version  : elastic search-0.90.7
 JAVA Version :JDK 1.7
 Data : we didn't have any parent child relations. Just we are loading
 with auto generated primary key.(29,666,482 documents=  21.9 GB).

 Data Loading : we are loading data into elastic by writing 20 threads.

 Client creation :

  Settings settings = ImmutableSettings.settingsBuilder().put("
 cluster.name", "UCS_DEV_ES_CLUSTER").build();

  elasticClient = new TransportClient(settings).addTransportAddress(new
 InetSocketTransportAddress("server1", 9300))
  .addTransportAddress(new InetSocketTransportAddress("server2", 9300));
  Below are the server details

 we have  4  nodes.

 Each node contains
 CPU: 4 , RAM: 16G ,  Storage: 122G
 ES_HEAP_SIZE 8 G

 Please find enclosed elasticsearch.yml file which we are using.




 Thanks,
 Srikanth.



 On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom >>> > wrote:

> It will help if you post (via a gist/pastebin/etc) what you have now.
>
> Also what sort of data is it, how are you loading it, what versions of
> things are you running on (OS, java, ES, etc etc)
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 19 December 2013 11:53, srikanth ramineni wrote:
>
>> Hi Team,
>>
>>
>> We have below server configurations.  We are trying to load more than
>> 500 million data into elastic. When data reaches 40 millions, shards are
>> going down and data is deleting . Can you please help us with best
>> configuration for above requirement.
>>
>> Below are the server details
>>
>> we have  4  nodes.
>>
>> Each node contains
>> CPU: 4 , RAM: 16G ,  Storage: 122G
>> ES_HEAP_SIZE 8 G
>>
>>
>>
>> Thanks,
>> Srikanth.
>>
>>  --
>> You received this message because you are subscribed to the Google
>> Groups "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to elasticsearch+unsubscr...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3

Re: Request for elastic search configuration

[Mark Walkom]

You will want to use Oracle java rather than OpenJDK, you will get a lot
better performance.

Now next question, what are you using for monitoring your cluster? Are you
using plugins like ElasticHQ, kopf, bighead, head, paramedic etc etc. If
not install one of those and it will give you insight into what is
happening with your cluster and what possible causes for the problems might
be.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 19 December 2013 14:49, srikanth ramineni  wrote:

> Hi Mark Walkom,
>
> I have updated the detail in  pastebin . Pleae find below url for details.
>
> http://pastebin.com/62aPBkPc
>
>
> Please get back to me for further quries.
>
> thanks,
> Srikanth.
>
>
> On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom wrote:
>
>> http://pastebin.com/ - Please don't send attachments to the list, it
>> goes to hundreds/thousands of people.
>>
>> What JDK is it, oracle or open?
>> What sort of data is it - logs? Web pages? Text?
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 19 December 2013 14:18, srikanth ramineni wrote:
>>
>>> Hi Mark Walkom,
>>>
>>>
>>> Thanks for your quick response...
>>>
>>> Sorry , I am not sure how to post from gist/pastebin/etc.   Below are
>>> your requested details
>>>
>>> Operating system : Linux
>>> ES version  : elastic search-0.90.7
>>> JAVA Version :JDK 1.7
>>> Data : we didn't have any parent child relations. Just we are loading
>>> with auto generated primary key.(29,666,482 documents=  21.9 GB).
>>>
>>> Data Loading : we are loading data into elastic by writing 20 threads.
>>>
>>> Client creation :
>>>
>>>  Settings settings = ImmutableSettings.settingsBuilder().put("
>>> cluster.name", "UCS_DEV_ES_CLUSTER").build();
>>>
>>>  elasticClient = new TransportClient(settings).addTransportAddress(new
>>> InetSocketTransportAddress("server1", 9300))
>>>  .addTransportAddress(new InetSocketTransportAddress("server2", 9300));
>>>  Below are the server details
>>>
>>> we have  4  nodes.
>>>
>>> Each node contains
>>> CPU: 4 , RAM: 16G ,  Storage: 122G
>>> ES_HEAP_SIZE 8 G
>>>
>>> Please find enclosed elasticsearch.yml file which we are using.
>>>
>>>
>>>
>>>
>>> Thanks,
>>> Srikanth.
>>>
>>>
>>>
>>> On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom 
>>> wrote:
>>>
 It will help if you post (via a gist/pastebin/etc) what you have now.

 Also what sort of data is it, how are you loading it, what versions of
 things are you running on (OS, java, ES, etc etc)

 Regards,
 Mark Walkom

 Infrastructure Engineer
 Campaign Monitor
 email: ma...@campaignmonitor.com
 web: www.campaignmonitor.com


 On 19 December 2013 11:53, srikanth ramineni wrote:

> Hi Team,
>
>
> We have below server configurations.  We are trying to load more than
> 500 million data into elastic. When data reaches 40 millions, shards are
> going down and data is deleting . Can you please help us with best
> configuration for above requirement.
>
> Below are the server details
>
> we have  4  nodes.
>
> Each node contains
> CPU: 4 , RAM: 16G ,  Storage: 122G
> ES_HEAP_SIZE 8 G
>
>
>
> Thanks,
> Srikanth.
>
>  --
> You received this message because you are subscribed to the Google
> Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to elasticsearch+unsubscr...@googlegroups.com.
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>

  --
 You received this message because you are subscribed to a topic in the
 Google Groups "elasticsearch" group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe
 .
 To unsubscribe from this group and all its topics, send an email to
 elasticsearch+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
 .

 For more options, visit https://groups.google.com/groups/opt_out.

>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
>>> .
>>>
>>> For mo

Re: Request for elastic search configuration

[srikanth ramineni]

Hi Mark Walkom,

I have updated the detail in  pastebin . Pleae find below url for details.

http://pastebin.com/62aPBkPc


Please get back to me for further quries.

thanks,
Srikanth.


On Wed, Dec 18, 2013 at 7:28 PM, Mark Walkom wrote:

> http://pastebin.com/ - Please don't send attachments to the list, it goes
> to hundreds/thousands of people.
>
> What JDK is it, oracle or open?
> What sort of data is it - logs? Web pages? Text?
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 19 December 2013 14:18, srikanth ramineni wrote:
>
>> Hi Mark Walkom,
>>
>>
>> Thanks for your quick response...
>>
>> Sorry , I am not sure how to post from gist/pastebin/etc.   Below are
>> your requested details
>>
>> Operating system : Linux
>> ES version  : elastic search-0.90.7
>> JAVA Version :JDK 1.7
>> Data : we didn't have any parent child relations. Just we are loading
>> with auto generated primary key.(29,666,482 documents=  21.9 GB).
>>
>> Data Loading : we are loading data into elastic by writing 20 threads.
>>
>> Client creation :
>>
>>  Settings settings = ImmutableSettings.settingsBuilder().put("
>> cluster.name", "UCS_DEV_ES_CLUSTER").build();
>>
>>  elasticClient = new TransportClient(settings).addTransportAddress(new
>> InetSocketTransportAddress("server1", 9300))
>>  .addTransportAddress(new InetSocketTransportAddress("server2", 9300));
>>  Below are the server details
>>
>> we have  4  nodes.
>>
>> Each node contains
>> CPU: 4 , RAM: 16G ,  Storage: 122G
>> ES_HEAP_SIZE 8 G
>>
>> Please find enclosed elasticsearch.yml file which we are using.
>>
>>
>>
>>
>> Thanks,
>> Srikanth.
>>
>>
>>
>> On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom 
>> wrote:
>>
>>> It will help if you post (via a gist/pastebin/etc) what you have now.
>>>
>>> Also what sort of data is it, how are you loading it, what versions of
>>> things are you running on (OS, java, ES, etc etc)
>>>
>>> Regards,
>>> Mark Walkom
>>>
>>> Infrastructure Engineer
>>> Campaign Monitor
>>> email: ma...@campaignmonitor.com
>>> web: www.campaignmonitor.com
>>>
>>>
>>> On 19 December 2013 11:53, srikanth ramineni wrote:
>>>
 Hi Team,


 We have below server configurations.  We are trying to load more than
 500 million data into elastic. When data reaches 40 millions, shards are
 going down and data is deleting . Can you please help us with best
 configuration for above requirement.

 Below are the server details

 we have  4  nodes.

 Each node contains
 CPU: 4 , RAM: 16G ,  Storage: 122G
 ES_HEAP_SIZE 8 G



 Thanks,
 Srikanth.

  --
 You received this message because you are subscribed to the Google
 Groups "elasticsearch" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to elasticsearch+unsubscr...@googlegroups.com.

 To view this discussion on the web visit
 https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
 .
 For more options, visit https://groups.google.com/groups/opt_out.

>>>
>>>  --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
>>> .
>>>
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
>> .
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups

Re: Request for elastic search configuration

[Mark Walkom]

http://pastebin.com/ - Please don't send attachments to the list, it goes
to hundreds/thousands of people.

What JDK is it, oracle or open?
What sort of data is it - logs? Web pages? Text?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 19 December 2013 14:18, srikanth ramineni  wrote:

> Hi Mark Walkom,
>
>
> Thanks for your quick response...
>
> Sorry , I am not sure how to post from gist/pastebin/etc.   Below are your
> requested details
>
> Operating system : Linux
> ES version  : elastic search-0.90.7
> JAVA Version :JDK 1.7
> Data : we didn't have any parent child relations. Just we are loading with
> auto generated primary key.(29,666,482 documents=  21.9 GB).
>
> Data Loading : we are loading data into elastic by writing 20 threads.
>
> Client creation :
>
>  Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
> "UCS_DEV_ES_CLUSTER").build();
>
>  elasticClient = new TransportClient(settings).addTransportAddress(new
> InetSocketTransportAddress("server1", 9300))
>  .addTransportAddress(new InetSocketTransportAddress("server2", 9300));
>  Below are the server details
>
> we have  4  nodes.
>
> Each node contains
> CPU: 4 , RAM: 16G ,  Storage: 122G
> ES_HEAP_SIZE 8 G
>
> Please find enclosed elasticsearch.yml file which we are using.
>
>
>
>
> Thanks,
> Srikanth.
>
>
>
> On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom wrote:
>
>> It will help if you post (via a gist/pastebin/etc) what you have now.
>>
>> Also what sort of data is it, how are you loading it, what versions of
>> things are you running on (OS, java, ES, etc etc)
>>
>> Regards,
>> Mark Walkom
>>
>> Infrastructure Engineer
>> Campaign Monitor
>> email: ma...@campaignmonitor.com
>> web: www.campaignmonitor.com
>>
>>
>> On 19 December 2013 11:53, srikanth ramineni wrote:
>>
>>> Hi Team,
>>>
>>>
>>> We have below server configurations.  We are trying to load more than
>>> 500 million data into elastic. When data reaches 40 millions, shards are
>>> going down and data is deleting . Can you please help us with best
>>> configuration for above requirement.
>>>
>>> Below are the server details
>>>
>>> we have  4  nodes.
>>>
>>> Each node contains
>>> CPU: 4 , RAM: 16G ,  Storage: 122G
>>> ES_HEAP_SIZE 8 G
>>>
>>>
>>>
>>> Thanks,
>>> Srikanth.
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
>>> .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
>> .
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624b2JyfYmC5_mZkz%2BXQe_4hzrr79ALx3Hj0ww6%3DTeqHOHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Request for elastic search configuration

[srikanth ramineni]

Hi Mark Walkom,


Thanks for your quick response...

Sorry , I am not sure how to post from gist/pastebin/etc.   Below are your
requested details

Operating system : Linux
ES version  : elastic search-0.90.7
JAVA Version :JDK 1.7
Data : we didn't have any parent child relations. Just we are loading with
auto generated primary key.(29,666,482 documents=  21.9 GB).

Data Loading : we are loading data into elastic by writing 20 threads.

Client creation :

 Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name",
"UCS_DEV_ES_CLUSTER").build();

 elasticClient = new TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress("server1", 9300))
 .addTransportAddress(new InetSocketTransportAddress("server2", 9300));
Below are the server details

we have  4  nodes.

Each node contains
CPU: 4 , RAM: 16G ,  Storage: 122G
ES_HEAP_SIZE 8 G

Please find enclosed elasticsearch.yml file which we are using.




Thanks,
Srikanth.



On Wed, Dec 18, 2013 at 5:02 PM, Mark Walkom wrote:

> It will help if you post (via a gist/pastebin/etc) what you have now.
>
> Also what sort of data is it, how are you loading it, what versions of
> things are you running on (OS, java, ES, etc etc)
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com
> web: www.campaignmonitor.com
>
>
> On 19 December 2013 11:53, srikanth ramineni wrote:
>
>> Hi Team,
>>
>>
>> We have below server configurations.  We are trying to load more than 500
>> million data into elastic. When data reaches 40 millions, shards are going
>> down and data is deleting . Can you please help us with best configuration
>> for above requirement.
>>
>> Below are the server details
>>
>> we have  4  nodes.
>>
>> Each node contains
>> CPU: 4 , RAM: 16G ,  Storage: 122G
>> ES_HEAP_SIZE 8 G
>>
>>
>>
>> Thanks,
>> Srikanth.
>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/QRGwAlqL-qY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGuNUAV5A_Vj9k285nAW_xW7UdnVr5K5n1DBndBLo%3D9KiQ8BaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


elasticsearch.yml
Description: Binary data

Re: Request for elastic search configuration

[Mark Walkom]

It will help if you post (via a gist/pastebin/etc) what you have now.

Also what sort of data is it, how are you loading it, what versions of
things are you running on (OS, java, ES, etc etc)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 19 December 2013 11:53, srikanth ramineni  wrote:

> Hi Team,
>
>
> We have below server configurations.  We are trying to load more than 500
> million data into elastic. When data reaches 40 millions, shards are going
> down and data is deleting . Can you please help us with best configuration
> for above requirement.
>
> Below are the server details
>
> we have  4  nodes.
>
> Each node contains
> CPU: 4 , RAM: 16G ,  Storage: 122G
> ES_HEAP_SIZE 8 G
>
>
>
> Thanks,
> Srikanth.
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YPB%3DkkJ3eOFRRc%2Bw%2BqG3junYQWpiD3EzPp677BLTbwvQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Request for elastic search configuration

[srikanth ramineni]

Hi Team,


We have below server configurations.  We are trying to load more than 500 
million data into elastic. When data reaches 40 millions, shards are going 
down and data is deleting . Can you please help us with best configuration 
for above requirement.

Below are the server details

we have  4  nodes.

Each node contains 
CPU: 4 , RAM: 16G ,  Storage: 122G
ES_HEAP_SIZE 8 G



Thanks,
Srikanth.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/436a22c0-7aca-4502-b229-db91c0ebce01%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.