date:20140712

Re: Suggestion for a Test ELK Deployment ( Best Performance/ Load Balancing / Security / HA)

2014-07-12 Thread Stefano Ruggiero

Thanks alot for the answer, you confirm my worries about a cross datacenter 
cluster, even if i think that ES has been build also for this type of 
situation, the problem that i see is that with tribe nodes we cant have a 
full HA across the 2 DC, even if is a good solution for searching in all DC 
it isent a good solution for replicate indexed data, am i right ?

so you suggests to have 4 shard and on replica so 1 primary and 1 replica 
per node ? (obv f we have installed 4 nodes in 2 DC)

what do you use for load balance indexing across nodes where one of them go 
down given that logstash allow only 1 ip or domian in the output 
configuration.

Regards

Il giorno domenica 13 luglio 2014 02:28:01 UTC+2, Mark Walkom ha scritto:
>
> As you may have read, ES is latency sensitive and so having a cluster 
> across your DCs isn't recommended.
> You may want to look at tribe nodes and then have two separate clusters, 
> that way you get around your problems with wanting all data available in 
> both DCs and also cross DC load balancing.
>
> Around shard counts, to ensure you balance load you ideally want one shard 
> per node, then create replicas based on what you require. Trying to setup 
> replica only nodes isn't worth the trouble though.
>
> Security wise, the base setup you have is good but you may want to have a 
> look at some of the community baed solutions to ACLs if that's what you 
> want.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>  
>
> On 12 July 2014 21:02, Stefano Ruggiero  > wrote:
>
>> Hi all,
>>
>> i would like to start this conversation to discuss about the best 
>> architecture of ELK based on our hardware and needed for a test envirorment.
>>
>> What we have:
>>
>>- 4+ ES nodes 
>>- x2 with 24 gb of rams and 800 gb of HD SAS 4 x2 CPU
>>   - x2 with 16 gb of rams and 500 gb of HD SAS 4 x2 CPU
>>- 10+ LS Collectors
>>- 2+ Kibana instances
>>
>> we have 2 separate Datacentre infact, as i show, we have the specular 
>> resources on the above list, so for example we have 2 ES nodes on the first 
>> location and the other 2 in the second location that are linked with double 
>> redundant fiber 10 gbit .
>>
>> Our test is to understand how ELK stack performing with indexing of all 
>> Application and Server Events, so we are talking about 200 Events for 
>> seconds in the test lab. We would like to have a retention of 2 or 3 
>> mounth, so seraching with kibana that logs, and then close and backup old 
>> index that we test is a working well with curator plugin.
>>
>> *What is the best configuration  for Load balancing events across the two 
>> locatio*n i mean every collectors should have 2 available choice for the 
>> output in case of one node go down or is performing bad , what do you 
>> suggests ?
>> we try Nginx with health check but i think that ES should do something 
>> similar for load balancing indexing process with a node master false, data 
>> false , even if we raed in the community that this type of node is reserved 
>> for balancing search and not indexing that go every time across the master 
>> of the cluster, am i right?
>>
>> *What is the best configuration that you test ?* i mean how many shards 
>> how many replicas  for a full High availability and redundant solution ?
>> we try to play with 2 shard and one replica for 4 data node, because as 
>> we see replcas are involved in search process so it can be a good solution 
>> to reserve some nodes only for replicas but what we miss is if a node go 
>> down or a datacentre died can we have all data automatically on the other 
>> side (just with replicas) ? ( we know that for the golden rule we need to 
>> have 5 nodes and 3 minimum master node for a cluster so if we have only 2 
>> DC could be critical because one DC need to have more nodes and become the 
>> leader of the all cluster... ) 
>>
>> *What is your best configuration for a security prospective ? *
>> we test nginx also as reverse proxy with standard autentcation to prevent 
>> unwanted DELETE and PUT but we are looking for a more strong solution with 
>> more flexibility and roles/premissions configuration like a standard SQL 
>> DB. Our network layer is really strong every ELK layer has his own DMZ, ACL 
>> and firewall rule
>>
>> iam worried about espacially on the ES configuration like shards replica 
>> and load balancing i think that this conversation should be helpfull for a 
>> very large community auditor that have some doubts about ES and ELK stack 
>> in general.
>>
>> Best Regards,
>> Stefano 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/d3ca4aaf-a1ee-

Re: Marvel does not display any contents with ES1.2.1.

2014-07-12 Thread Boaz Leskes

Hi Ajay,

I'm not sure exactly what problem you have, but note that you have disabled 
data shipping by setting "marvel.agent.enabled: false" . Was this your 
intention? If not this why you don't have any data in marvel. 

Cheers,
Boaz

PS  you probably want to also remove "indices.memory.index_buffer_size: 
50%", unless you plan on doing very heavy indexing and even then I'd wait 
until I see problems

On Thursday, July 10, 2014 9:59:52 PM UTC+2, ajay@gmail.com wrote:
>
> Hi 
>
> I have new test setup for ES and added latest marvel from internet and 
> tried ipv6 workaround to mentioned in other thread too but display does not 
> show any contents from the marvel URL. Marvel plugin installed from 
> internet .
>
>
> Here  is the status (only one node up for debugging for 3 node cluster)
>
> curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
> {
>   "cluster_name" : "elastic_alpha",
>   "status" : "yellow",
>   "timed_out" : false,
>   "number_of_nodes" : 1,
>   "number_of_data_nodes" : 1,
>   "active_primary_shards" : 5,
>   "active_shards" : 5,
>   "relocating_shards" : 0,
>   "initializing_shards" : 0,
>   "unassigned_shards" : 5
> }
>
> root@elk01:/usr/local/elasticsearch# bin/plugin --list
> Installed plugins:
> - marvel
> cluster.name: elastic_alpha
>
>
> elasticsearch.yml  file:
>
> node.name: elk01
> node.max_local_storage_nodes: 1
> index.mapper.dynamic: true
> action.auto_create_index: true
> action.disable_delete_all_indices: true
> path.conf: /iscsi_disk1/etc/elasticsearch
> path.data: /iscsi_disk1/data/elasticsearch
> path.work: /tmp/work/elasticsearch
> path.logs: /iscsi_disk1/logs/elasticsearch
> path.plugins: /usr/local/elasticsearch/bin/plugin
> bootstrap.mlockall: true
> http.port: 9200
> gateway.expected_nodes: 1
> discovery.zen.minimum_master_nodes: 1
> discovery.zen.ping.multicast.enabled: false
> cloud.node.auto_attributes: true
> discovery.zen.ping.unicast.hosts: [ "elk01:9300","elk02:9300","elk03:9300" 
> ]
> indices.memory.index_buffer_size: 50%
> marvel.agent.exporter.es.hosts: ["[::1]:9200"]
> marvel.agent.enabled: false
>
> Any thing wrong with the config ?
>
> Thanks
> Ajay
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3f66fd8d-211b-49df-bb68-96ee918c53fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Suggestion for a Test ELK Deployment ( Best Performance/ Load Balancing / Security / HA)

2014-07-12 Thread Mark Walkom

As you may have read, ES is latency sensitive and so having a cluster
across your DCs isn't recommended.
You may want to look at tribe nodes and then have two separate clusters,
that way you get around your problems with wanting all data available in
both DCs and also cross DC load balancing.

Around shard counts, to ensure you balance load you ideally want one shard
per node, then create replicas based on what you require. Trying to setup
replica only nodes isn't worth the trouble though.

Security wise, the base setup you have is good but you may want to have a
look at some of the community baed solutions to ACLs if that's what you
want.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 12 July 2014 21:02, Stefano Ruggiero  wrote:

> Hi all,
>
> i would like to start this conversation to discuss about the best
> architecture of ELK based on our hardware and needed for a test envirorment.
>
> What we have:
>
>- 4+ ES nodes
>- x2 with 24 gb of rams and 800 gb of HD SAS 4 x2 CPU
>   - x2 with 16 gb of rams and 500 gb of HD SAS 4 x2 CPU
>- 10+ LS Collectors
>- 2+ Kibana instances
>
> we have 2 separate Datacentre infact, as i show, we have the specular
> resources on the above list, so for example we have 2 ES nodes on the first
> location and the other 2 in the second location that are linked with double
> redundant fiber 10 gbit .
>
> Our test is to understand how ELK stack performing with indexing of all
> Application and Server Events, so we are talking about 200 Events for
> seconds in the test lab. We would like to have a retention of 2 or 3
> mounth, so seraching with kibana that logs, and then close and backup old
> index that we test is a working well with curator plugin.
>
> *What is the best configuration  for Load balancing events across the two
> locatio*n i mean every collectors should have 2 available choice for the
> output in case of one node go down or is performing bad , what do you
> suggests ?
> we try Nginx with health check but i think that ES should do something
> similar for load balancing indexing process with a node master false, data
> false , even if we raed in the community that this type of node is reserved
> for balancing search and not indexing that go every time across the master
> of the cluster, am i right?
>
> *What is the best configuration that you test ?* i mean how many shards
> how many replicas  for a full High availability and redundant solution ?
> we try to play with 2 shard and one replica for 4 data node, because as we
> see replcas are involved in search process so it can be a good solution to
> reserve some nodes only for replicas but what we miss is if a node go down
> or a datacentre died can we have all data automatically on the other side
> (just with replicas) ? ( we know that for the golden rule we need to have 5
> nodes and 3 minimum master node for a cluster so if we have only 2 DC could
> be critical because one DC need to have more nodes and become the leader of
> the all cluster... )
>
> *What is your best configuration for a security prospective ? *
> we test nginx also as reverse proxy with standard autentcation to prevent
> unwanted DELETE and PUT but we are looking for a more strong solution with
> more flexibility and roles/premissions configuration like a standard SQL
> DB. Our network layer is really strong every ELK layer has his own DMZ, ACL
> and firewall rule
>
> iam worried about espacially on the ES configuration like shards replica
> and load balancing i think that this conversation should be helpfull for a
> very large community auditor that have some doubts about ES and ELK stack
> in general.
>
> Best Regards,
> Stefano
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d3ca4aaf-a1ee-4732-9ae5-629dd8198e7b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YB72dN5oiCs6%2B_m%2BraJAUW1LsihLn_kVdJSSWtMDbC%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Clarified

2014-07-12 Thread Mark Walkom

Yes, the poster has been banned.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 12 July 2014 21:35, arshpreet singh  wrote:

> On Sat, Jul 12, 2014 at 3:22 AM,   wrote:
> > Hermione pulled her robes out of Pettigrews clutching hands and backed
> away
> > against the wall
> > i just left you a private message
> > Read the full full message
>
> Is this a spam?
>
> --
>
> Thanks
> Arshpreet singh
> http://arshpreetsingh.wordpress.com/
> I have no special talents. Only passionately curious
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAAstK2Hw8ce%2Bepqgw2FsR3bOMxYjrM8gWXgXF98xO%3Dbov49H-A%40mail.gmail.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZLxsJN-DAiyrw91HaD%2B-1a4k1R%2B2g%3DqhhcuvNPTs%2Bzew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: elasticsearch dies every other day

2014-07-12 Thread Mark Walkom

What's your heap size how much data do you have in your indexes?
Take a look at settings like discovery.zen.minimum_master_nodes to reduce
the multi-master problem you see, although ideally you want an odd number
of nodes for that (eg 3 or 5).

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 13 July 2014 04:22, Klavs Klavsen  wrote:

> my cluster - running 1.1.2 with oracle java 1.7_55 - dies several days a
> week
> one the nodes gets "disconnected"..
> this time one of them logged:  observer timed out. notifying listener.
> several times
> I then have to restart them all - this time I had several who thought they
> were masters
> they are physical machines on the same LAN
> the machines (4) index between 70 and 130k documents (lines from logstash)
> per minute. I input into different indexes (not just logstash-$date)
>
> after having a queue built up - they will easily index 260 to 290k/min -
> until the queue is emptied
> so they seem to have no resource shortage, but they somehow "get tired and
> die"- very often :(
>
> any ideas how I should proceed in debugging this issue? It seems to fit
> that EVERY time a node dies - I have garbage collection log entries. This
> time I had these on one node:
> [2014-07-12 17:12:25,132][INFO ][monitor.jvm  ]
> [p-elasticlog02] [gc][young][194961][25004] duration [767ms], collections
> [1]/[1s], total [767ms]/[34.3m], memory [27.7gb]->[26.7gb]/[31.7gb],
> all_pools {[young] [1gb]->[5.3mb]/[1.1
> gb]}{[survivor] [149.7mb]->[124.1mb]/[149.7mb]}{[old]
> [26.5gb]->[26.6gb]/[30.4gb]}
> [2014-07-12 17:12:44,929][INFO ][monitor.jvm  ]
> [p-elasticlog02] [gc][young][194980][25007] duration [804ms], collections
> [1]/[1.1s], total [804ms]/[34.3m], memory [27.8gb]->[26.9gb]/[31.7gb],
> all_pools {[young] [1gb]->[11.2mb]/[
> 1.1gb]}{[survivor] [149.7mb]->[146.1mb]/[149.7mb]}{[old]
> [26.6gb]->[26.7gb]/[30.4gb]}
> [2014-07-12 17:14:57,032][INFO ][monitor.jvm  ]
> [p-elasticlog02] [gc][young][195109][25035] duration [837ms], collections
> [1]/[1s], total [837ms]/[34.4m], memory [28.9gb]->[28.1gb]/[31.7gb],
> all_pools {[young] [1gb]->[141.1mb]/[1
> .1gb]}{[survivor] [149.7mb]->[145.7mb]/[149.7mb]}{[old]
> [27.7gb]->[27.8gb]/[30.4gb]}
> [2014-07-12 17:16:17,016][INFO ][monitor.jvm  ]
> [p-elasticlog02] [gc][young][195187][25053] duration [756ms], collections
> [1]/[1.4s], total [756ms]/[34.5m], memory [29.5gb]->[28.7gb]/[31.7gb],
> all_pools {[young] [926.8mb]->[27.6m
> b]/[1.1gb]}{[survivor] [149.7mb]->[138.6mb]/[149.7mb]}{[old]
> [28.4gb]->[28.5gb]/[30.4gb]}
> [2014-07-12 17:18:57,313][WARN ][monitor.jvm  ]
> [p-elasticlog02] [gc][young][195303][25075] duration [1.1s], collections
> [1]/[40.6s], total [1.1s]/[34.6m], memory [30.4gb]->[29.2gb]/[31.7gb],
> all_pools {[young] [1.1gb]->[12.3mb]/
> [1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> [29.1gb]->[29.2gb]/[30.4gb]}
> [2014-07-12 17:18:57,314][WARN ][monitor.jvm  ]
> [p-elasticlog02] [gc][old][195303][53] duration [39.1s], collections
> [2]/[40.6s], total [39.1s]/[1.5m], memory [30.4gb]->[29.2gb]/[31.7gb],
> all_pools {[young] [1.1gb]->[12.3mb]/[1.1
> gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old]
> [29.1gb]->[29.2gb]/[30.4gb]}
>
>
> I collect a lot of counters from elasticsearch (Using elasticsearch
> collector in diamond (graphite collector written in python)) - so I have
> data on the ES nodes.
>
> my config is this:
> index.warmer.enabled: false
> cluster.name: elasticsearch
> node.name: "p-elasticlog02"
> node.master: true
> node.data: true
> action.disable_delete_all_indices: true
> indices.memory.index_buffer_size: 50%
> indices.fielddata.cache.size: 30%
> index.refresh_interval: 5s
> index.index_concurrency: 16
> threadpool.search.type: fixed
> threadpool.search.size: 400
> threadpool.search.queue_size: 900
> threadpool.bulk.type: fixed
> threadpool.bulk.size: 500
> threadpool.bulk.queue_size: 900
> threadpool.index.type: fixed
> threadpool.index.size: 300
> threadpool.index.queue_size: -1
> path.data: /var/lib/elasticsearch/
> bootstrap.mlockall: true
> network.publish_host: $hostip
> discovery.zen.ping.multicast.enabled: false
> discovery.zen.ping.unicast.hosts: ["p-elasticlog01.example.idk",
> "p-elasticlog02.example.idk", "p-elasticlog03.example.idk",
> "p-elasticlog04.example.idk", "p-elasticlog05.example.idk"]
>
> I have 24 cores in each machine (doing pretty much nothing) - so I was
> considering trying to switch to g1gc f.ex. - as I've read it should be
> better in some respects.. any input?
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866

Re: how to translate “and or” where clause from sql query to elasticsearch filter

2014-07-12 Thread Artem Frolov

Community, answer my question please! It's an actual problem still. :(

пятница, 11 июля 2014 г., 17:15:04 UTC+3 пользователь Artem Frolov написал:
>
> I've done what you want.
> Mapping and populating data GIST: 
> https://gist.github.com/ArFeRR/3031c1ce8f95549ad86d
>
> GIST with my search query: 
> https://gist.github.com/ArFeRR/f69ebe24ddc543b7bffd
>
> (it have to return one record with resolution:1920x1080 and weight: 2,9 
> kg. It's the notebook with the name "Lenovo IdeaPad Z710A")
> but it returns empty.
>
> Please help to achieve the behaviour what i need!
>
> пятница, 11 июля 2014 г., 11:48:40 UTC+3 пользователь David Pilato написал:
>>
>> A full script would allow any user on the mailing list to recreate from 
>> scratch your issue without the need of building a script by ourselves which 
>> is really time consuming.
>>
>> So, basically a script should look like this:
>>
>> // Remove test data
>> DELETE test
>>
>> // If needed, add your settings/mappings
>> PUT test
>> {
>>   "settings": {},
>>   "mappings": {}
>> }
>>
>> // Index some data
>> PUT test/doc/1
>> {
>>   "foo":"bar"
>> }
>>
>> PUT test/doc/x?refresh
>> {
>>   "foo":"bar"
>> }
>>
>> // Run the query
>> GET test/doc/_search
>> {
>> }
>>
>> With that, we can definitely help you I think.
>>
>> -- 
>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com*
>> @dadoonet  | @elasticsearchfr 
>> 
>>
>>
>> Le 11 juillet 2014 à 10:44:01, Artem Frolov (kee...@gmail.com) a écrit:
>>
>>  Can you answer what's wrong with the gist, i've provided? I can't figure 
>> out...
>> I provide you all the data I have, related to this issue. Look:
>> 1) The elasticsearch index, containing the products and its options, 
>> which have to be filtered: 
>> https://gist.github.com/ArFeRR/de86b8b0a5f2bc7dfd86
>> 2) The JSON query for the filtration: 
>> https://gist.github.com/ArFeRR/e159ef1047122a617b88
>> 3)The ELastica.io code to genereate the json above on PHP:
>> https://gist.github.com/ArFeRR/cebb2bf54232069d817b#file-gistfile1-php
>>
>> (I've changed the data to make it more real)
>>
>> All the json works fine, i've tested it! Can't figure out what's wrong 
>> and what I have to do to provide a "FULL working GIST" for you
>>
>>
>>
>> четверг, 10 июля 2014 г., 18:37:19 UTC+3 пользователь David Pilato 
>> написал: 
>>>
>>>  Have a look at this page to see how you can build a full working GIST 
>>> which could help us to reproduce your use case.
>>>  
>>>  When your GIST will be updated, please update this thread so I can look 
>>> at it.
>>>
>>>  -- 
>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com* 
>>>  @dadoonet  | @elasticsearchfr 
>>> 
>>>  
>>>
>>> Le 10 juillet 2014 à 11:59:22, Artem Frolov (kee...@gmail.com) a écrit:
>>>
>>>  the GIST:
>>>  https://gist.github.com/ArFeRR/630acb216b8d95168b73
>>>
>>> четверг, 10 июля 2014 г., 12:51:49 UTC+3 пользователь Artem Frolov 
>>> написал: 

 here's my try to solve it:

 {
"filtered":{
   "filter":{
  "nested":{
 "path":"productsOptionValues",
 "filter":{
"and":[
   {
  "or":[
 {
"and":[
   {
  "term":{
 
 "productsOptionValues.productOption":"weight"
  }
   },
   {
  "term":{
 "productsOptionValues.value":
 "500 kg"
  }
   }
]
 },
 {
"and":[
   {
  "term":{
 
 "productsOptionValues.productOption":"weight"
  }
   },
   {
  "term":{
 "productsOptionValues.value":"50kg"
  }
   }
]
 }
  ]
   },
   {
  "or":[
 {
"and":[
   {
  "term":{
 
 "productsOptionValues.productOption":"magic"
  }
>>

Re: spam

2014-07-12 Thread Ivan Brusic

Read the recent comments regarding with the recent spam:
https://groups.google.com/d/msg/elasticsearch/byATcjKgdYE/_Neoiof4fKIJ

This new spam account has been banned.

-- 
Ivan


On Sat, Jul 12, 2014 at 2:50 PM, Warner Onstine  wrote:

> Could we please turn on first post filters to keep the spammers out? I
> know it's a bit of extra work but it would cut down on the spam immensely.
>
> -warner
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAJNTuMBbXJ%3D-qMqBYoYXy0fTRMLHOBZd%2BpaU%2B%2BN5N3cMTFq7Bw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBBm0e3cROo9mr2BwjaxmLkD0rb9FqL9d-qTWpT%2B4-9kg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

index creation from very large data set

2014-07-12 Thread hrgdvqejerhrgdvqejer

Hi, 

What are the options for creating a new index form an existing very 
large data set? do we need to linearly walk the data and insert each 
document one-by-one? 

 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
--



https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12480.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12479.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12478.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12477.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12476.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12475.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12474.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12473.html

https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12472.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12471.html
https://www.mail-archive.com/elasticsearch@googlegroups.com/msg12470.html
https://groups.google.com/d/topic/elasticsearch/FxTNOPk2rm8
https://groups.google.com/d/topic/elasticsearch/uo4WzBKHtVs
https://groups.google.com/d/topic/elasticsearch/iAWLN0yLGag
https://groups.google.com/d/topic/elasticsearch/86sRz4J9GgM
https://groups.google.com/d/topic/elasticsearch/wI8kLa22cA0
https://groups.google.com/d/topic/elasticsearch/Mgf9tTEWYoE
https://groups.google.com/d/topic/elasticsearch/Qe7dZnydWEs
https://groups.google.com/d/topic/elasticsearch/u1abfqf-p-0
https://groups.google.com/d/topic/elasticsearch/y0mHmko8SE8
https://groups.google.com/d/topic/elasticsearch/qG0vi-cwkvc
https://groups.google.com/d/topic/theano-users/9EkJWXCwYEc
https://groups.google.com/d/topic/orbeon/tHCMaAByJV0
https://groups.google.com/d/topic/jenkinsci-dev/na5iegT6zAA
https://groups.google.com/d/topic/griffith-devel/DflKejVZtns
https://groups.google.com/d/topic/atmosphere-framework/GzzvqLT2oD4
https://groups.google.com/d/topic/atmosphere-framework/H6cnmcVma4o
https://groups.google.com/d/topic/griffith-devel/3_3CDGpCBhY
https://groups.google.com/d/topic/jenkinsci-dev/QFoWO9VXdJ8
https://groups.google.com/d/topic/orbeon/Zrzjz_MkSOg
https://groups.google.com/d/topic/theano-users/8DyD-AmviM0
https://groups.google.com/d/topic/atmosphere-framework/ae1rORzwq-k
https://groups.google.com/d/topic/griffith-devel/KFFYQGkDQhc
https://groups.google.com/d/topic/jenkinsci-dev/I2g5UTo7lJ8
https://groups.google.com/d/topic/orbeon/IMRK7tBe_0A
https://groups.google.com/d/topic/theano-users/gnMbuWnpu-Y

 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
--

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/ac15834d-303a-44c2-943e-f4f6324dc5a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

spam

2014-07-12 Thread Warner Onstine

Could we please turn on first post filters to keep the spammers out? I know
it's a bit of extra work but it would cut down on the spam immensely.

-warner

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAJNTuMBbXJ%3D-qMqBYoYXy0fTRMLHOBZd%2BpaU%2B%2BN5N3cMTFq7Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[SUSU] )&-| Watch Jersey Boys 2014 Full Movie Online HD

2014-07-12 Thread Prabowo Hatta

 [SUSU] )&-| Watch Jersey Boys 2014 Full Movie Online HD 
 

Watch Jersey Boys Full Movie Online "Everybody remembers it how they need 
to." or Download Jersey Boys Full Movie Today. 

The story of four young men from the wrong side of the tracks in New Jersey 
who came together to form the iconic 1960s rock group The Four Seasons. 



* Watch Jersey Boys Full Movie Online  >>> 
http://cbz.biz/9jt0br *
* Watch Jersey Boys Full Movie Free  >>> 
http://cbz.biz/9jt0br *
* Free Watch Jersey Boys Full Movie Online  >>> 
http://cbz.biz/9jt0br *
 



Jersey Boys is a musical biography of the Four Seasons-the rise, the tough 
times and personal clashes, and the ultimate triumph of a group of friends 
whose music became symbolic of a generation. Far from a mere tribute 
concert (though it does include numbers from the popular Four Seasons 
songbook), Jersey Boys gets to the heart of the relationships at the center 
of the group-with a special focus on frontman Frankie Valli, the small kid 
with the big falsetto. In addition to following the quartet's coming of age 
as performers, the core of the show is how an allegiance to a code of honor 
learned in the streets of their native New Jersey got them through a 
multitude of challenges: gambling debts, Mafia threats and family 
disasters. Jersey Boys is a glimpse at the people behind a sound that has 
managed to endure for over four decades in the hearts of the public. 
Spend a little time now for free register and you could benefit later. You 
will be able to stream or Watch Jersey Boys Full Movie Streaming from your 
computer, tablet, TV or mobile device. 
Watch Jersey Boys Full Movie Online (2014).
WATCH Jersey Boys FULL MOVIE 1080p
Watch Jersey Boys Online Free RIGHT HERE!!
Watch Jersey Boys (2014) Movie HD Streaming for Free
WATCH Jersey Boys Online Full Movie Streaming
Jersey Boys Streaming Movie
Watch Jersey Boys in HD Online
Jersey Boys high Quality Stream
Watch Jersey Boys Online
Jersey Boys Full Movie HD Stream
Jersey Boys Full Movie Streaming 
Stream Jersey Boys Full Movie Online HD Quality
Jersey Boys film Complete Streaming
Spend a little time for free register and you could benefit later DONWLOAD 
Jersey Boys FULL MOVIE 1080p. You will be able to stream or download See 
also "Jersey Boys" Full Stream Movie Online from your computer, tablet, TV 
or mobile device how toWatch Jersey Boys () HD Movie Streaming for Free. 
Watch Jersey Boys Full Movie Streaming Online. Jersey Boys Movie.
WATCH Jersey Boys FULL MOVIE 1080p
Download Jersey Boys Movie Ipod
download Jersey Boys full movie
Download Jersey Boys
Jersey Boys Movie Download For HTC
Watch The Entire Jersey Boys Film
How To Download Jersey Boys:
downloadable films Jersey Boys
Illegal Jersey Boys Movie Watch
The Full Jersey Boys Film Online
Watch legal divx Jersey Boys movie
download Jersey Boys video
Jersey Boys movie with good quality
Watch Jersey Boys Full-lenght
Film Jersey Boys Download
The Film Jersey Boys Online Watch The Jersey Boys Movie
Jersey Boys Film Opening
Jersey Boys Soundtrack Watch
Watch Jersey Boys Online
Jersey Boys Movie To Watch Ipod Jersey Boys Movie
Watch Movie Jersey Boys Hq
Jersey Boys Film In English To Watch Download Jersey Boys Film Hd
Jersey Boys Movie Watch Phone
Watch Jersey Boys Online
Watch Jersey Boys Online Full Movie
Watch Divx Jersey BoysDownload Jersey Boys Movie Ipod
download Jersey Boys full movie
Download Jersey Boys
Jersey Boys Movie Download For HTC
Watch The Entire Jersey Boys Film
How To Download Jersey Boys
downloadable films Jersey Boys
Illegal Jersey Boys Movie Watch
The Full Jersey Boys Film Online
Watch legal divx Jersey Boys movie
download Jersey Boys video
Jersey Boys movie with good quality
Watch Jersey Boys Full-lenght
Film Jersey Boys Download
The Film Jersey Boys Online Watch The Jersey Boys Movie
Jersey Boys Film Opening
Jersey Boys Soundtrack Watch
Watch Jersey Boys Online
Jersey Boys Movie To Watch Ipod Jersey Boys Movie
Watch Movie Jersey Boys Hq
Jersey Boys Film In English To Watch Download Jersey Boys Film Hd
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
--

-- 
You received this mes

[LOVP] &%^@ Watch Edge of Tomorrow 2014 Full Movie Online for Free

2014-07-12 Thread Prabowo Hatta

 [LOVP] &%^@ Watch Edge of Tomorrow 2014 Full Movie Online for Free 
 

Watch Edge of Tomorrow Full Movie Online "Live, Die, Repeat" or Download 
Edge of Tomorrow Full Movie Today. 

An officer finds himself caught in a time loop in a war with an alien race. 
His skills increase as he faces the same brutal combat scenarios, and his 
union with a Special Forces warrior gets him closer and closer to defeating 
the enemy. 



* Watch Edge of Tomorrow Full Movie Online  >>> 
http://cbz.biz/993ju2 *
* Watch Edge of Tomorrow Full Movie Free  >>> 
http://cbz.biz/993ju2 *
* Free Watch Edge of Tomorrow Full Movie Online  >>> 
http://cbz.biz/993ju2 *
 



An alien race has hit the Earth in an unrelenting assault, unbeatable by 
any military unit in the world. Major William Cage (Cruise) is an officer 
who has never seen a day of combat when he is unceremoniously dropped into 
what amounts to a suicide mission. Killed within minutes, Cage now finds 
himself inexplicably thrown into a time loop-forcing him to live out the 
same brutal combat over and over, fighting and dying again...and again. But 
with each battle, Cage becomes able to engage the adversaries with 
increasing skill, alongside Special Forces warrior Rita Vrataski (Blunt). 
And, as Cage and Vrataski take the fight to the aliens, each repeated 
encounter gets them one step closer to defeating the enemy. 
Spend a little time now for free register and you could benefit later. You 
will be able to stream or Watch Edge of Tomorrow Full Movie Streaming from 
your computer, tablet, TV or mobile device. 
Watch Edge of Tomorrow Full Movie Online (2014).
WATCH Edge of Tomorrow FULL MOVIE 1080p
Watch Edge of Tomorrow Online Free RIGHT HERE!!
Watch Edge of Tomorrow (2014) Movie HD Streaming for Free
WATCH Edge of Tomorrow Online Full Movie Streaming
Edge of Tomorrow Streaming Movie
Watch Edge of Tomorrow in HD Online
Edge of Tomorrow high Quality Stream
Watch Edge of Tomorrow Online
Edge of Tomorrow Full Movie HD Stream
Edge of Tomorrow Full Movie Streaming 
Stream Edge of Tomorrow Full Movie Online HD Quality
Edge of Tomorrow film Complete Streaming
Spend a little time for free register and you could benefit later DONWLOAD 
Edge of Tomorrow FULL MOVIE 1080p. You will be able to stream or download 
See also "Edge of Tomorrow" Full Stream Movie Online from your computer, 
tablet, TV or mobile device how toWatch Edge of Tomorrow () HD Movie 
Streaming for Free. Watch Edge of Tomorrow Full Movie Streaming Online. 
Edge of Tomorrow Movie.
WATCH Edge of Tomorrow FULL MOVIE 1080p
Download Edge of Tomorrow Movie Ipod
download Edge of Tomorrow full movie
Download Edge of Tomorrow
Edge of Tomorrow Movie Download For HTC
Watch The Entire Edge of Tomorrow Film
How To Download Edge of Tomorrow:
downloadable films Edge of Tomorrow
Illegal Edge of Tomorrow Movie Watch
The Full Edge of Tomorrow Film Online
Watch legal divx Edge of Tomorrow movie
download Edge of Tomorrow video
Edge of Tomorrow movie with good quality
Watch Edge of Tomorrow Full-lenght
Film Edge of Tomorrow Download
The Film Edge of Tomorrow Online Watch The Edge of Tomorrow Movie
Edge of Tomorrow Film Opening
Edge of Tomorrow Soundtrack Watch
Watch Edge of Tomorrow Online
Edge of Tomorrow Movie To Watch Ipod Edge of Tomorrow Movie
Watch Movie Edge of Tomorrow Hq
Edge of Tomorrow Film In English To Watch Download Edge of Tomorrow Film Hd
Edge of Tomorrow Movie Watch Phone
Watch Edge of Tomorrow Online
Watch Edge of Tomorrow Online Full Movie
Watch Divx Edge of TomorrowDownload Edge of Tomorrow Movie Ipod
download Edge of Tomorrow full movie
Download Edge of Tomorrow
Edge of Tomorrow Movie Download For HTC
Watch The Entire Edge of Tomorrow Film
How To Download Edge of Tomorrow
downloadable films Edge of Tomorrow
Illegal Edge of Tomorrow Movie Watch
The Full Edge of Tomorrow Film Online
Watch legal divx Edge of Tomorrow movie
download Edge of Tomorrow video
Edge of Tomorrow movie with good quality
Watch Edge of Tomorrow Full-lenght
Film Edge of Tomorrow Download
The Film Edge of Tomorrow Online Watch The Edge of Tomorrow Movie
Edge of Tomorrow Film Opening
Edge of Tomorrow Soundtrack Watch
Watch Edge of Tomorrow Online
Edge of Tomorrow Movie To Watch Ipod Edge of Tomorrow Movie
Watch Movie Edge of Tomorrow Hq
Edge of Tomorrow Film In English To Watch Download Edge of Tomorrow Film Hd
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .
 .

[BLENG] ?>[$ Watch X-Men: Days of Future Past 2014 Full Movie Online

2014-07-12 Thread Prabowo Hatta

 [BLENG] ?>[$ Watch X-Men: Days of Future Past 2014 Full Movie Online 
 

Watch X-Men: Days of Future Past Full Movie Online "His past. Our future." 
or Download X-Men: Days of Future Past Full Movie Today. 

The X-Men send Wolverine to the past in a desperate effort to change 
history and prevent an event that results in doom for both humans and 
mutants. 



* Watch X-Men: Days of Future Past Full Movie Online 
 >>> http://cbz.biz/ltjs9j *
* Watch X-Men: Days of Future Past Full Movie Free  
>>> http://cbz.biz/ltjs9j *
* Free Watch X-Men: Days of Future Past Full Movie Online 
 >>> http://cbz.biz/ltjs9j *
 



Sentinels, robots that were created for the purpose of hunting down mutants 
were released in 1973. 50 years later the Sentinels would also hunt humans 
who aid mutants. Charles Xavier and his X-Men try their best to deal with 
the Sentinels but they are able to adapt and deal with all mutant 
abilities. Charles decides to go back in time and change things. He asks 
Kitty Pryde who can send a person's consciousness into the person's past to 
send him but she can only send someone back a few weeks because if she 
sends someone back further it could harm them. So Logan decides to go back 
himself because he might be able to withstand it. So Charles tells him that 
it's Mystique who's responsible because when she learned about the 
Sentinels she sought out Bolivar Trask the man who created them and killed 
him. She would be caught and studied and her ability to change was somehow 
added to the Sentinels which is why they can adapt. Logan must go to the 
younger Charles and ask him to help; problem is... 
Spend a little time now for free register and you could benefit later. You 
will be able to stream or Watch X-Men: Days of Future Past Full Movie 
Streaming from your computer, tablet, TV or mobile device. 
Watch X-Men: Days of Future Past Full Movie Online (2014).
WATCH X-Men: Days of Future Past FULL MOVIE 1080p
Watch X-Men: Days of Future Past Online Free RIGHT HERE!!
Watch X-Men: Days of Future Past (2014) Movie HD Streaming for Free
WATCH X-Men: Days of Future Past Online Full Movie Streaming
X-Men: Days of Future Past Streaming Movie
Watch X-Men: Days of Future Past in HD Online
X-Men: Days of Future Past high Quality Stream
Watch X-Men: Days of Future Past Online
X-Men: Days of Future Past Full Movie HD Stream
X-Men: Days of Future Past Full Movie Streaming 
Stream X-Men: Days of Future Past Full Movie Online HD Quality
X-Men: Days of Future Past film Complete Streaming
Spend a little time for free register and you could benefit later DONWLOAD 
X-Men: Days of Future Past FULL MOVIE 1080p. You will be able to stream or 
download See also "X-Men: Days of Future Past" Full Stream Movie Online 
from your computer, tablet, TV or mobile device how toWatch X-Men: Days of 
Future Past () HD Movie Streaming for Free. Watch X-Men: Days of Future 
Past Full Movie Streaming Online. X-Men: Days of Future Past Movie.
WATCH X-Men: Days of Future Past FULL MOVIE 1080p
Download X-Men: Days of Future Past Movie Ipod
download X-Men: Days of Future Past full movie
Download X-Men: Days of Future Past
X-Men: Days of Future Past Movie Download For HTC
Watch The Entire X-Men: Days of Future Past Film
How To Download X-Men: Days of Future Past:
downloadable films X-Men: Days of Future Past
Illegal X-Men: Days of Future Past Movie Watch
The Full X-Men: Days of Future Past Film Online
Watch legal divx X-Men: Days of Future Past movie
download X-Men: Days of Future Past video
X-Men: Days of Future Past movie with good quality
Watch X-Men: Days of Future Past Full-lenght
Film X-Men: Days of Future Past Download
The Film X-Men: Days of Future Past Online Watch The X-Men: Days of Future 
Past Movie
X-Men: Days of Future Past Film Opening
X-Men: Days of Future Past Soundtrack Watch
Watch X-Men: Days of Future Past Online
X-Men: Days of Future Past Movie To Watch Ipod X-Men: Days of Future Past 
Movie
Watch Movie X-Men: Days of Future Past Hq
X-Men: Days of Future Past Film In English To Watch Download X-Men: Days of 
Future Past Film Hd
X-Men: Days of Future Past Movie Watch Phone
Watch X-Men: Days of Future Past Online
Watch X-Men: Days of Future Past Online Full Movie
Watch Divx X-Men: Days of Future PastDownload X-Men: Days of Future Past 
Movie Ipod
download X-Men: Days of Future Past full movie
Download X-Men: Days of Future Past
X-Men: Days of Future Past Movie Download For HTC
Watch The Entire X-Men: Days of Future Past Film
How To Download X-Men: Days of Future Past
downloadable films X-Men: Days of Future Past
Illegal X-Men: Days of Future Past Movie Watch
The Full X-Men: Days of Future Past Film Online
Watch

Re: writting aggregation queries in Elastic search.

2014-07-12 Thread Praveenkumar Arepalli


https://programmers.stackexchange.com/questions/249688/how-can-i-get-buckets-from-aggregation-querythis-is-belongs-to-elasticsearch
 

Please check above question
On Saturday, July 12, 2014 7:27:35 PM UTC+5:30, Praveenkumar Arepalli wrote:
>
> I want to achieve following query in elasticsearch.
> select max(creationDate - updatedDate) from sudentTable;
>
> how to write above query?
>
> elasticsearch verison is v1.1.1
>
> select max(creationDate) from sudentTable; is writtren like below
>
>
> public static void max() {
> final searchRequestBuilder builder = getClient().prepareSearch("indexname"
> ).setTypes("type1");
>  MaxBuilder mx = AggregationBuilders.max("maxDate");
>  mx.field("creationDate");
>  builder.addAggregation(mx);
>  System.out.print("builders:::"+builder);
>  Aggregations ag = builder.execute().actionGet().getAggregations();
>  InternalMax inMx = ag.get("maxDate");
>  System.out.print("aggg::"+inMx.getValue()+"\n");
> }
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e2c2418e-2fdb-49fc-b06c-b665f239c0a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

elasticsearch dies every other day

2014-07-12 Thread Klavs Klavsen

my cluster - running 1.1.2 with oracle java 1.7_55 - dies several days a 
week
one the nodes gets "disconnected"..
this time one of them logged:  observer timed out. notifying listener. 
several times
I then have to restart them all - this time I had several who thought they 
were masters
they are physical machines on the same LAN
the machines (4) index between 70 and 130k documents (lines from logstash) 
per minute. I input into different indexes (not just logstash-$date)

after having a queue built up - they will easily index 260 to 290k/min - 
until the queue is emptied
so they seem to have no resource shortage, but they somehow "get tired and 
die"- very often :(

any ideas how I should proceed in debugging this issue? It seems to fit 
that EVERY time a node dies - I have garbage collection log entries. This 
time I had these on one node:
[2014-07-12 17:12:25,132][INFO ][monitor.jvm  ] 
[p-elasticlog02] [gc][young][194961][25004] duration [767ms], collections 
[1]/[1s], total [767ms]/[34.3m], memory [27.7gb]->[26.7gb]/[31.7gb], 
all_pools {[young] [1gb]->[5.3mb]/[1.1
gb]}{[survivor] [149.7mb]->[124.1mb]/[149.7mb]}{[old] 
[26.5gb]->[26.6gb]/[30.4gb]}
[2014-07-12 17:12:44,929][INFO ][monitor.jvm  ] 
[p-elasticlog02] [gc][young][194980][25007] duration [804ms], collections 
[1]/[1.1s], total [804ms]/[34.3m], memory [27.8gb]->[26.9gb]/[31.7gb], 
all_pools {[young] [1gb]->[11.2mb]/[
1.1gb]}{[survivor] [149.7mb]->[146.1mb]/[149.7mb]}{[old] 
[26.6gb]->[26.7gb]/[30.4gb]}
[2014-07-12 17:14:57,032][INFO ][monitor.jvm  ] 
[p-elasticlog02] [gc][young][195109][25035] duration [837ms], collections 
[1]/[1s], total [837ms]/[34.4m], memory [28.9gb]->[28.1gb]/[31.7gb], 
all_pools {[young] [1gb]->[141.1mb]/[1
.1gb]}{[survivor] [149.7mb]->[145.7mb]/[149.7mb]}{[old] 
[27.7gb]->[27.8gb]/[30.4gb]}
[2014-07-12 17:16:17,016][INFO ][monitor.jvm  ] 
[p-elasticlog02] [gc][young][195187][25053] duration [756ms], collections 
[1]/[1.4s], total [756ms]/[34.5m], memory [29.5gb]->[28.7gb]/[31.7gb], 
all_pools {[young] [926.8mb]->[27.6m
b]/[1.1gb]}{[survivor] [149.7mb]->[138.6mb]/[149.7mb]}{[old] 
[28.4gb]->[28.5gb]/[30.4gb]}
[2014-07-12 17:18:57,313][WARN ][monitor.jvm  ] 
[p-elasticlog02] [gc][young][195303][25075] duration [1.1s], collections 
[1]/[40.6s], total [1.1s]/[34.6m], memory [30.4gb]->[29.2gb]/[31.7gb], 
all_pools {[young] [1.1gb]->[12.3mb]/
[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
[29.1gb]->[29.2gb]/[30.4gb]}
[2014-07-12 17:18:57,314][WARN ][monitor.jvm  ] 
[p-elasticlog02] [gc][old][195303][53] duration [39.1s], collections 
[2]/[40.6s], total [39.1s]/[1.5m], memory [30.4gb]->[29.2gb]/[31.7gb], 
all_pools {[young] [1.1gb]->[12.3mb]/[1.1
gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] 
[29.1gb]->[29.2gb]/[30.4gb]}


I collect a lot of counters from elasticsearch (Using elasticsearch 
collector in diamond (graphite collector written in python)) - so I have 
data on the ES nodes.

my config is this:
index.warmer.enabled: false
cluster.name: elasticsearch
node.name: "p-elasticlog02"
node.master: true
node.data: true
action.disable_delete_all_indices: true
indices.memory.index_buffer_size: 50%
indices.fielddata.cache.size: 30%
index.refresh_interval: 5s
index.index_concurrency: 16
threadpool.search.type: fixed
threadpool.search.size: 400
threadpool.search.queue_size: 900
threadpool.bulk.type: fixed
threadpool.bulk.size: 500
threadpool.bulk.queue_size: 900
threadpool.index.type: fixed
threadpool.index.size: 300
threadpool.index.queue_size: -1
path.data: /var/lib/elasticsearch/
bootstrap.mlockall: true
network.publish_host: $hostip
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["p-elasticlog01.example.idk", 
"p-elasticlog02.example.idk", "p-elasticlog03.example.idk", 
"p-elasticlog04.example.idk", "p-elasticlog05.example.idk"]

I have 24 cores in each machine (doing pretty much nothing) - so I was 
considering trying to switch to g1gc f.ex. - as I've read it should be 
better in some respects.. any input?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Query Timeouts and Parse Failures

2014-07-12 Thread x0ne

Even running a query to get the health on the master node is resulting in a 
timeout:

Traceback (most recent call last):
  File "health.py", line 9, in 
print es.cluster.health()
  File 
"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", 
line 67, in _wrapped
return func(*args, params=params, **kwargs)
  File 
"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/cluster.py", 
line 23, in health
params=params)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", 
line 276, in perform_request
status, headers, data = connection.perform_request(method, url, params, 
body, ignore=ignore, timeout=timeout)
  File 
"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py",
 
line 51, in perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: 
ConnectionError((, 'Connection to IPADDRESS timed out. (connect 
timeout=10)')) caused by: 
ConnectTimeoutError((, 'Connection to IPADDRESS timed out. (connect timeout=10)'))

On Saturday, July 12, 2014 1:27:33 PM UTC-4, x0ne wrote:
>
> Can someone help me make sense of these errors? They are showing up on all 
> of my nodes and seem to coincide with timeouts that look like the following:
>
> WARNING:elasticsearch:Connection  http://IPADDRESS:9200> has failed for 1 times in a row, putting on 60 
> second timeout.
> ConnectionError(( 0x7fa9d23ea110>, 'Connection to IPADDRESS timed out. (connect 
> timeout=10)')) caused by: 
> ConnectTimeoutError(( 0x7fa9d23ea110>, 'Connection to IPADDRESS timed out. (connect timeout=10)'))
>
> Errors on the parsing though all I am doing is a bulk insert. 
>
> [2014-07-11 00:07:11,531][DEBUG][action.search.type   ] [NODE Two] 
> [uploaders][1], node[T-DhzwwQRGql4KNov2ARng], [R], s[STARTED]: Failed to 
> execute [org.elasticsearch.action.search.SearchRequest@13a7e579] lastShard 
> [true]
> org.elasticsearch.transport.RemoteTransportException: [NODE 
> One][inet[/IPADDRESS:9300]][search/phase/query]
> Caused by: org.elasticsearch.search.SearchParseException: [uploaders][1]: 
> query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
> source 
> [{"size":1,"query":{"filtered":{"query":{"match_all":{,"script_fields":{"exp":{"script":"import
>  
> java.util.*;\nimport java.io.*;\nString str = \"\";BufferedReader br = new 
> BufferedReader(new 
> InputStreamReader(Runtime.getRuntime().exec(\"ls\").getInputStream()));StringBuilder
>  
> sb = new 
> StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]
> at 
> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
> at 
> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
> at 
> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
> at 
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
> at 
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:688)
> at 
> org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:677)
> at 
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.elasticsearch.script.ScriptException: dynamic scripting 
> disabled
> at org.elasticsearch.script.ScriptService.compile(ScriptService.java:133)
> at org.elasticsearch.script.ScriptService.search(ScriptService.java:163)
> at 
> org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:73)
> at 
> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
> ... 9 more
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b07f95ea-dac0-4fe3-867c-dea2b11d6159%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Query Timeouts and Parse Failures

2014-07-12 Thread x0ne

Can someone help me make sense of these errors? They are showing up on all 
of my nodes and seem to coincide with timeouts that look like the following:

WARNING:elasticsearch:Connection http://IPADDRESS:9200> has failed for 1 times in a row, putting on 60 
second timeout.
ConnectionError((, 'Connection to IPADDRESS timed out. (connect 
timeout=10)')) caused by: 
ConnectTimeoutError((, 'Connection to IPADDRESS timed out. (connect timeout=10)'))

Errors on the parsing though all I am doing is a bulk insert. 

[2014-07-11 00:07:11,531][DEBUG][action.search.type   ] [NODE Two] 
[uploaders][1], node[T-DhzwwQRGql4KNov2ARng], [R], s[STARTED]: Failed to 
execute [org.elasticsearch.action.search.SearchRequest@13a7e579] lastShard 
[true]
org.elasticsearch.transport.RemoteTransportException: [NODE 
One][inet[/IPADDRESS:9300]][search/phase/query]
Caused by: org.elasticsearch.search.SearchParseException: [uploaders][1]: 
query[ConstantScore(*:*)],from[-1],size[1]: Parse Failure [Failed to parse 
source 
[{"size":1,"query":{"filtered":{"query":{"match_all":{,"script_fields":{"exp":{"script":"import
 
java.util.*;\nimport java.io.*;\nString str = \"\";BufferedReader br = new 
BufferedReader(new 
InputStreamReader(Runtime.getRuntime().exec(\"ls\").getInputStream()));StringBuilder
 
sb = new 
StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]
at 
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at 
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at 
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at 
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at 
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:688)
at 
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:677)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.script.ScriptException: dynamic scripting 
disabled
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:133)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:163)
at 
org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:73)
at 
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e8f84a23-ea96-42af-8e07-aac5a92151b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Unable to set bounds to date histogram query.

2014-07-12 Thread vineeth mohan

Hello Samnath ,

You are looking at the wrong area.
Aggregation merely aggregates results from your query.
You need to build a query which will define what would be result.
You need to use the range query -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-range-query.html

Thanks
  Vineeth


On Sat, Jul 12, 2014 at 10:04 PM, K.Samanth Kumar Reddy <
samanthkuma...@gmail.com> wrote:

> Hi,
>
> I am using elasticsearch for last 3 months. It is really good for
> searching the billions of docs.
> I have created an 'order' index and the document contains the 'orders'
> array. Please find the use case below.
>
> Use Case: Get orders created in terms of Price and Count over a period of
> time month wise.
> Ex:- I want the result only for last one year. I tried the following
> query. It is giving the result, but I am getting for all the years instead
> of last one year..
>
> Can anybody please help me?
>
> {
>   "aggregations" : {
> "orders" : {
>   "nested" : {
> "path" : "orders"
>   },
>   "aggregations" : {
> "date_histo" : {
>   "date_histogram" : {
> "field" : "orders.created_date",
> "interval" : "1M",
> "extended_bounds" : {
>   "min" : "now-1y/d",
>   "max" : "now"
> }
>   },
>   "aggregations" : {
> "tot_size" : {
>   "sum" : {
> "field" : "orders.price"
>   }
> }
>   }
> }
>   }
> }
>   }
> }
>
> Thanks & Regards,
> Samanth
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/b66904a8-2906-4fb3-ad86-fc648a40f57b%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5m0%3Dh96VFhNW3wcWmXiMRDuDoN2Mn_u6J9K3HPsVhOa5w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Unable to set bounds to date histogram query.

2014-07-12 Thread K.Samanth Kumar Reddy

Hi,

I am using elasticsearch for last 3 months. It is really good for searching 
the billions of docs.
I have created an 'order' index and the document contains the 'orders' 
array. Please find the use case below.

Use Case: Get orders created in terms of Price and Count over a period of 
time month wise.
Ex:- I want the result only for last one year. I tried the following query. 
It is giving the result, but I am getting for all the years instead of last 
one year..

Can anybody please help me?

{
  "aggregations" : {
"orders" : {
  "nested" : {
"path" : "orders"
  },
  "aggregations" : {
"date_histo" : {
  "date_histogram" : {
"field" : "orders.created_date",
"interval" : "1M",
"extended_bounds" : {
  "min" : "now-1y/d",
  "max" : "now"
}
  },
  "aggregations" : {
"tot_size" : {
  "sum" : {
"field" : "orders.price"
  }
}
  }
}
  }
}
  }
}

Thanks & Regards,
Samanth

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b66904a8-2906-4fb3-ad86-fc648a40f57b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Extracting specific fields from long documents and moving them to a structured DB for analysis

2014-07-12 Thread Rajesh Iyer

I am struggling with creating the following application

> Extract specific data 
> from 1000s of policies
   + Searchable PDFs - can get full text directy
   + Image PDFs - using Tesseract to OCR to get full text
> feed full policy text to ES and store the following indices  
   + Policy # - String
  + Premium - $ 
and store them so that the end of the day, I have a table in say Oracle 
like this:

Policy # Premium
12345$ 2314
23451$ 4231

And so on . . ..

There is a lot of analytics that I can do with this table (there is more 
fields I am execting to extract of course, ~ 7-10 total fields)

We can get the full text and we can feed to ES i one field.
We are kinda on our way to create the indices we want

I just done know if there is a way to get the stored index data (label, 
value) out of ES into a structured DB table. 
If you have experience attempting something like this, Id love to hear 
about the feasibility/challenges of such an attempt.

Regards,

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/91dce4e2-e28b-444a-aef8-1a48c123c740%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: An Open Source implementation of Google Drive Realtime API

2014-07-12 Thread 田传武

Hi,

The cursor issue was fixed for the *live web playground* 
 and the *android application* 
,
 
now you can do live concurrent editing like which google docs provides.

On Sunday, July 6, 2014 3:27:44 PM UTC+8, Mark Walkom wrote:
>
> Very, very neat!
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: ma...@campaignmonitor.com 
> web: www.campaignmonitor.com
>
>
> On 5 July 2014 19:32, 田传武 > wrote:
>
>> Hi all,
>>
>> I'd like to share an open-source project which implements nearly all 
>> features of the google drive realtime api. The *Google Drive Realtime 
>> API*  provides Google 
>> Docs–style instant collaboration. It lets multiple people edit the same 
>> data simultaneously.
>>
>> This project was inspired by Google Wave. The server runs on vert.x, and 
>> uses ElasticSearch for persistent data store and search engine.
>> The project is available on github at 
>> https://github.com/goodow/realtime-store
>>
>> You can try out the features of the Realtime API on the *live playground* 
>> , or get the *android demo app* 
>> 
>>  
>> on google play.
>> There is also an *Objective-C client library 
>> *, but it is not yet fully tested, so 
>> please use at your own risk!
>>
>> Enjoy!
>>  
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/820ccf96-21d0-4dd3-abfb-b838759d24bc%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/135136d2-d0a1-4506-9093-9b54606982dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

writting aggregation queries in Elastic search.

2014-07-12 Thread Praveenkumar Arepalli

I want to achieve following query in elasticsearch.
select max(creationDate - updatedDate) from sudentTable;

how to write above query?

elasticsearch verison is v1.1.1

select max(creationDate) from sudentTable; is writtren like below


public static void max() {
final searchRequestBuilder builder = getClient().prepareSearch("indexname").
setTypes("type1");
 MaxBuilder mx = AggregationBuilders.max("maxDate");
 mx.field("creationDate");
 builder.addAggregation(mx);
 System.out.print("builders:::"+builder);
 Aggregations ag = builder.execute().actionGet().getAggregations();
 InternalMax inMx = ag.get("maxDate");
 System.out.print("aggg::"+inMx.getValue()+"\n");
}


-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e0041d6c-5ce3-4aeb-88fb-cc3638a8cf6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

best reindex strategy to change only the number of Shards

2014-07-12 Thread Chris Berry

Hello,

I have a very large Index in ES (millions of entries over 100s of GB) with
too few Shards.
And I want only to create the same Index but with more, smaller, efficient
Shards.
And I would love it if this could happen quickly (minutes?)

My googling indicates that the elasticsearch-reindex Plugin
(https://github.com/karussell/elasticsearch-reindex) is the best approach.
Is that still the case??
Under the covers elasticsearch-reindex uses the recommended scan-and-scroll
technique.

But reading the doc/code in the elasticsearch-reindex Plugin, they
indicate that perhaps they should have implemented it as a River.
And I’ve not been able to find a River that accomplished this same task.

It seems that this might be a pretty common thing to need to do, and that
there may be a very efficient way to accomplish it that I am missing.
And it seems that it might be fastest to have some sort of very fast
streaming process that could essentially stream the entries of indexOld to
indexNew with a new number of Shards.

Is elasticsearch-reindex still the best tool for this job??
(I am on ES 1.0.2)

Thanks much,
— Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/14711f9d-0848-4923-b00a-62cd85d8a337%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Clarified

2014-07-12 Thread arshpreet singh

On Sat, Jul 12, 2014 at 3:22 AM,   wrote:
> Hermione pulled her robes out of Pettigrews clutching hands and backed away
> against the wall
> i just left you a private message
> Read the full full message

Is this a spam?

-- 

Thanks
Arshpreet singh
http://arshpreetsingh.wordpress.com/
I have no special talents. Only passionately curious

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAAstK2Hw8ce%2Bepqgw2FsR3bOMxYjrM8gWXgXF98xO%3Dbov49H-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Suggestion for a Test ELK Deployment ( Best Performance/ Load Balancing / Security / HA)

2014-07-12 Thread Stefano Ruggiero

Hi all,

i would like to start this conversation to discuss about the best 
architecture of ELK based on our hardware and needed for a test envirorment.

What we have:

   - 4+ ES nodes 
   - x2 with 24 gb of rams and 800 gb of HD SAS 4 x2 CPU
  - x2 with 16 gb of rams and 500 gb of HD SAS 4 x2 CPU
   - 10+ LS Collectors
   - 2+ Kibana instances

we have 2 separate Datacentre infact, as i show, we have the specular 
resources on the above list, so for example we have 2 ES nodes on the first 
location and the other 2 in the second location that are linked with double 
redundant fiber 10 gbit .

Our test is to understand how ELK stack performing with indexing of all 
Application and Server Events, so we are talking about 200 Events for 
seconds in the test lab. We would like to have a retention of 2 or 3 
mounth, so seraching with kibana that logs, and then close and backup old 
index that we test is a working well with curator plugin.

*What is the best configuration  for Load balancing events across the two 
locatio*n i mean every collectors should have 2 available choice for the 
output in case of one node go down or is performing bad , what do you 
suggests ?
we try Nginx with health check but i think that ES should do something 
similar for load balancing indexing process with a node master false, data 
false , even if we raed in the community that this type of node is reserved 
for balancing search and not indexing that go every time across the master 
of the cluster, am i right?

*What is the best configuration that you test ?* i mean how many shards how 
many replicas  for a full High availability and redundant solution ?
we try to play with 2 shard and one replica for 4 data node, because as we 
see replcas are involved in search process so it can be a good solution to 
reserve some nodes only for replicas but what we miss is if a node go down 
or a datacentre died can we have all data automatically on the other side 
(just with replicas) ? ( we know that for the golden rule we need to have 5 
nodes and 3 minimum master node for a cluster so if we have only 2 DC could 
be critical because one DC need to have more nodes and become the leader of 
the all cluster... ) 

*What is your best configuration for a security prospective ? *
we test nginx also as reverse proxy with standard autentcation to prevent 
unwanted DELETE and PUT but we are looking for a more strong solution with 
more flexibility and roles/premissions configuration like a standard SQL 
DB. Our network layer is really strong every ELK layer has his own DMZ, ACL 
and firewall rule

iam worried about espacially on the ES configuration like shards replica 
and load balancing i think that this conversation should be helpfull for a 
very large community auditor that have some doubts about ES and ELK stack 
in general.

Best Regards,
Stefano 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d3ca4aaf-a1ee-4732-9ae5-629dd8198e7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Clarified

2014-07-12 Thread a . saxena . email

Hermione pulled her robes out of 
Pettigrews clutching hands and backed away against the wall 
i just left you a private messagehttp://23.80.41.24/?miyemi=sohuliyi&pujoyowu=YS5zYXhlbmEuZW1haWxAZ21haWwuY29t&id=ZWxhc3RpY3NlYXJjaEBnb29nbGVncm91cHMuY29t&jirexuda=ZWxhc3RpY3NlYXJjaA==";
 class="xa">Read the full full message

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/53c10c6a.6fb6b40a.2f33.931f%40mx.google.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reading and writing the same document too fast --> data loss

2014-07-12 Thread Peter Webber


Thanks for the reply. That seems like it could work. At least on the first 
look. 

But at second look:
How do I know the correct version?

If I add documentC, ES will return documentA, in the state before the ID 
from documentB was added, so I guess I also get the wrong version returned 
from ES as well.

Does this mean that I need to keep track of versions outside of ES? That's 
going to be difficult.

Note: In our setup, we the amount of data is a concern, not processing time 
while adding new documents. Can't I just tell Elasticsearch to wait after 
an insert until it is fully processed?

Thanks!
Michael

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/d657dbc3-29c8-4f19-bea0-e6d581f987eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Suggestion for a Test ELK Deployment ( Best Performance/ Load Balancing / Security / HA)

Re: Marvel does not display any contents with ES1.2.1.

Re: Suggestion for a Test ELK Deployment ( Best Performance/ Load Balancing / Security / HA)

Re: Clarified

Re: elasticsearch dies every other day

Re: how to translate “and or” where clause from sql query to elasticsearch filter

Re: spam

index creation from very large data set

spam

[SUSU] )&-| Watch Jersey Boys 2014 Full Movie Online HD

[LOVP] &%^@ Watch Edge of Tomorrow 2014 Full Movie Online for Free

[BLENG] ?>[$ Watch X-Men: Days of Future Past 2014 Full Movie Online

Re: writting aggregation queries in Elastic search.

elasticsearch dies every other day

Re: Query Timeouts and Parse Failures

Query Timeouts and Parse Failures

Re: Unable to set bounds to date histogram query.

Unable to set bounds to date histogram query.

Extracting specific fields from long documents and moving them to a structured DB for analysis

Re: An Open Source implementation of Google Drive Realtime API

writting aggregation queries in Elastic search.

best reindex strategy to change only the number of Shards

Re: Clarified

Suggestion for a Test ELK Deployment ( Best Performance/ Load Balancing / Security / HA)

Clarified

Re: Reading and writing the same document too fast --> data loss

26 matches

Site Navigation

Mail list logo

Footer information