Re: How to compare data based on different dates Using Kibana3

[Elvar Böðvarsson]

This will be possible in Kibana4 with scripted fields

On Monday, December 29, 2014 9:22:51 PM UTC, Ramakrishna N wrote:
>
> Hi,
>
> I have a question regarding the ElasticSearch and Kibana.
>
> My goal is I have to compare two different dates of data by hour/min/sec.
>
> All my data is indexed and everything in single type.
>
> Regards,
> Ramakrishna Namburu
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fde48619-c6bf-45ab-94db-819c170e3c35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Visualize stats of MS SQL tables using ElasticSearch

[Elvar Böðvarsson]

1. JDBC River to get data into Elasticsearch
2. Kibana to visualize everything

On Tuesday, December 30, 2014 4:45:31 AM UTC, Ashutosh Parab wrote:
>
> What I am doing is loading my MS SQL database into ElasticSearch. I want 
> to perform different types of aggregations/statistics correaltions on the 
> rows of those tables. So I wanted to know whether there is any tool to 
> visualize such data. 
> Is there a tutorial to demonstrate how this can be done?
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9fc4dfe2-daa8-4fed-bfe5-4bdaea4c9d6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Connecting Linux and Windows Eastic servers

[Elvar Böðvarsson]

Should be fine if you configure the Windows nodes as master:no and data:no

I read discussion a while ago where a user had a mix of windows and linux 
machines as data nodes. Worked fine, but it was kinda risky, fine for a 
short time during a migration or upgrade, but dont do it for a long time.

On Wednesday, December 17, 2014 10:02:59 AM UTC, Gal Artsi wrote:
>
> hey guys! i need a little help here.
> i am trying to create a cluster with 3 linux machines and a few windows 
> machines, but i want the windows machine to do nothing and that all of the 
> work will be on the linux maching.
> my question is this:
> can i configure the windows machine to see the linux servers, and the 
> linux servers will only see thmeselves? will it work?
> thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b5ce194a-d8a4-47ad-b180-58999780aef8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is ElasticSearch truly scalable for analytics?

[Elvar Böðvarsson]

Elasticsearch supports tribe nodes, so you can combine multiple clusters, 
you then query the tribe node to access data on all of them.

On Monday, December 15, 2014 9:52:45 PM UTC, Yifan Wang wrote:
>
> If I understand correctly, ElasticSearch directly sends query to and 
> collects aggregated results from each shard. With number of shards 
> increases, Reduce phase on the Client node will become overwhelmed. 
>
> One would assume, if ElasticSearch support node level aggregation, the 
> "Reduce" becomes distributed so Client node will not become overwhelmed for 
> large clusters with lots of shards. I am wondering if ElasticSearch 
> supports node level reduce. If not, why? I think this is critical if we 
> like ElasticSearch to be truly scalable for analytics. 
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/baffc6eb-2f28-4498-adf7-8b9628da7d0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: To Raid or not to Raid

[Elvar Böðvarsson]

That is quite scary

Lets say I have 3x nodes each with 4x disks, a total of 12x disks. Two disk 
fails would mean a data loss if its set to 1 replicas, with 2 replicas it 
would mean no data loss but with the cluster availability of n+1 nodes it 
would take down the whole cluster if one disk failsin two hosts.

Maybe It would be smart to split the elasticsearch processes or run two 
VM's per host.

https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/

If I did it like this then each process would have access to 2x disks 
meaning if one disk fails only two disks will be unavailable to the cluster 
instead of four.



On Monday, December 15, 2014 12:05:18 PM UTC, Mark Walkom wrote:
>
> Unfortunately you lose all data on the node as ES will stripe segments 
> across the disks/mount points.
>
> On 15 December 2014 at 11:45, Elvar Böðvarsson  > wrote:
>>
>> If you have a node that has 4x disks as JBOD and you configure 
>> Elasticsearch to use all of them, so it will write to as like its Raid0. 
>>
>> How does Elasticsearch handle a failure of one disk?
>> Will the whole node go down or will Elasticsearch continue to function 
>> just with lower total available storage? (and then recreate the shards that 
>> went down)
>>
>>
>>
>> On Saturday, December 13, 2014 3:48:55 PM UTC, Jörg Prante wrote:
>>>
>>> The statement is related to performance and I can't agree with it. You 
>>> can easily build a RAID 0 system which has massive I/O throughput 
>>> performance and is superior to JBOD, because RAID striping does not slow 
>>> things down, it is as always as much as fast than a single drive and in 
>>> most RAID levels it is much faster. 
>>>
>>> In the past, RAID was invented for mirroring cheap and error-prone 
>>> spindle disk arrays, while mirrors increase costs but decrease fault 
>>> probability.
>>>
>>> With Elasticsearch, the decision is if you still want to handle disk 
>>> faults by drive redundancy (RAID) and all other hardware faults like power 
>>> outages by server downtime. This is just a matter of organization and of 
>>> cost. I would suggest from my experience: take control over your complete 
>>> hardware setup, equip your systems with expensive SAS2 (or even better) 
>>> controllers with RAID 0 to reduce cost and maximize performance, and handle 
>>> all kind of hardware faults by server downtime, because ES replica level > 
>>> 0 allows that.
>>>
>>> There is also a simplification of SAN/NAS in the statement but that is a 
>>> different discussion. Never use SAN/NAS for ES local gateway.
>>>
>>> Jörg
>>>
>>> On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson  
>>> wrote:
>>>>
>>>>
>>>> Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, 
>>>> maybe then to be safe go with 2x replicas, goes well with having 3x nodes
>>>>
>>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7e2260da-7082-4957-8302-57bf2a912b7b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Marvel Index Taking too much disk space

[Elvar Böðvarsson]

Curator or just make a script that calls the API

On Monday, December 15, 2014 6:31:06 AM UTC, Chetan Dev wrote:
>
> Hi,
>
> I installed the plugin for marvel , 
> but its creating its own indexes which are even larger than my original 
> indexed data.
> is there a way to delete these indexes on daily basis or any other way ?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/2278326e-b90e-4fe9-b5c6-c3929cf2e046%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: To Raid or not to Raid

[Elvar Böðvarsson]

If you have a node that has 4x disks as JBOD and you configure 
Elasticsearch to use all of them, so it will write to as like its Raid0. 

How does Elasticsearch handle a failure of one disk?
Will the whole node go down or will Elasticsearch continue to function just 
with lower total available storage? (and then recreate the shards that went 
down)



On Saturday, December 13, 2014 3:48:55 PM UTC, Jörg Prante wrote:
>
> The statement is related to performance and I can't agree with it. You can 
> easily build a RAID 0 system which has massive I/O throughput performance 
> and is superior to JBOD, because RAID striping does not slow things down, 
> it is as always as much as fast than a single drive and in most RAID levels 
> it is much faster. 
>
> In the past, RAID was invented for mirroring cheap and error-prone spindle 
> disk arrays, while mirrors increase costs but decrease fault probability.
>
> With Elasticsearch, the decision is if you still want to handle disk 
> faults by drive redundancy (RAID) and all other hardware faults like power 
> outages by server downtime. This is just a matter of organization and of 
> cost. I would suggest from my experience: take control over your complete 
> hardware setup, equip your systems with expensive SAS2 (or even better) 
> controllers with RAID 0 to reduce cost and maximize performance, and handle 
> all kind of hardware faults by server downtime, because ES replica level > 
> 0 allows that.
>
> There is also a simplification of SAN/NAS in the statement but that is a 
> different discussion. Never use SAN/NAS for ES local gateway.
>
> Jörg
>
> On Fri, Dec 12, 2014 at 7:28 PM, Elvar Böðvarsson  > wrote:
>>
>>
>> Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, 
>> maybe then to be safe go with 2x replicas, goes well with having 3x nodes
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/fad86579-2072-438d-94da-80219e200b67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: To Raid or not to Raid

[Elvar Böðvarsson]

Just went through these slides

https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup

First of all, thats one HUGE cluster

Second, "Prefer JBODs for data disks over RAID, SAN/NAS", would be ok, 
maybe then to be safe go with 2x replicas, goes well with having 3x nodes


On Friday, December 12, 2014 4:18:01 PM UTC, Jack Park wrote:
>
> I cannot put my finger on it, but I think I recall that someone in this 
> group once said that ES is IO intensive, that RAID would slow things down, 
> arguing in favor of redundant servers over RAID. Does that still make sense?
>
> On Fri, Dec 12, 2014 at 8:00 AM, Mark Walkom  > wrote:
>
>> It's a risk thing, you need to be comfy with the risk of losing one disk 
>> and all that it entails.
>> If you can mitigate that through a process and you are happy with the 
>> remaining risk, then :)
>>
>> On 12 December 2014 at 16:13, Elvar Böðvarsson > > wrote:
>>
>>> This would be for log storage, plan is to go with HDD's for the long 
>>> term storage and SSD's for short term storage. Total storage will be maybe 
>>> 8tb total, using replica's would bring that to 16tb and if using raid1 or 
>>> raid10 would bring total raw storage up to 32tb
>>>
>>>
>>>
>>> On Friday, December 12, 2014 1:30:32 PM UTC, Nikolas Everett wrote:
>>>>
>>>> Striping raid is viable for 2 or 3 disks because of the redundancy.  
>>>> Software raid works fine for me. Hardware raid enables battery backed 
>>>> write 
>>>> behind but I don't know how important that is with ssds. Either way, we go 
>>>> 2xSSDs per server with os in mirrored raid and data striped.
>>>>
>>>> Depending on your data you may want spinning disks instead, then 
>>>> battery backed writes are probably a bigger win. 
>>>> On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson"  wrote:
>>>>
>>>>> When running Elasticsearch on physical hardware you have it create 
>>>>> replicas to make sure no node is a single point of failure. From 
>>>>> everyone's 
>>>>> experiance should I use Hardware Raid as well, or is it not needed?
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>> an email to elasticsearc...@googlegroups.com.
>>>>> To view this discussion on the web visit https://groups.google.com/d/
>>>>> msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%
>>>>> 40googlegroups.com 
>>>>> <https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>  -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to elasticsearc...@googlegroups.com .
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9_ULd9j7jFVJaMYvLz3NNwYsqCAwHsfUxkxCwg1oV3-A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/02040a16-9718-4fc8-be28-4ff781de5c40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: To Raid or not to Raid

[Elvar Böðvarsson]

This would be for log storage, plan is to go with HDD's for the long term 
storage and SSD's for short term storage. Total storage will be maybe 8tb 
total, using replica's would bring that to 16tb and if using raid1 or 
raid10 would bring total raw storage up to 32tb



On Friday, December 12, 2014 1:30:32 PM UTC, Nikolas Everett wrote:
>
> Striping raid is viable for 2 or 3 disks because of the redundancy.  
> Software raid works fine for me. Hardware raid enables battery backed write 
> behind but I don't know how important that is with ssds. Either way, we go 
> 2xSSDs per server with os in mirrored raid and data striped.
>
> Depending on your data you may want spinning disks instead, then battery 
> backed writes are probably a bigger win. 
> On Dec 12, 2014 7:32 AM, "Elvar Böðvarsson"  > wrote:
>
>> When running Elasticsearch on physical hardware you have it create 
>> replicas to make sure no node is a single point of failure. From everyone's 
>> experiance should I use Hardware Raid as well, or is it not needed?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/13b36ab6-7fb6-4f6f-b11d-79069aed74ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

To Raid or not to Raid

[Elvar Böðvarsson]

When running Elasticsearch on physical hardware you have it create replicas 
to make sure no node is a single point of failure. From everyone's 
experiance should I use Hardware Raid as well, or is it not needed?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/4827efb1-b5cf-4407-97ef-f6c4c3029cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Shield release date

[Elvar Böðvarsson]

Do you know if it will it be priced similarly to Marvel?

On Friday, December 12, 2014 8:34:41 AM UTC, Mark Walkom wrote:
>
> It's still undergoing beta testing, I haven't heard a GA release yet but 
> I'll ask!
>
>
> (If you're not aware, Shield is a paid product, like Marvel it's not an 
> open source product like the core ELK stack.)
>
> On 12 December 2014 at 00:04, Ben McCann  > wrote:
>
>> Hi,
>>
>> Is there any rough information you can share (privately is fine) about a 
>> rough Shield release date? Anything rough would help (weeks, months, 
>> quarters away, etc.)
>>
>> Thanks,
>> Ben
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/b93b5668-afcf-4d81-a560-bb951579a30b%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/afb0dfd5-464b-438a-acda-6b5bdb7156b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Securing apis

[Elvar Böðvarsson]

Setting up nginx on windows is actually very easy since they provide a 
native binary.

But, take a look at IIS ARR to do reverse proxy for elasticsearch, can do 
kerberos authentication, give access based on active directory and limit 
what methods are available.

On Thursday, December 11, 2014 6:06:09 AM UTC, Chetan Dev wrote:
>
> Hi,
>
> I am not able to set up nginx on windows can you help me ?
>
>
> On Monday, December 8, 2014 7:22:02 PM UTC+5:30, Elvar Böðvarsson wrote:
>>
>> Front it with a reverse proxy, limit access to the DELETE method
>>
>> On Monday, December 8, 2014 6:04:52 AM UTC, Chetan Dev wrote:
>>>
>>> Hi,
>>>
>>> Is there any way i can secure the Apis ?
>>> I  want  to restrict the rights so that nobody other than administrator 
>>> is able to delete or update indexes 
>>>
>>> Thanks
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/b892f4f7-182d-4e10-a7c3-38d95d30d6b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Nginx

[Elvar Böðvarsson]

If you can, then use IIS ARR as a reverse proxy, can do easy kerberos 
authentication as well

On Thursday, December 11, 2014 7:48:41 AM UTC, Chetan Dev wrote:
>
> Hi,
>
> How do i generate nginx htpassword in windows?
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/f2442f32-d4cf-486d-879e-83a95b6aa169%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enabling Live/Live Disaster Recovery while avoiding split brain

[Elvar Böðvarsson]

If I remember correctly, version 1.4 can turn nodes that cant connect to 
the cluster to read only mode.

On Thursday, December 11, 2014 4:44:28 PM UTC, David Artus wrote:
>
>
> My understanding is that recovery from Split Brain situations is 
> troublesome and we are encourages to ensure that a cluster is only active 
> if there is a quorum of candidate masters , we do that by setting 
> *discovery.zen.minimum_master_nodes*  to ((cm/2) + 1), that is a true 
> majority of candidate masters.
>
> I also see that many folks want to enable resilience in the event of 
> disaster by running a cluster across two data centres. Lose one data center 
> and we just keep running in the other - we still have half our servers, so 
> with correct over-provisioning we can cope with the workload. No need to 
> rebuild the cluster and restore from backup. We have what may be called 
> Live/Live Disaster Recovery (DR)
>
> I see two issues with this approach to DR. If we lose half our candidate 
> master nodes, by definition we cannot achieve a quorum, we won't have a 
> true majority. 
>
> A slightly more subtle problem is that we probabaly also set 
> *gateway.recover_after_nodes* to a high value (say 8 of 10) so that in 
> normal running with a few missing nodes we don't inadvertantly get into 
> shard shuffling. Once again with a loss of half our estate we will not 
> have enough nodes to satisfy that setting.
>
> My conclusion is that while Live/Live operation across two Data Centers is 
> possible the transition to a reduced state requires reconfiguration, the 
> system will not "just keep running" unless we open ourselves to Split Brain 
> and replica shuffling.
>
> Comments please?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/6dcfdeed-370f-426a-a5b1-f72767838d7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are rivers still being deprecated?

[Elvar Böðvarsson]

I know that feeders will replace them, for example the JDBC plugin ships 
with a early version of a feeder but it only works in Linux at the moment.

Are there any project out there that provide a feeder functionality similar 
to what rivers are now?

Also, the bad thing about rivers is that the river is running inside a 
Elasticsearch JVM. Would that not be mitigated by having a seperate 
instance of Elasticsearch running the river process that would be a client 
to the cluster?


On Friday, November 7, 2014 2:17:36 AM UTC, Alexandre Rafalovitch wrote:
>
> ElasticSearch 1.4 is out and I can't see any mentions that Rivers are 
> deprecated. 
>
> Has that (informal) decision been reversed? Or was the timeline 
> further out? What's the currently recommended approach? 
>
> Regards, 
>Alex. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e853e0cb-7cfb-4f72-a256-08fa74b5e735%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Securing apis

[Elvar Böðvarsson]

Front it with a reverse proxy, limit access to the DELETE method

On Monday, December 8, 2014 6:04:52 AM UTC, Chetan Dev wrote:
>
> Hi,
>
> Is there any way i can secure the Apis ?
> I  want  to restrict the rights so that nobody other than administrator is 
> able to delete or update indexes 
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5634b9b5-8d41-48fa-8951-580e5e26cbf2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch hardware planning

[Elvar Böðvarsson]

The good thing if you use a raid controller for RAID0 is that you get all 
the queuing and buffering features. What I will end up using in this case 
will most likely go down to cost and recovery options in case of a failure. 
Might even go with other Raid levels, 1, 10 or 6.

Read this 
article, 
https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/
 
that brings up a very interesting point. To split elasticsearch processes 
into 8gb chunks. 

Regarding the SSD options, what would be the best way to go about doing 
that?

Also, has anyone done a performance comparison between Windows VS. Linux 
for Elasticsearch?
* I have read articles, old ones, that state that JVM performance is better 
on Windows
* Went to a high performance session at VMworld regarding JVM performance, 
the speaker did not want to go into the pit of discussing witch was better. 
He did though point out that JVM in Windows works in 1mb memory chunks 
while Linux JVM uses 256k chunks.


On Friday, December 5, 2014 1:57:27 AM UTC, Mark Walkom wrote:
>
> RAID is useful, you just need to understand the limits. And the potential 
> for data loss with multiple ES nodes writing to multiple data directories 
> is not inconsequential if it's an important system with business 
> requirements.
> To reiterate because it's really important this is known - if you lose one 
> of the data.dir points on a node you lose *all* data on the node. The ES 
> dev team has had talks about improving this so things are written on a 
> segment level rather than a direct stripe, but there's no ETA on that that 
> I am aware of.
>
> ES will obviously be limited by the OS/FS/hardware/etc throughput of any 
> given channel, I haven't seen anyone do testing of RAID0 versus ES striping 
> though so it's an interesting question.
>
> On 5 December 2014 at 12:11, Kevin Burton  > wrote:
>
>> Saying that RAID is good for anything is a bit of a stretch :-P
>>
>> I'm not sure how good ES is with splitting the index across volumes but 
>> the database has a lot more options here for load distribution.  RAID is 
>> naive by design and the optimizations a RAID controller/impl are limited.
>>
>> If ES can outperform RAID0 zero then that might be the better route ... 
>> just not sure how good ES is at it though.
>>
>> On Thursday, December 4, 2014 2:42:25 PM UTC-8, Mark Walkom wrote:
>>>
>>> Be aware that using multiple data locations in ES is akin to RAID0; 
>>> which means if you lose a disk then you lose all the data on that node.
>>> Personally, I'd suggest you leverage hardware RAID and let it do what it 
>>> is good at, otherwise you just have more management overhead and greater 
>>> risk of a hardware failure causing a bigger problem.
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/68a76c15-e363-4d01-9133-bdebd8797a21%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/9b1295b9-1155-4761-beb6-4ef4d30026b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: When a master node goes down, how the client query works?

[Elvar Böðvarsson]

Two options

1. Have a client instance of elasticsearch on a different server or on the 
same server that does the query. That node must be set to master=false and 
data=false. Being a member of the cluster means you know where the data is.
2. Use a http reverse proxy that connects to all the nodes in the cluster, 
if http 9200 is unavailable on one node then traffic is sent to the other 
nodes.

Best option is to combine the two. Have two elasticsearch nodes as client 
members of the cluster, install a reverse proxy on those two and 
loadbalance between them on the IP level with a solution of choice.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/7e0c1382-399f-4a0b-ba77-59fa64c9ac2c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to Upgrade from 1.1.1 to 1.2.2 in a windows enviroment (as windows service)

[Elvar Böðvarsson]

Use NSSM ( http://nssm.cc/ ) to create the service instead.

Organize your folders like this

C:\Elasticsearch\
C:\Elasticsearch\nssm.exe
C:\Elasticsearch\elasticsearch.bat
C:\Elasticsearch\elasticsearch-1.1.0\
C:\Elasticsearch\elasticsearch-1.2.2\
C:\Elasticsearch\data
C:\Elasticsearch\logs
C:\Elasticsearch\the other folders

in elasticsearch.bat only add the full path to the elasticsearch.bat that 
is included with each version, for example 
C:\Elasticsearch\elasticsearch-1.1.0\bin\elasticsearch.bat

Make sure your config has the folders C:\Elasticsearch\data defined

When upgrading, just add the new version in a folder, modify the bat in the 
root and restart the service.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/02085eb9-68a5-4d9d-8538-3504822a10c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Advices on migrating 1.3.2 to 1.4.1

[Elvar Böðvarsson]

I upgraded our logging cluster to 1.4 without any problems.

When I looked into upgrading a separate dev/test instance used for a 
different purpose I ran into problems with the plugins. If you are using 
plugins, make sure they are supported in 1.4.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1873d1cb-6f49-413d-8157-1220b64411e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch hardware planning

[Elvar Böðvarsson]

I am preparing proposals on hardware for our Elasticsearch log storage.

What I would love to have are SSD's for most recent logs or SSD's for hot 
data. For that I have come down to two solutions with 3x physical servers.

1. Use Windows 2012 R2 as the OS, use Storage Spaces to prvide a tiered 
storage for SSD's and HDD's to Elasticsearch.

2. Install ESXi. Create two VM's on each server, one that has access to SSD 
disks and one who has access to HDD's. That will give me a 6 node 
Elasticsearch cluster. Use tags to keep latest indices on the SSD nodes, 
when they are x days old a script or curator will remove the SSD tag from 
the index and add a HDD tag, hopefully resulting in a migration to the HDD 
nodes. Either Windows 2012 R2 or Ubuntu will be used here.

Each Elasticsearch node will get either 32gig memory or 64gig memory. 
Undecided on that at the moment, might go with the lower amount to have the 
option of expanding it if there is need. With 192gigs in the cluster there 
might not even be a need for SSD's.

Not sure about the number of cores.

Plan to skip raid on everything except the OS disks and use Elasticsearch 
striping for the data. Total storage will be about 20gigs a day without 
replication, and total storage will be about 15tb.

Any angles I'm missing?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/1e6bbc44-fc3b-4724-b657-63722a951d06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using ES as a primary datastore.

[Elvar Böðvarsson]

Like others have mentioned, there is a risk of data loss at the moment but 
Elasticsearch is working on making it better and better.

I watched this video about Couchbase and 
Elasticsearch, https://www.youtube.com/watch?v=rpwtxpmuDb0

After that then I highly recommend researching Couchbase as an option.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: alerts from Kibana/ES

[Elvar Böðvarsson]

Use this input filter in Logstash to search the logs

http://logstash.net/docs/1.4.1/inputs/elasticsearch

On Tuesday, May 27, 2014 9:02:35 AM UTC, NF wrote:
>
> Hi,
>
> We’re using Kibana/Elasticsearch to visualize different kind of logs in 
> our company. Now, we would need a feature that would allow us to send an 
> alert/notification (email or other) when a certain event/trigger is 
> captured.
>
> I’d like to know if in Kibana/Elasticsearch backlog there is such a 
> feature planned? If so, when might we expect it available? 
>
> If not, could you please suggest any (open source) solution to satisfy our 
> need?
>
> Thanks,
>
> Natalia
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/cef00f3f-1e7d-44be-9af8-6e963d1e8c24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marvel Sense when its on a different cluster?

[Elvar Böðvarsson]

I run one machine that gathers all the Marvel data from the main ES 
cluster. Works fine and Marvel shows me all the information from the main 
cluster.

The issues is when I start Sense it only connects to the Marvel ES instance 
and not the main one. How can I get it to connect to the main cluster?

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/527f846c-3eb9-4343-a39f-228e79ba6b01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.