[graylog2] Re: Graylog 2.x upgrade

2016-11-04 Thread Denny Gebel 
Did you manage to fix this issue?

I am still in need to upgrade...

Thanks,

Denny

Am Montag, 6. Juni 2016 15:05:13 UTC+2 schrieb Robert Hough:
>
> Between getting sick on Friday and becoming fire marshall bill last 
> week... I haven't had any time to work on this.  The biggest issue is that 
> I'm not at all familiar with logstash, so I need to figure out the 
> filtering stuff. I still believe the idea is sound;  its just a matter of 
> time.  I'm most likely going to shelf the reindex portion of our upgrade 
> for the time being and circle back to that when I have a little more time. 
> I'll hit you up with details when I have some though.
>
>
> On Thursday, June 2, 2016 at 1:12:27 PM UTC-4, Jimmy Chen wrote:
>>
>> Good luck with the fires and I'll check back to see how it went.
>>
>> On Thursday, June 2, 2016 at 6:03:34 AM UTC-7, Robert Hough wrote:
>>>
>>> Well, "out of the box", no that didn't work.  I've got faith that it can 
>>> be done using this approach, but we'll also need to utilize Elastic's 
>>> "de_dot" filter plugin.  I'm hoping to make some progress with that today, 
>>> and I'll provide an update by the end of the day. I've got about 10 fires 
>>> to put out first... :(
>>>
>>> Here's the link to the de_dot documentation:
>>>
>>> https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html
>>>
>>>
>>> In a nutshell:
>>>
>>> 1) Logstash pulls in old index data from old ES cluster
>>> 2) Logstash sends that through filter
>>>   1a)  Match any dots in fields  (user.id)
>>>   2a)  Add new field as replacement for old field (user.id == user_id)
>>>   3a)  Populate user_id with value from user.id
>>>   4a)  remove old field (user.id)
>>> 3) Logstash pushes new index data to new ES cluster
>>>
>>> I'm sure I've left out something crucial here. Seems to be par for the 
>>> course, but I'm hopeful. :)
>>>
>>>
>>>
>>>
>>> On Wednesday, June 1, 2016 at 3:06:34 PM UTC-4, Jimmy Chen wrote:

 Did this work for you? I am going to be looking into upgrading our 
 existing cluster to 2.x too.

 On Tuesday, May 31, 2016 at 5:08:05 PM UTC-7, Robert Hough wrote:
>
> Came across this:  
> https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06 
> 
>
> third time's the charm?  :)
>
>
> On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:
>>
>> Recently built a Graylog 2.x cluster, and that seems to be working 
>> fine.  I had some questions though, but right now the biggest nagging 
>> question has been...
>>
>> How do we migrate our existing indexes over to the new system?  The 
>> whole dots in field names issue seems to be what is preventing us from 
>> pulling this off.  How do we correct these, and then import them into 
>> the 
>> our new system? 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/95734cbb-5577-4a4c-9266-76806d4c7246%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.x upgrade

2016-06-06 Thread Robert Hough 
Between getting sick on Friday and becoming fire marshall bill last week... 
I haven't had any time to work on this.  The biggest issue is that I'm not 
at all familiar with logstash, so I need to figure out the filtering stuff. 
I still believe the idea is sound;  its just a matter of time.  I'm most 
likely going to shelf the reindex portion of our upgrade for the time being 
and circle back to that when I have a little more time. I'll hit you up 
with details when I have some though.


On Thursday, June 2, 2016 at 1:12:27 PM UTC-4, Jimmy Chen wrote:
>
> Good luck with the fires and I'll check back to see how it went.
>
> On Thursday, June 2, 2016 at 6:03:34 AM UTC-7, Robert Hough wrote:
>>
>> Well, "out of the box", no that didn't work.  I've got faith that it can 
>> be done using this approach, but we'll also need to utilize Elastic's 
>> "de_dot" filter plugin.  I'm hoping to make some progress with that today, 
>> and I'll provide an update by the end of the day. I've got about 10 fires 
>> to put out first... :(
>>
>> Here's the link to the de_dot documentation:
>>
>> https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html
>>
>>
>> In a nutshell:
>>
>> 1) Logstash pulls in old index data from old ES cluster
>> 2) Logstash sends that through filter
>>   1a)  Match any dots in fields  (user.id)
>>   2a)  Add new field as replacement for old field (user.id == user_id)
>>   3a)  Populate user_id with value from user.id
>>   4a)  remove old field (user.id)
>> 3) Logstash pushes new index data to new ES cluster
>>
>> I'm sure I've left out something crucial here. Seems to be par for the 
>> course, but I'm hopeful. :)
>>
>>
>>
>>
>> On Wednesday, June 1, 2016 at 3:06:34 PM UTC-4, Jimmy Chen wrote:
>>>
>>> Did this work for you? I am going to be looking into upgrading our 
>>> existing cluster to 2.x too.
>>>
>>> On Tuesday, May 31, 2016 at 5:08:05 PM UTC-7, Robert Hough wrote:

 Came across this:  
 https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06 
 

 third time's the charm?  :)


 On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:
>
> Recently built a Graylog 2.x cluster, and that seems to be working 
> fine.  I had some questions though, but right now the biggest nagging 
> question has been...
>
> How do we migrate our existing indexes over to the new system?  The 
> whole dots in field names issue seems to be what is preventing us from 
> pulling this off.  How do we correct these, and then import them into the 
> our new system? 
>


-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8b100acf-e9b1-4cbc-b00b-571cac245da2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.x upgrade

2016-06-02 Thread Jimmy Chen 
Good luck with the fires and I'll check back to see how it went.

On Thursday, June 2, 2016 at 6:03:34 AM UTC-7, Robert Hough wrote:
>
> Well, "out of the box", no that didn't work.  I've got faith that it can 
> be done using this approach, but we'll also need to utilize Elastic's 
> "de_dot" filter plugin.  I'm hoping to make some progress with that today, 
> and I'll provide an update by the end of the day. I've got about 10 fires 
> to put out first... :(
>
> Here's the link to the de_dot documentation:
>
> https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html
>
>
> In a nutshell:
>
> 1) Logstash pulls in old index data from old ES cluster
> 2) Logstash sends that through filter
>   1a)  Match any dots in fields  (user.id)
>   2a)  Add new field as replacement for old field (user.id == user_id)
>   3a)  Populate user_id with value from user.id
>   4a)  remove old field (user.id)
> 3) Logstash pushes new index data to new ES cluster
>
> I'm sure I've left out something crucial here. Seems to be par for the 
> course, but I'm hopeful. :)
>
>
>
>
> On Wednesday, June 1, 2016 at 3:06:34 PM UTC-4, Jimmy Chen wrote:
>>
>> Did this work for you? I am going to be looking into upgrading our 
>> existing cluster to 2.x too.
>>
>> On Tuesday, May 31, 2016 at 5:08:05 PM UTC-7, Robert Hough wrote:
>>>
>>> Came across this:  
>>> https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06 
>>> 
>>>
>>> third time's the charm?  :)
>>>
>>>
>>> On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:

 Recently built a Graylog 2.x cluster, and that seems to be working 
 fine.  I had some questions though, but right now the biggest nagging 
 question has been...

 How do we migrate our existing indexes over to the new system?  The 
 whole dots in field names issue seems to be what is preventing us from 
 pulling this off.  How do we correct these, and then import them into the 
 our new system? 

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b679c252-8fb9-489e-ab0a-f4f51bb302fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.x upgrade

2016-06-02 Thread Robert Hough 
Well, "out of the box", no that didn't work.  I've got faith that it can be 
done using this approach, but we'll also need to utilize Elastic's "de_dot" 
filter plugin.  I'm hoping to make some progress with that today, and I'll 
provide an update by the end of the day. I've got about 10 fires to put out 
first... :(

Here's the link to the de_dot documentation:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html


In a nutshell:

1) Logstash pulls in old index data from old ES cluster
2) Logstash sends that through filter
  1a)  Match any dots in fields  (user.id)
  2a)  Add new field as replacement for old field (user.id == user_id)
  3a)  Populate user_id with value from user.id
  4a)  remove old field (user.id)
3) Logstash pushes new index data to new ES cluster

I'm sure I've left out something crucial here. Seems to be par for the 
course, but I'm hopeful. :)




On Wednesday, June 1, 2016 at 3:06:34 PM UTC-4, Jimmy Chen wrote:
>
> Did this work for you? I am going to be looking into upgrading our 
> existing cluster to 2.x too.
>
> On Tuesday, May 31, 2016 at 5:08:05 PM UTC-7, Robert Hough wrote:
>>
>> Came across this:  
>> https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06 
>> 
>>
>> third time's the charm?  :)
>>
>>
>> On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:
>>>
>>> Recently built a Graylog 2.x cluster, and that seems to be working 
>>> fine.  I had some questions though, but right now the biggest nagging 
>>> question has been...
>>>
>>> How do we migrate our existing indexes over to the new system?  The 
>>> whole dots in field names issue seems to be what is preventing us from 
>>> pulling this off.  How do we correct these, and then import them into the 
>>> our new system? 
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7d21cb0c-078e-4385-9058-e5124ec64b95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.x upgrade

2016-06-01 Thread Jimmy Chen 
Did this work for you? I am going to be looking into upgrading our existing 
cluster to 2.x too.

On Tuesday, May 31, 2016 at 5:08:05 PM UTC-7, Robert Hough wrote:
>
> Came across this:  https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06
>
> third time's the charm?  :)
>
>
> On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:
>>
>> Recently built a Graylog 2.x cluster, and that seems to be working fine.  
>> I had some questions though, but right now the biggest nagging question has 
>> been...
>>
>> How do we migrate our existing indexes over to the new system?  The whole 
>> dots in field names issue seems to be what is preventing us from pulling 
>> this off.  How do we correct these, and then import them into the our new 
>> system? 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/213fe3a2-fbbd-430d-a685-6de286a56335%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.x upgrade

2016-05-31 Thread Robert Hough 
Came across this:  https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

third time's the charm?  :)


On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:
>
> Recently built a Graylog 2.x cluster, and that seems to be working fine.  
> I had some questions though, but right now the biggest nagging question has 
> been...
>
> How do we migrate our existing indexes over to the new system?  The whole 
> dots in field names issue seems to be what is preventing us from pulling 
> this off.  How do we correct these, and then import them into the our new 
> system? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/20da9ee9-4ec0-40a8-b2de-aed4ce6a520b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog 2.x upgrade

2016-05-31 Thread Robert Hough 
21 views and no replies.  Either this is the best kept secret in town, or 
everyone else is just as baffled as I am?  :)

So far I've tried using ElasticDump, The ElasticSearch Exporter and even 
tried using straight curl.   I'm admittedly ignorant to most of these tools 
in terms of setup.  I kind of inherited this and now feel like I have to 
beat it into submission. Perhaps I'm just going about it wrong?  Here was 
my initial approach:

1) Create a new index on the new ES cluster with the same name as the one 
on the old cluster
2) Export the index mapping,  update the fields (user.name, user.id, 
session.id, etc) and replace the dots with underscores.
3) Import the updated mapping into the new index
3) Export the index data
4) import said data into new index

I can get to step 3.  Step 4 fails, and I'm not getting any real 
explanation as to why.  So if you guys have can set my on the right track, 
that would be handy.



On Friday, May 27, 2016 at 4:43:18 PM UTC-4, Robert Hough wrote:
>
> Recently built a Graylog 2.x cluster, and that seems to be working fine.  
> I had some questions though, but right now the biggest nagging question has 
> been...
>
> How do we migrate our existing indexes over to the new system?  The whole 
> dots in field names issue seems to be what is preventing us from pulling 
> this off.  How do we correct these, and then import them into the our new 
> system? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/8a41ef62-8ed4-4b3a-87bd-de02517bdf2d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.