[heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Alex Jiao
Hi, I'm writing an S3 output plugin for heka. As I am new to golang and heka, I'm not too sure what's the best way to go about writing the plugin. Hence I wish to seek you guys' help to resolve some issues that are plaguing my code. heka-s3 plugin code: https://github.com/uohzxela/heka-s3/blob/mas

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Michael Trinkala
You may want to take a look at https://github.com/mozilla-services/data-pipeline/blob/master/heka/plugins/s3splitfile/s3splitfile_output.go Trink On Wed, May 13, 2015 at 3:42 AM, Alex Jiao wrote: > Hi, I'm writing an S3 output plugin for heka. As I am new to golang and > heka, I'm not too sure

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Jarek Śmiejczak
Could you post your config file for heka .toml? What is the value for ticker_interval? On Wed, May 13, 2015 at 12:42 PM, Alex Jiao wrote: > Hi, I'm writing an S3 output plugin for heka. As I am new to golang and > heka, I'm not too sure what's the best way to go about writing the plugin. > Hence

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Alex Jiao
[input_log] type = "LogstreamerInput" log_directory = "/tmp" file_match = "hello.log" [output_s3] type = "S3Output" message_matcher = "TRUE" secret_key = "4fvHMt5lekgPWzxLwUgwb+kcFu7lVi695NxUJ0a7" access_key = "0BRRE6DFWV2WSVJD3882" bucket_name = "wego-logs-staging" path_name = "/hello" region = "

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Alex Jiao
Hey sorry, could you guys delete/edit my previous message? I have realised I accidentally exposed my access key and secret key. On Thu, May 14, 2015 at 2:17 AM, Alex Jiao wrote: > [input_log] > type = "LogstreamerInput" > log_directory = "/tmp" > file_match = "hello.log" > > [output_s3] > type =

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Curt Micol
Hey Alex, You should regenerate your keys, even if the message is deleted people will still have the cache of the email. I'd do this immediately btw. On Wed, May 13, 2015 at 6:21 PM, Alex Jiao wrote: > Hey sorry, could you guys delete/edit my previous message? I have realised I > accidentally ex

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Rob Miller
Your code is put together very strangely. The ticker channel is firing correctly, but once it fires the first time you immediately get trapped in the for loop that is over the range of the inChan. So you aren't getting any of the input messages until the first tick fires, but after it fires you