[heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Alex Jiao
Hi, I'm writing an S3 output plugin for heka. As I am new to golang and heka, I'm not too sure what's the best way to go about writing the plugin. Hence I wish to seek you guys' help to resolve some issues that are plaguing my code. heka-s3 plugin code: https://github.com/uohzxela/heka-s3/blob/mas

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Alex Jiao
cket) for testing purposes. However in production, we will configure the plugin to have kafka as the input and S3 bucket as the output, as required by my company. On Thu, May 14, 2015 at 12:20 AM, Jarek ƚmiejczak wrote: > Could you post your config file for heka .toml? What is the value for &g

Re: [heka] Need help writing heka-s3 output plugin

2015-05-13 Thread Alex Jiao
Hey sorry, could you guys delete/edit my previous message? I have realised I accidentally exposed my access key and secret key. On Thu, May 14, 2015 at 2:17 AM, Alex Jiao wrote: > [input_log] > type = "LogstreamerInput" > log_directory = "/tmp" > file_match = &q

Re: [heka] Need help writing heka-s3 output plugin

2015-05-18 Thread Alex Jiao
strates this pattern is the CounterFilter: > > > https://github.com/mozilla-services/heka/blob/dev/pipeline/counter_filter.go > > Hope this helps, > > -r > > > > On 05/13/2015 03:42 AM, Alex Jiao wrote: > >> Hi, I'm writing an S3 output plugin for heka. As I am

Re: [heka] Need help writing heka-s3 output plugin

2015-05-18 Thread Alex Jiao
BTW, as I am still new to golang and writing plugins, any feedback will be very useful! On Mon, May 18, 2015 at 9:15 PM, Alex Jiao wrote: > Hey Rob, thanks a lot for your input. It clears up a lot of doubts > regarding my code. I have rewritten it according to your suggestions and it &