[input_log] type = "LogstreamerInput" log_directory = "/tmp" file_match = "hello.log"
[output_s3] type = "S3Output" message_matcher = "TRUE" secret_key = "4fvHMt5lekgPWzxLwUgwb+kcFu7lVi695NxUJ0a7" access_key = "0BRRE6DFWV2WSVJD3882" bucket_name = "wego-logs-staging" path_name = "/hello" region = "ap-southeast-1" ticker_interval = 10 max_buffer_size = 10 encoder = "PayloadEncoder" [PayloadEncoder] append_newlines = false It is just a simple pipeline (from local filesystem to S3 bucket) for testing purposes. However in production, we will configure the plugin to have kafka as the input and S3 bucket as the output, as required by my company. On Thu, May 14, 2015 at 12:20 AM, Jarek Śmiejczak <j...@smiejczak.net> wrote: > Could you post your config file for heka .toml? What is the value for > ticker_interval? > > On Wed, May 13, 2015 at 12:42 PM, Alex Jiao <uohzx...@gmail.com> wrote: > >> Hi, I'm writing an S3 output plugin for heka. As I am new to golang and >> heka, I'm not too sure what's the best way to go about writing the plugin. >> Hence I wish to seek you guys' help to resolve some issues that are >> plaguing my code. >> >> heka-s3 plugin code: >> https://github.com/uohzxela/heka-s3/blob/master/s3.go >> >> My plugin needs to send message packs from the pipeline at regular >> intervals as specified in the .toml file. It writes to a buffer during each >> interval and upload to a S3 bucket at the next tick. The interval is >> specified by the 'ticker_interval' option. However, I'm not too sure >> whether you need to take care of the ticker logic in the for loop or that >> the output Run function will be invoked at the specified ticker interval by >> Heka service itself. >> >> I have implemented the first case as seen in my code and it led to a >> weird issue. The plugin can be loaded upon starting the Heka service, >> however the ticker can only run once after the elapsed time interval. It's >> only after I sent a TERM signal to the Heka process using kill (kill >> <heka-pid>), that the ticker starts to work. >> >> Could you guys shed light on resolving this issue? Why does the ticker >> not work upon starting the plugin? Does the programmer have to take care of >> the ticker logic or the output runner will automatically be invoked at >> regular intervals by the Heka service? I'd appreciate it as well if you can >> point me to some output plugins that have the ticker functionality. >> >> Thank you. >> >> -- >> Alex >> >> _______________________________________________ >> Heka mailing list >> Heka@mozilla.org >> https://mail.mozilla.org/listinfo/heka >> >> > > > -- > jarek@reijutsu:~$ fortune > You have Egyptian flu: you're going to be a mummy. > jarek@reijutsu:~$ fortune > You now have Asian Flu. > -- Alex
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
