Hello community, here is the log from the commit of package rubygem-fluentd for openSUSE:Factory checked in at 2019-04-01 12:37:12 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/rubygem-fluentd (Old) and /work/SRC/openSUSE:Factory/.rubygem-fluentd.new.25356 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "rubygem-fluentd" Mon Apr 1 12:37:12 2019 rev:10 rq:689713 version:1.4.1 Changes: -------- --- /work/SRC/openSUSE:Factory/rubygem-fluentd/rubygem-fluentd.changes 2019-01-21 10:54:07.575745147 +0100 +++ /work/SRC/openSUSE:Factory/.rubygem-fluentd.new.25356/rubygem-fluentd.changes 2019-04-01 12:37:12.805881247 +0200 @@ -1,0 +2,12 @@ +Fri Mar 29 06:00:25 UTC 2019 - Stephan Kulow <co...@suse.com> + +- updated to version 1.4.1 + see installed CHANGELOG.md + +------------------------------------------------------------------- +Sat Mar 2 15:14:08 UTC 2019 - Stephan Kulow <co...@suse.com> + +- updated to version 1.4.0 + see installed CHANGELOG.md + +------------------------------------------------------------------- Old: ---- fluentd-1.3.3.gem New: ---- fluentd-1.4.1.gem ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ rubygem-fluentd.spec ++++++ --- /var/tmp/diff_new_pack.eToZ38/_old 2019-04-01 12:37:13.357881518 +0200 +++ /var/tmp/diff_new_pack.eToZ38/_new 2019-04-01 12:37:13.361881520 +0200 @@ -24,7 +24,7 @@ # Name: rubygem-fluentd -Version: 1.3.3 +Version: 1.4.1 Release: 0 %define mod_name fluentd %define mod_full_name %{mod_name}-%{version} ++++++ fluentd-1.3.3.gem -> fluentd-1.4.1.gem ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.github/ISSUE_TEMPLATE.md new/.github/ISSUE_TEMPLATE.md --- old/.github/ISSUE_TEMPLATE.md 2019-01-07 04:07:42.000000000 +0100 +++ new/.github/ISSUE_TEMPLATE.md 2019-03-19 14:48:00.000000000 +0100 @@ -1,6 +1,8 @@ Check [CONTRIBUTING guideline](https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md) first and here is the list to help us investigate the problem. - fluentd or td-agent version. -- Environment information, e.g. OS. +- Environment information: + - Operating system: `cat /etc/os-release` + - Kernel version: `uname -r` - Your configuration - Your problem explanation. If you have an error logs, write it together. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.github/PULL_REQUEST_TEMPLATE.md new/.github/PULL_REQUEST_TEMPLATE.md --- old/.github/PULL_REQUEST_TEMPLATE.md 1970-01-01 01:00:00.000000000 +0100 +++ new/.github/PULL_REQUEST_TEMPLATE.md 2019-03-19 14:48:00.000000000 +0100 @@ -0,0 +1,13 @@ +<!-- +Thank you for contributing to Fluentd! +Please provide the following information to help us make the most of your pull request: +--> + +**Which issue(s) this PR fixes**: +Fixes # + +**What this PR does / why we need it**: + +**Docs Changes**: + +**Release Note**: diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.gitignore new/.gitignore --- old/.gitignore 2019-01-07 04:07:42.000000000 +0100 +++ new/.gitignore 2019-03-19 14:48:00.000000000 +0100 @@ -25,4 +25,5 @@ coverage/* .vagrant/ cov-int/ -cov-fluentd.tar.gz \ No newline at end of file +cov-fluentd.tar.gz +.vscode \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/.travis.yml new/.travis.yml --- old/.travis.yml 2019-01-07 04:07:42.000000000 +0100 +++ new/.travis.yml 2019-03-19 14:48:00.000000000 +0100 @@ -11,8 +11,6 @@ os: linux - rvm: 2.2.10 os: linux - - rvm: 2.3.7 - os: linux - rvm: 2.4.5 os: linux - rvm: 2.5.3 @@ -37,6 +35,8 @@ os: osx osx_image: xcode8.3 # OSX 10.12 allow_failures: + - rvm: 2.3.8 + os: linux - rvm: 2.1.10 os: osx osx_image: xcode8.3 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/CHANGELOG.md new/CHANGELOG.md --- old/CHANGELOG.md 2019-01-07 04:07:42.000000000 +0100 +++ new/CHANGELOG.md 2019-03-19 14:48:00.000000000 +0100 @@ -1,6 +1,55 @@ +# v1.4 + +## Release v1.4.1 - 2019/03/18 + +### Enhancements + +* system: Add worker_id to process_name when workers is larger than 1 + https://github.com/fluent/fluentd/pull/2321 +* in_syslog: Check message length when read from buffer in octet counting + https://github.com/fluent/fluentd/pull/2323 +* parser_regexp: Check named captures. When no named captures, configuration error is raised + https://github.com/fluent/fluentd/pull/2331 + +### Bug fixes + +* out_forward: Make tls_client_private_key_passphrase secret + https://github.com/fluent/fluentd/pull/2324 + +## Release v1.4.0 - 2019/02/24 + +### New features + +* multiprocess: Support <worker N-M> syntax + https://github.com/fluent/fluentd/pull/2292 +* output: Work <secondary> and retry_forever together + https://github.com/fluent/fluentd/pull/2276 +* out_file: Support placeholders in symlink_path + https://github.com/fluent/fluentd/pull/2254 + +### Enhancements + +* output: Add MessagePack unpacker error to unrecoverable error list + https://github.com/fluent/fluentd/pull/2301 +* output: Reduce flush delay when large timekey and small timekey_wait are specified + https://github.com/fluent/fluentd/pull/2291 +* config: Support embedded ruby code in section argument. + https://github.com/fluent/fluentd/pull/2295 +* in_tail: Improve encoding parameter handling + https://github.com/fluent/fluentd/pull/2305 +* in_tcp/in_udp: Add <parse> section check + https://github.com/fluent/fluentd/pull/2267 + +### Bug fixes + +* server: Ignore IOError and related errors in UDP + https://github.com/fluent/fluentd/pull/2310 +* server: Ignore EPIPE in TLS accept to avoid fluentd restart + https://github.com/fluent/fluentd/pull/2253 + # v1.3 -## Release v1.3.3 - 2018/01/06 +## Release v1.3.3 - 2019/01/06 ### Enhancements @@ -642,12 +691,12 @@ * output: Secondary calculation should consider 'retry_max_times' https://github.com/fluent/fluentd/pull/1452 -* Fix regression of deprecatd 'process' module +* Fix regression of deprecated 'process' module https://github.com/fluent/fluentd/pull/1443 * Fix missing parser_regex require https://github.com/fluent/fluentd/issues/1458 https://github.com/fluent/fluentd/pull/1453 -* Keep 'Fluent::BufferQueueLimitError' for exsting plugins +* Keep 'Fluent::BufferQueueLimitError' for existing plugins https://github.com/fluent/fluentd/pull/1456 * in_tail: Untracked files should be removed from watching list to avoid memory bloat https://github.com/fluent/fluentd/pull/1467 @@ -899,7 +948,7 @@ ### Bug fixes -* Raise configuration errors to clearify what's wrong when "@type" is missing +* Raise configuration errors to clarify what's wrong when "@type" is missing https://github.com/fluent/fluentd/pull/1202 * Fix the bug not to launch Fluentd when v0.12 MultiOutput plugin is configured https://github.com/fluent/fluentd/pull/1206 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/CONTRIBUTING.md new/CONTRIBUTING.md --- old/CONTRIBUTING.md 2019-01-07 04:07:42.000000000 +0100 +++ new/CONTRIBUTING.md 2019-03-19 14:48:00.000000000 +0100 @@ -27,7 +27,7 @@ * **Documentation**: Use [fluentd-docs](https://github.com/fluent/fluentd-docs) repository. If you find a bug of 3rd party plugins, please submit an issue to each plugin repository. -And use [omnibus-td-agent](https://github.com/treasure-data/omnibus-td-agent) repository for td-agent releated issues. +And use [omnibus-td-agent](https://github.com/treasure-data/omnibus-td-agent) repository for td-agent related issues. Note: Before report the issue, check latest version first. Sometimes users report fixed bug with older version. @@ -37,7 +37,7 @@ * Write tests. * Run tests before send Pull Request by `bundle exec rake test` -* Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). +* Write a [good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). * Fluentd repositories needs [DCO](https://github.com/apps/dco) on PR. Please add `Signed-off-by` to the commit(See DCO link for more detail). There are some patches which are hard to write tests, e.g. process handling, concurrency issue or etc. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/GithubWorkflow.md new/GithubWorkflow.md --- old/GithubWorkflow.md 1970-01-01 01:00:00.000000000 +0100 +++ new/GithubWorkflow.md 2019-03-19 14:48:00.000000000 +0100 @@ -0,0 +1,78 @@ +# Github workflow for contributing to fluentd + +Table of Contents + +* [Fork a repository](#fork-a-repository) +* [Clone fork repository to local](#clone-fork-repository-to-local) +* [Create a branch to add a new feature or fix issues](#create-a-branch-to-add-a-new-feature-or-fix-issues) +* [Commit and Push](#commit-and-push) +* [Create a Pull Request](#create-a-pull-request) + + +The [fluentd](https://github.com/fluent/fluentd.git) code is hosted on Github (https://github.com/fluent/fluentd). The repository is called `upstream`. Contributors will develop and commit their changes in a clone of upstream repository. Then contributors push their change to their forked repository (`origin`) and create a Pull Request (PR), the PR will be merged to `upstream` repository if it meets the all the necessary requirements. + +## Fork a repository + + Go to https://github.com/fluent/fluentd then hit the `Fork` button to fork your own copy of repository **fluentd** to your github account. + +## Clone the forked repository to local + +Clone the forked repo in [above step](#fork-a-repository) to your local working directory: +```sh +$ git clone https://github.com/$your_github_account/fluentd.git +``` + +Keep your fork in sync with the main repo, add an `upstream` remote: +```sh +$ cd fluentd +$ git remote add upstream https://github.com/fluentd/fluentd.git +$ git remote -v + +origin https://github.com/$your_github_account/fluentd.git (fetch) +origin https://github.com/$your_github_account/fluentd.git (push) +upstream https://github.com/fluentd/fluentd.git (fetch) +upstream https://github.com/fluentd/fluentd.git (push) +``` + +Sync your local `master` branch: +```sh +$ git checkout master +$ git pull origin master +$ git fetch upstream +$ git rebase upstream/master +``` + +## Create a branch to add a new feature or fix issues + +Before making any change, create a new branch: +```sh +$ git checkout master +$ git pull upstream master +$ git checkout -b new-feature +``` + +## Commit and Push + +Make any change on the branch `new-feature` then build and test your codes. +Include in what will be committed: +```sh +$ git add <file> +``` + +Commit your changes with `sign-offs` +```sh +$ git commit -s +``` + +Enter your commit message to describe the changes. See the tips for a good commit message at [here](https://chris.beams.io/posts/git-commit/). +Likely you go back and edit/build/test some more then `git commit --amend` + +Push your branch `new-feature` to your forked repository: +```sh +$ git push -u origin new-feature +``` + +## Create a Pull Request + +* Go to your fork at https://github.com/$your_github_account/fluentd +* Create a Pull Request from the branch you recently pushed by hitting the button `Compare & pull request` next to branch name. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/MAINTAINERS.md new/MAINTAINERS.md --- old/MAINTAINERS.md 2019-01-07 04:07:42.000000000 +0100 +++ new/MAINTAINERS.md 2019-03-19 14:48:00.000000000 +0100 @@ -1,6 +1,6 @@ # Fluentd Maintainers -- [Naotoshi Seo](https://github.com/sonots), [DeNA](http://dena.com/intl/) +- [Naotoshi Seo](https://github.com/sonots), [ZOZO Technologies](https://tech.zozo.com/en/) - [Okkez](https://github.com/okkez), [Clearcode](https://www.clear-code.com/) - [Hiroshi Hatake](https://github.com/cosmo0920), [Clearcode](https://www.clear-code.com/) - [Masahiro Nakagawa](https://github.com/repeatedly), [Treasure Data](https://www.treasuredata.com/) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/README.md new/README.md --- old/README.md 2019-01-07 04:07:42.000000000 +0100 +++ new/README.md 2019-03-19 14:48:00.000000000 +0100 @@ -71,8 +71,8 @@ - Project repository: https://github.com/fluent - Discussion: https://groups.google.com/group/fluentd - Slack / Community: https://slack.fluentd.org -- Newsletters: https://www.fluentd.org/newsletter_signup -- Author: Sadayuki Furuhashi +- Newsletters: https://www.fluentd.org/newsletter +- Author: [Sadayuki Furuhashi](https://github.com/frsyuki) - Copyright: 2011-2018 Fluentd Authors - License: Apache License, Version 2.0 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/appveyor.yml new/appveyor.yml --- old/appveyor.yml 2019-01-07 04:07:42.000000000 +0100 +++ new/appveyor.yml 2019-03-19 14:48:00.000000000 +0100 @@ -30,14 +30,9 @@ devkit: C:\Ruby23\DevKit - ruby_version: "22-x64" devkit: C:\Ruby23-x64\DevKit - - ruby_version: "21-x64" - devkit: C:\Ruby23-x64\DevKit - ruby_version: "22" devkit: C:\Ruby23\DevKit WIN_RAPID: true - - ruby_version: "21" - devkit: C:\Ruby23\DevKit - WIN_RAPID: true matrix: allow_failures: - ruby_version: "21" Binary files old/checksums.yaml.gz and new/checksums.yaml.gz differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/command/plugin_config_formatter.rb new/lib/fluent/command/plugin_config_formatter.rb --- old/lib/fluent/command/plugin_config_formatter.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/command/plugin_config_formatter.rb 2019-03-19 14:48:00.000000000 +0100 @@ -32,7 +32,8 @@ "buffer", "parser", "formatter", "storage" ] - DOCS_BASE_URL = "https://docs.fluentd.org/v1.0/articles/" + DOCS_BASE_URL = "https://docs.fluentd.org/v1.0/articles/quickstart" + DOCS_ARTICLE_BASE_URL = "https://docs.fluentd.org/v1.0/articles/" def initialize(argv = ARGV) @argv = argv @@ -185,7 +186,7 @@ end def plugin_helper_url(plugin_helper) - "#{DOCS_BASE_URL}api-plugin-helper-#{plugin_helper}" + "#{DOCS_ARTICLE_BASE_URL}api-plugin-helper-#{plugin_helper}" end def plugin_helper_markdown_link(plugin_helper) @@ -194,7 +195,7 @@ def plugin_overview_url(class_name) plugin_type = class_name.slice(/::(\w+)\z/, 1).downcase - "#{DOCS_BASE_URL}#{plugin_type}-plugin-overview" + "#{DOCS_ARTICLE_BASE_URL}#{plugin_type}-plugin-overview" end def plugin_overview_markdown_link(class_name) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/config/element.rb new/lib/fluent/config/element.rb --- old/lib/fluent/config/element.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/config/element.rb 2019-03-19 14:48:00.000000000 +0100 @@ -36,12 +36,12 @@ # it's global logger, not plugin logger: deprecated message should be global warning, not plugin level. @logger = defined?($log) ? $log : nil - @target_worker_id = nil + @target_worker_ids = [] end attr_accessor :name, :arg, :unused, :v1_config, :corresponding_proxies, :unused_in attr_writer :elements - attr_reader :target_worker_id + attr_reader :target_worker_ids RESERVED_PARAMETERS_COMPAT = { '@type' => 'type', @@ -223,22 +223,29 @@ end def set_target_worker_id(worker_id) - @target_worker_id = worker_id + @target_worker_ids = [worker_id] @elements.each { |e| e.set_target_worker_id(worker_id) } end + def set_target_worker_ids(worker_ids) + @target_worker_ids = worker_ids.uniq + @elements.each { |e| + e.set_target_worker_ids(worker_ids.uniq) + } + end + def for_every_workers? - @target_worker_id == nil + @target_worker_ids.empty? end def for_this_worker? - @target_worker_id == Fluent::Engine.worker_id + @target_worker_ids.include?(Fluent::Engine.worker_id) end def for_another_worker? - @target_worker_id != nil && @target_worker_id != Fluent::Engine.worker_id + !@target_worker_ids.empty? && !@target_worker_ids.include?(Fluent::Engine.worker_id) end end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/config/v1_parser.rb new/lib/fluent/config/v1_parser.rb --- old/lib/fluent/config/v1_parser.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/config/v1_parser.rb 2019-03-19 14:48:00.000000000 +0100 @@ -82,7 +82,7 @@ elsif skip(/\</) e_name = scan(ELEMENT_NAME) spacing - e_arg = scan_nonquoted_string(/(?:#{ZERO_OR_MORE_SPACING}\>)/) + e_arg = scan_string(/(?:#{ZERO_OR_MORE_SPACING}\>)/) spacing unless skip(/\>/) parse_error! "expected '>'" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/counter/client.rb new/lib/fluent/counter/client.rb --- old/lib/fluent/counter/client.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/counter/client.rb 2019-03-19 14:48:00.000000000 +0100 @@ -288,7 +288,7 @@ def join until @set @mutex.synchronize do - @loop.run_once(0.0001) # retun a lock as soon as possible + @loop.run_once(0.0001) # return a lock as soon as possible end end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/base.rb new/lib/fluent/plugin/base.rb --- old/lib/fluent/plugin/base.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/base.rb 2019-03-19 14:48:00.000000000 +0100 @@ -52,7 +52,12 @@ def configure(conf) if conf.respond_to?(:for_this_worker?) && conf.for_this_worker? - system_config_override(workers: 1) + workers = if conf.target_worker_ids && !conf.target_worker_ids.empty? + conf.target_worker_ids.size + else + 1 + end + system_config_override(workers: workers) end super @_state ||= State.new(false, false, false, false, false, false, false, false, false) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/compressable.rb new/lib/fluent/plugin/compressable.rb --- old/lib/fluent/plugin/compressable.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/compressable.rb 2019-03-19 14:48:00.000000000 +0100 @@ -44,7 +44,7 @@ # check compressed_data(String) is 0 length compressed_data when output_io - # exeucte after checking compressed_data is empty or not + # execute after checking compressed_data is empty or not io = StringIO.new(compressed_data) io_decompress(io, output_io) else diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_http.rb new/lib/fluent/plugin/in_http.rb --- old/lib/fluent/plugin/in_http.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/in_http.rb 2019-03-19 14:48:00.000000000 +0100 @@ -163,6 +163,7 @@ # Skip nil record if record.nil? + log.debug { "incoming event is invalid: path=#{path_info} params=#{params.to_json}" } if @respond_with_empty_img return ["200 OK", {'Content-Type'=>'image/gif; charset=utf-8'}, EMPTY_GIF_IMAGE] else diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_syslog.rb new/lib/fluent/plugin/in_syslog.rb --- old/lib/fluent/plugin/in_syslog.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/in_syslog.rb 2019-03-19 14:48:00.000000000 +0100 @@ -169,6 +169,10 @@ num = Integer(buffer[pos..idx]) pos = idx + num msg = buffer[idx + 1...pos] + if msg.size < num - 1 + pos = pos - num - num.to_s.size + break + end message_handler(msg, conn) end else diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tail.rb new/lib/fluent/plugin/in_tail.rb --- old/lib/fluent/plugin/in_tail.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/in_tail.rb 2019-03-19 14:48:00.000000000 +0100 @@ -171,6 +171,9 @@ @encoding = parse_encoding_param(@encoding) if @encoding @from_encoding = parse_encoding_param(@from_encoding) if @from_encoding + if @encoding == @from_encoding + log.warn "'encoding' and 'from_encoding' are same encoding. No effect" + end end def parse_encoding_param(encoding_name) @@ -657,6 +660,7 @@ def initialize(from_encoding, encoding) @from_encoding = from_encoding @encoding = encoding + @need_enc = from_encoding != encoding @buffer = ''.force_encoding(from_encoding) @eol = "\n".encode(from_encoding).freeze end @@ -682,11 +686,13 @@ end def convert(s) - if @from_encoding == @encoding - s + if @need_enc + s.encode!(@encoding, @from_encoding) else - s.encode(@encoding, @from_encoding) + s end + rescue + s.encode!(@encoding, @from_encoding, :invalid => :replace, :undef => :replace) end def next_line diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tcp.rb new/lib/fluent/plugin/in_tcp.rb --- old/lib/fluent/plugin/in_tcp.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/in_tcp.rb 2019-03-19 14:48:00.000000000 +0100 @@ -41,11 +41,15 @@ def configure(conf) compat_parameters_convert(conf, :parser) + parser_config = conf.elements('parse').first + unless parser_config + raise Fluent::ConfigError, "<parse> section is required." + end super @_event_loop_blocking_timeout = @blocking_timeout @source_hostname_key ||= @source_host_key if @source_host_key - @parser = parser_create + @parser = parser_create(conf: parser_config) end def multi_workers_ready? diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_udp.rb new/lib/fluent/plugin/in_udp.rb --- old/lib/fluent/plugin/in_udp.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/in_udp.rb 2019-03-19 14:48:00.000000000 +0100 @@ -47,12 +47,16 @@ def configure(conf) compat_parameters_convert(conf, :parser) + parser_config = conf.elements('parse').first + unless parser_config + raise Fluent::ConfigError, "<parse> section is required." + end super @_event_loop_blocking_timeout = @blocking_timeout @source_hostname_key ||= @source_host_key if @source_host_key @message_length_limit = @body_size_limit if @body_size_limit - @parser = parser_create + @parser = parser_create(conf: parser_config) end def multi_workers_ready? diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/out_file.rb new/lib/fluent/plugin/out_file.rb --- old/lib/fluent/plugin/out_file.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/out_file.rb 2019-03-19 14:48:00.000000000 +0100 @@ -71,6 +71,10 @@ attr_accessor :last_written_path # for tests module SymlinkBufferMixin + def output_plugin_for_symlink=(output_plugin) + @_output_plugin_for_symlink = output_plugin + end + def symlink_path=(path) @_symlink_path = path end @@ -83,7 +87,7 @@ # These chunks will be enqueued immediately, and will be flushed soon. latest_metadata = metadata_list.select{|m| m.timekey }.sort_by(&:timekey).last if chunk.metadata == latest_metadata - FileUtils.ln_sf(chunk.path, @_symlink_path) + FileUtils.ln_sf(chunk.path, @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk)) end chunk end @@ -161,6 +165,7 @@ else @buffer.extend SymlinkBufferMixin @buffer.symlink_path = @symlink_path + @buffer.output_plugin_for_symlink = self end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/out_forward.rb new/lib/fluent/plugin/out_forward.rb --- old/lib/fluent/plugin/out_forward.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/out_forward.rb 2019-03-19 14:48:00.000000000 +0100 @@ -102,7 +102,7 @@ desc 'The client private key path for TLS.' config_param :tls_client_private_key_path, :string, default: nil desc 'The client private key passphrase for TLS.' - config_param :tls_client_private_key_passphrase, :string, default: nil + config_param :tls_client_private_key_passphrase, :string, default: nil, secret: true config_section :security, required: false, multi: false do desc 'The hostname' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/output.rb new/lib/fluent/plugin/output.rb --- old/lib/fluent/plugin/output.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/output.rb 2019-03-19 14:48:00.000000000 +0100 @@ -364,7 +364,9 @@ raise Fluent::ConfigError, "Invalid <secondary> section for non-buffered plugin" unless @buffering raise Fluent::ConfigError, "<secondary> section cannot have <buffer> section" if @secondary_config.buffer raise Fluent::ConfigError, "<secondary> section cannot have <secondary> section" if @secondary_config.secondary - raise Fluent::ConfigError, "<secondary> section and 'retry_forever' are exclusive" if @buffer_config.retry_forever + if @buffer_config.retry_forever + log.warn "<secondary> with 'retry_forever', only unrecoverable errors are moved to secondary" + end secondary_type = @secondary_config[:@type] unless secondary_type @@ -1081,7 +1083,7 @@ end end - UNRECOVERABLE_ERRORS = [Fluent::UnrecoverableError, TypeError, ArgumentError, NoMethodError] + UNRECOVERABLE_ERRORS = [Fluent::UnrecoverableError, TypeError, ArgumentError, NoMethodError, MessagePack::UnpackError] def try_flush chunk = @buffer.dequeue_chunk @@ -1334,7 +1336,7 @@ end if @chunk_key_time if !value_for_interval || @buffer_config.timekey < value_for_interval - value_for_interval = @buffer_config.timekey + value_for_interval = [@buffer_config.timekey, @buffer_config.timekey_wait].min end end unless value_for_interval diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/parser_regexp.rb new/lib/fluent/plugin/parser_regexp.rb --- old/lib/fluent/plugin/parser_regexp.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin/parser_regexp.rb 2019-03-19 14:48:00.000000000 +0100 @@ -40,6 +40,10 @@ @expression = Regexp.compile(@expression.source, options) end @regexp = @expression # For backward compatibility + + if @expression.named_captures.empty? + raise Fluent::ConfigError, "No named captures in 'expression' parameter. The regexp must have at least one named capture" + end end def parse(text) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin_helper/retry_state.rb new/lib/fluent/plugin_helper/retry_state.rb --- old/lib/fluent/plugin_helper/retry_state.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin_helper/retry_state.rb 2019-03-19 14:48:00.000000000 +0100 @@ -53,10 +53,6 @@ @randomize = randomize @randomize_width = randomize_width - if forever && secondary - raise "BUG: forever and secondary are exclusive to each other" - end - @forever = forever @max_steps = max_steps @@ -118,12 +114,12 @@ end def secondary? - @secondary && (@current == :secondary || current_time >= @secondary_transition_at) + !@forever && @secondary && (@current == :secondary || current_time >= @secondary_transition_at) end def step @steps += 1 - if @secondary && @current != :secondary && current_time >= @secondary_transition_at + if !@forever && @secondary && @current != :secondary && current_time >= @secondary_transition_at @current = :secondary @secondary_transition_steps = @steps end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin_helper/server.rb new/lib/fluent/plugin_helper/server.rb --- old/lib/fluent/plugin_helper/server.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/plugin_helper/server.rb 2019-03-19 14:48:00.000000000 +0100 @@ -523,7 +523,7 @@ def on_readable_without_sock begin data = @sock.recv(@max_bytes, @flags) - rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET + rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET, IOError, Errno::EBADF return end @callback.call(data) @@ -536,7 +536,7 @@ def on_readable_with_sock begin data, addr = @sock.recvfrom(@max_bytes) - rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET + rescue Errno::EAGAIN, Errno::EWOULDBLOCK, Errno::EINTR, Errno::ECONNRESET, IOError, Errno::EBADF return end @callback.call(data, UDPCallbackSocket.new(@sock, addr, close_socket: @close_socket)) @@ -722,7 +722,7 @@ return true end - rescue Errno::ECONNRESET, OpenSSL::SSL::SSLError => e + rescue Errno::EPIPE, Errno::ECONNRESET, OpenSSL::SSL::SSLError => e @log.trace "unexpected error before accepting TLS connection", error: e close rescue nil end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/root_agent.rb new/lib/fluent/root_agent.rb --- old/lib/fluent/root_agent.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/root_agent.rb 2019-03-19 14:48:00.000000000 +0100 @@ -64,26 +64,64 @@ attr_reader :labels def configure(conf) + used_worker_ids = [] + available_worker_ids = (0..Fluent::Engine.system_config.workers - 1).to_a # initialize <worker> elements conf.elements(name: 'worker').each do |e| target_worker_id_str = e.arg if target_worker_id_str.empty? - raise ConfigError, "Missing worker id on <worker> directive" + raise Fluent::ConfigError, "Missing worker id on <worker> directive" end - target_worker_id = target_worker_id_str.to_i - if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1) - raise ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}" - end + target_worker_ids = target_worker_id_str.split("-") + if target_worker_ids.size == 2 + first_worker_id = target_worker_ids.first.to_i + last_worker_id = target_worker_ids.last.to_i + if first_worker_id > last_worker_id + raise Fluent::ConfigError, "greater first_worker_id<#{first_worker_id}> than last_worker_id<#{last_worker_id}> specified by <worker> directive is not allowed. Available multi worker assign syntax is <smaller_worker_id>-<greater_worker_id>" + end + target_worker_ids = [] + first_worker_id.step(last_worker_id, 1) do |worker_id| + target_worker_id = worker_id.to_i + target_worker_ids << target_worker_id + + if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1) + raise Fluent::ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}" + end + available_worker_ids.delete(target_worker_id) if available_worker_ids.include?(target_worker_id) + if used_worker_ids.include?(target_worker_id) && !Fluent::Engine.dry_run_mode + raise Fluent::ConfigError, "specified worker_id<#{worker_id}> collisions is detected on <worker> directive. Available worker id(s): #{available_worker_ids}" + end + used_worker_ids << target_worker_id + + e.elements.each do |elem| + unless ['source', 'match', 'filter', 'label'].include?(elem.name) + raise Fluent::ConfigError, "<worker> section cannot have <#{elem.name}> directive" + end + end + + # On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0). + target_worker_ids = [0] if Fluent::Engine.dry_run_mode + + unless target_worker_ids.empty? + e.set_target_worker_ids(target_worker_ids.uniq) + end + end + else + target_worker_id = target_worker_id_str.to_i + if target_worker_id < 0 || target_worker_id > (Fluent::Engine.system_config.workers - 1) + raise Fluent::ConfigError, "worker id #{target_worker_id} specified by <worker> directive is not allowed. Available worker id is between 0 and #{(Fluent::Engine.system_config.workers - 1)}" + end - ## On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0). - target_worker_id = 0 if Fluent::Engine.dry_run_mode + ## On dry_run mode, all worker sections have to be configured on supervisor (recognized as worker_id = 0). + target_worker_id = 0 if Fluent::Engine.dry_run_mode - e.elements.each do |elem| - unless ['source', 'match', 'filter', 'label'].include?(elem.name) - raise ConfigError, "<worker> section cannot have <#{elem.name}> directive" + e.elements.each do |elem| + unless ['source', 'match', 'filter', 'label'].include?(elem.name) + raise Fluent::ConfigError, "<worker> section cannot have <#{elem.name}> directive" + end + elem.set_target_worker_id(target_worker_id) end - elem.set_target_worker_id(target_worker_id) end conf += e end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/supervisor.rb new/lib/fluent/supervisor.rb --- old/lib/fluent/supervisor.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/supervisor.rb 2019-03-19 14:48:00.000000000 +0100 @@ -716,7 +716,13 @@ end def main_process(&block) - Process.setproctitle("worker:#{@process_name}") if @process_name + if @process_name + if @workers > 1 + Process.setproctitle("worker:#{@process_name}#{ENV['SERVERENGINE_WORKER_ID']}") + else + Process.setproctitle("worker:#{@process_name}") + end + end unrecoverable_error = false diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/version.rb new/lib/fluent/version.rb --- old/lib/fluent/version.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/lib/fluent/version.rb 2019-03-19 14:48:00.000000000 +0100 @@ -16,6 +16,6 @@ module Fluent - VERSION = '1.3.3' + VERSION = '1.4.1' end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/metadata new/metadata --- old/metadata 2019-01-07 04:07:42.000000000 +0100 +++ new/metadata 2019-03-19 14:48:00.000000000 +0100 @@ -1,14 +1,14 @@ --- !ruby/object:Gem::Specification name: fluentd version: !ruby/object:Gem::Version - version: 1.3.3 + version: 1.4.1 platform: ruby authors: - Sadayuki Furuhashi autorequire: bindir: bin cert_chain: [] -date: 2019-01-07 00:00:00.000000000 Z +date: 2019-03-19 00:00:00.000000000 Z dependencies: - !ruby/object:Gem::Dependency name: msgpack @@ -329,6 +329,7 @@ extra_rdoc_files: [] files: - ".github/ISSUE_TEMPLATE.md" +- ".github/PULL_REQUEST_TEMPLATE.md" - ".gitignore" - ".travis.yml" - ADOPTERS.md @@ -337,6 +338,7 @@ - CONTRIBUTING.md - GOVERNANCE.md - Gemfile +- GithubWorkflow.md - LICENSE - MAINTAINERS.md - README.md diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/command/test_fluentd.rb new/test/command/test_fluentd.rb --- old/test/command/test_fluentd.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/command/test_fluentd.rb 2019-03-19 14:48:00.000000000 +0100 @@ -340,7 +340,7 @@ end end - sub_test_case 'configured to suppress configration dump' do + sub_test_case 'configured to suppress configuration dump' do setup do @basic_conf = <<CONF <source> @@ -614,7 +614,7 @@ ) end - test 'failed to start workers when configured plugins as chidren of MultiOutput do not support multi worker configuration' do + test 'failed to start workers when configured plugins as children of MultiOutput do not support multi worker configuration' do script = <<-EOC require 'fluent/plugin/output' module Fluent::Plugin diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/command/test_plugin_config_formatter.rb new/test/command/test_plugin_config_formatter.rb --- old/test/command/test_plugin_config_formatter.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/command/test_plugin_config_formatter.rb 2019-03-19 14:48:00.000000000 +0100 @@ -72,7 +72,7 @@ desc "username" config_param :username, :string desc "password" - config_param :passowrd, :string, secret: true + config_param :password, :string, secret: true end config_section :parent do @@ -162,7 +162,7 @@ <secondary>: optional, single <authentication>: required, single username: string: (nil) - passowrd: string: (nil) + password: string: (nil) <parent>: optional, multiple <child>: optional, multiple names: array: (nil) @@ -217,7 +217,7 @@ username -#### passowrd (string) (required) +#### password (string) (required) password diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/compat/test_parser.rb new/test/compat/test_parser.rb --- old/test/compat/test_parser.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/compat/test_parser.rb 2019-03-19 14:48:00.000000000 +0100 @@ -84,7 +84,7 @@ multiline: Regexp::MULTILINE, both: Regexp::IGNORECASE & Regexp::MULTILINE) def test_regexp_parser_config(options) - source = "a" + source = "(?<test>.*)" parser = Fluent::TextParser::RegexpParser.new(Regexp.new(source, options), { "dummy" => "dummy" }) regexp = parser.instance_variable_get("@regexp") assert_equal(options, regexp.options) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_config_parser.rb new/test/config/test_config_parser.rb --- old/test/config/test_config_parser.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/config/test_config_parser.rb 2019-03-19 14:48:00.000000000 +0100 @@ -329,6 +329,36 @@ end end + sub_test_case "Embedded Ruby Code in section attributes" do + setup do + ENV["EMBEDDED_VAR"] = "embedded" + ENV["NESTED_EMBEDDED_VAR"] = "nested-embedded" + @hostname = Socket.gethostname + end + + teardown do + ENV["EMBEDDED_VAR"] = nil + ENV["NESTED_EMBEDDED_VAR"] = nil + end + + test "embedded Ruby code should be expanded" do + assert_text_parsed_as(root( + e("test", 'embedded', {'key'=>'1'}, [ + e('nested1', 'nested-embedded'), + e('nested2', "#{@hostname}") + ])), <<-EOF + <test "#{ENV["EMBEDDED_VAR"]}"> + key 1 + <nested1 "#{ENV["NESTED_EMBEDDED_VAR"]}"> + </nested1> + <nested2 "#{Socket.gethostname}"> + </nested2> + </test> + EOF + ) + end + end + # port from test_config.rb sub_test_case '@include parsing' do TMP_DIR = File.dirname(__FILE__) + "/tmp/v1_config#{ENV['TEST_ENV_NUMBER']}" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_configurable.rb new/test/config/test_configurable.rb --- old/test/config/test_configurable.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/config/test_configurable.rb 2019-03-19 14:48:00.000000000 +0100 @@ -733,7 +733,7 @@ assert_nothing_raised { b4.configure(config_element('ROOT', '', BASE_ATTRS, [d1.dup, d2.dup, d3.dup, d3.dup])) } end - test 'constructs confuguration object tree for Base3' do + test 'constructs configuration object tree for Base3' do conf = config_element( 'ROOT', '', diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_element.rb new/test/config/test_element.rb --- old/test/config/test_element.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/config/test_element.rb 2019-03-19 14:48:00.000000000 +0100 @@ -157,7 +157,7 @@ Fluent::Config::Element.new('ROOT', 'mydata', {}, [])], "differ keys" => [Fluent::Config::Element.new('ROOT', 'mydata', {}, []), Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, [])], - "differ elemnts" => + "differ elements" => [Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, []), Fluent::Config::Element.new('ROOT', 'mydata', {"k1" => "v1"}, [ Fluent::Config::Element.new('test', 'mydata', {'k3' => 'v3'}, []) @@ -406,11 +406,11 @@ test 'set target_worker_id recursively' do e = element('label', '@mytest', {}, [ element('filter', '**'), element('match', '**', {}, [ element('store'), element('store') ]) ]) e.set_target_worker_id(1) - assert_equal 1, e.target_worker_id - assert_equal 1, e.elements[0].target_worker_id - assert_equal 1, e.elements[1].target_worker_id - assert_equal 1, e.elements[1].elements[0].target_worker_id - assert_equal 1, e.elements[1].elements[1].target_worker_id + assert_equal [1], e.target_worker_ids + assert_equal [1], e.elements[0].target_worker_ids + assert_equal [1], e.elements[1].target_worker_ids + assert_equal [1], e.elements[1].elements[0].target_worker_ids + assert_equal [1], e.elements[1].elements[1].target_worker_ids end end @@ -434,12 +434,24 @@ assert e.for_this_worker? end + test 'target_worker_ids includes current worker_id' do + e = element() + e.set_target_worker_ids([0]) + assert e.for_this_worker? + end + test 'target_worker_id != current worker_id' do e = element() e.set_target_worker_id(1) assert_false e.for_this_worker? end + test 'target_worker_ids does not includes current worker_id' do + e = element() + e.set_target_worker_ids([1, 2]) + assert_false e.for_this_worker? + end + test "doesn't have target_worker_id" do e = element() assert_false e.for_this_worker? @@ -453,12 +465,24 @@ assert_false e.for_another_worker? end + test 'target_worker_ids contains current worker_id' do + e = element() + e.set_target_worker_ids([0, 1]) + assert_false e.for_another_worker? + end + test 'target_worker_id != current worker_id' do e = element() e.set_target_worker_id(1) assert e.for_another_worker? end + test 'target_worker_ids does not contains current worker_id' do + e = element() + e.set_target_worker_ids([1, 2]) + assert e.for_another_worker? + end + test "doesn't have target_worker_id" do e = element() assert_false e.for_another_worker? diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/counter/test_store.rb new/test/counter/test_store.rb --- old/test/counter/test_store.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/counter/test_store.rb 2019-03-19 14:48:00.000000000 +0100 @@ -89,7 +89,7 @@ assert_equal nil, @store.get('unknown_key') end - test "raise a error when when a passed key doesn't exist and raise_error option is true" do + test "raise a error when a passed key doesn't exist and raise_error option is true" do assert_raise Fluent::Counter::UnknownKey do @store.get('unknown_key', raise_error: true) end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_buffer.rb new/test/plugin/test_buffer.rb --- old/test/plugin/test_buffer.rb 2019-01-07 04:07:42.000000000 +0100 +++ new/test/plugin/test_buffer.rb 2019-03-19 14:48:00.000000000 +0100 @@ -211,7 +211,7 @@ assert_equal 2, @p.queued_num[@dm1] end - test '#close closes all chunks in in dequeued, enqueued and staged' do + test '#close closes all chunks in dequeued, enqueued and staged' do dmx = create_metadata(Time.parse('2016-04-11 15:50:00 +0000').to_i, nil, nil) cx = create_chunk(dmx, ["x" * 1024]) @p.dequeued[cx.unique_id] = cx @@ -1027,7 +1027,7 @@ ##### 900 + 9500 + 9900 * 4 == 5000 + 45000 end - test '#write raises BufferChunkOverflowError if a record is biggar than chunk limit size' do + test '#write raises BufferChunkOverflowError if a record is bigger than chunk limit size' do assert_equal [@dm0], @p.stage.keys assert_equal [], @p.queue.map(&:metadata) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_filter.rb new/test/plugin/test_filter.rb --- old/test/plugin/test_filter.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_filter.rb 2019-03-19 14:48:00.000000000 +0100 @@ -47,7 +47,7 @@ end end class InvalidPlugin < Fluent::Plugin::Filter - # Because of implemnting `filter_with_time` and `filter` methods + # Because of implementing `filter_with_time` and `filter` methods def filter_with_time(tag, time, record); end def filter(tag, time, record); end end @@ -277,7 +277,7 @@ end end - sub_test_case 'filter plugins that is implmented `filter_with_time`' do + sub_test_case 'filter plugins that is implemented `filter_with_time`' do setup do Fluent::Test.setup @p = FluentPluginFilterTest::NumDoublePluginWithTime.new diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_filter_record_transformer.rb new/test/plugin/test_filter_record_transformer.rb --- old/test/plugin/test_filter_record_transformer.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_filter_record_transformer.rb 2019-03-19 14:48:00.000000000 +0100 @@ -229,7 +229,7 @@ end %w[yes no].each do |enable_ruby| - test "hostname with enble_ruby #{enable_ruby}" do + test "hostname with enable_ruby #{enable_ruby}" do config = %[ enable_ruby #{enable_ruby} <record> diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_tail.rb new/test/plugin/test_in_tail.rb --- old/test/plugin/test_in_tail.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_in_tail.rb 2019-03-19 14:48:00.000000000 +0100 @@ -649,6 +649,26 @@ assert_equal(Encoding::UTF_8, events[0][2]['message'].encoding) end + def test_encoding_with_bad_character + conf = config_element( + "", "", { + "format" => "/(?<message>.*)/", + "read_from_head" => "true", + "from_encoding" => "ASCII-8BIT", + "encoding" => "utf-8" + }) + d = create_driver(conf) + + d.run(expect_emits: 1) do + File.open("#{TMP_DIR}/tail.txt", "w") { |f| + f.write "te\x86st\n" + } + end + + events = d.events + assert_equal("te\uFFFDst", events[0][2]['message']) + assert_equal(Encoding::UTF_8, events[0][2]['message'].encoding) + end sub_test_case "multiline" do data(flat: MULTILINE_CONFIG, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_tcp.rb new/test/plugin/test_in_tcp.rb --- old/test/plugin/test_in_tcp.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_in_tcp.rb 2019-03-19 14:48:00.000000000 +0100 @@ -48,6 +48,12 @@ assert_equal "\n", d.instance.delimiter end + test ' configure w/o parse section' do + assert_raise(Fluent::ConfigError.new("<parse> section is required.")) { + create_driver(BASE_CONFIG) + } + end + test_case_data = { 'none' => { 'format' => 'none', diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_udp.rb new/test/plugin/test_in_udp.rb --- old/test/plugin/test_in_udp.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_in_udp.rb 2019-03-19 14:48:00.000000000 +0100 @@ -59,6 +59,12 @@ assert_equal nil, d.instance.receive_buffer_size end + test ' configure w/o parse section' do + assert_raise(Fluent::ConfigError.new("<parse> section is required.")) { + create_driver(BASE_CONFIG) + } + end + data( 'ipv4' => [CONFIG, '127.0.0.1', :ipv4], 'ipv6' => [IPv6_CONFIG, '::1', :ipv6], diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_out_file.rb new/test/plugin/test_out_file.rb --- old/test/plugin/test_out_file.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_out_file.rb 2019-03-19 14:48:00.000000000 +0100 @@ -14,7 +14,6 @@ end TMP_DIR = File.expand_path(File.dirname(__FILE__) + "/../tmp/out_file#{ENV['TEST_ENV_NUMBER']}") - SYMLINK_PATH = File.expand_path("#{TMP_DIR}/current") CONFIG = %[ path #{TMP_DIR}/out_file_test @@ -663,15 +662,43 @@ end end - test 'symlink' do - omit "Windows doesn't support symlink" if Fluent.windows? - conf = CONFIG + %[ - symlink_path #{SYMLINK_PATH} - ] - symlink_path = "#{SYMLINK_PATH}" + SYMLINK_PATH = File.expand_path("#{TMP_DIR}/current") + + sub_test_case 'symlink' do + test 'static symlink' do + omit "Windows doesn't support symlink" if Fluent.windows? + conf = CONFIG + %[ + symlink_path #{SYMLINK_PATH} + ] + symlink_path = "#{SYMLINK_PATH}" + + d = create_driver(conf) + begin + run_and_check(d, symlink_path) + ensure + FileUtils.rm_rf(symlink_path) + end + end + + test 'symlink with placeholders' do + omit "Windows doesn't support symlink" if Fluent.windows? + conf = %[ + path #{TMP_DIR}/${tag}/out_file_test + symlink_path #{SYMLINK_PATH}-${tag} + <buffer tag,time> + </buffer> + ] + symlink_path = "#{SYMLINK_PATH}-tag" + + d = create_driver(conf) + begin + run_and_check(d, symlink_path) + ensure + FileUtils.rm_rf(symlink_path) + end + end - d = create_driver(conf) - begin + def run_and_check(d, symlink_path) d.run(default_tag: 'tag') do es = Fluent::OneEventStream.new(event_time("2011-01-02 13:14:15 UTC"), {"a"=>1}) d.feed(es) @@ -688,8 +715,6 @@ meta = d.instance.metadata('tag', event_time("2011-01-03 14:15:16 UTC"), {}) assert_equal d.instance.buffer.instance_eval{ @stage[meta].path }, File.readlink(symlink_path) end - ensure - FileUtils.rm_rf(symlink_path) end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_out_secondary_file.rb new/test/plugin/test_out_secondary_file.rb --- old/test/plugin/test_out_secondary_file.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_out_secondary_file.rb 2019-03-19 14:48:00.000000000 +0100 @@ -39,7 +39,7 @@ c.configure(conf) end - sub_test_case 'configture' do + sub_test_case 'configure' do test 'default configuration' do d = create_driver %[directory #{TMP_DIR}] assert_equal 'dump.bin', d.instance.basename diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output.rb new/test/plugin/test_output.rb --- old/test/plugin/test_output.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_output.rb 2019-03-19 14:48:00.000000000 +0100 @@ -547,7 +547,7 @@ assert_equal 86400, s assert_equal :day, t assert_equal '%d', e - s, t, e = @i.get_placeholders_time("my birthiday! at %F") + s, t, e = @i.get_placeholders_time("my birthday! at %F") assert_equal 86400, s assert_equal :day, t assert_equal '%d', e diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output_as_buffered.rb new/test/plugin/test_output_as_buffered.rb --- old/test/plugin/test_output_as_buffered.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_output_as_buffered.rb 2019-03-19 14:48:00.000000000 +0100 @@ -1074,6 +1074,44 @@ end end + sub_test_case 'buffered output with large timekey and small timekey_wait' do + setup do + chunk_key = 'time' + hash = { + 'timekey' => 86400, # per 1 day + 'timekey_wait' => 10, # 10 seconds delay for flush + 'flush_thread_count' => 1, + 'flush_thread_burst_interval' => 0.01, + } + @i = create_output(:buffered) + @i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash)])) + @i.start + @i.after_start + end + + test '#configure raises config error if timekey is not specified' do + Timecop.freeze( Time.parse('2019-02-08 00:01:00 +0900') ) + ary = [] + @i.register(:write){|chunk| ary << chunk.read } + @i.thread_wait_until_start + events = [ + [event_time('2019-02-08 00:02:00 +0900'), {"message" => "foobar"}] + ] + @i.emit_events("test.tag", Fluent::ArrayEventStream.new(events)) + @i.enqueue_thread_wait + assert{ @i.write_count == 0 } + + Timecop.freeze( Time.parse('2019-02-09 00:00:08 +0900') ) + @i.enqueue_thread_wait + assert{ @i.write_count == 0 } + + Timecop.freeze( Time.parse('2019-02-09 00:00:12 +0900') ) + # wirte should be called in few seconds since + # running interval of enque thread is timekey_wait / 11.0. + waiting(5){ sleep 0.1 until @i.write_count == 1 } + end + end + sub_test_case 'buffered output feature with tag key' do setup do chunk_key = 'tag' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output_as_buffered_backup.rb new/test/plugin/test_output_as_buffered_backup.rb --- old/test/plugin/test_output_as_buffered_backup.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_output_as_buffered_backup.rb 2019-03-19 14:48:00.000000000 +0100 @@ -177,7 +177,12 @@ } end - test 'backup chunk without secondary' do + data('unrecoverable error' => Fluent::UnrecoverableError, + 'type error' => TypeError, + 'argument error' => ArgumentError, + 'no method error' => NoMethodError, + 'msgpack unpack error' => MessagePack::UnpackError) + test 'backup chunk without secondary' do |error_class| Fluent::SystemConfig.overwrite_system_config('root_dir' => TMP_DIR) do id = 'backup_test' hash = { @@ -188,7 +193,7 @@ @i.configure(config_element('ROOT', '', {'@id' => id}, [config_element('buffer', 'tag', hash)])) @i.register(:write) { |chunk| chunk_id = chunk.unique_id; - raise Fluent::UnrecoverableError, "yay, your #write must fail" + raise error_class, "yay, your #write must fail" } flush_chunks diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output_as_buffered_retries.rb new/test/plugin/test_output_as_buffered_retries.rb --- old/test/plugin/test_output_as_buffered_retries.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_output_as_buffered_retries.rb 2019-03-19 14:48:00.000000000 +0100 @@ -706,16 +706,20 @@ end sub_test_case 'buffered output configured as retry_forever' do - test 'configuration error will be raised if secondary section is configured' do + setup do + Fluent::Plugin.register_output('output_retries_secondary_test', FluentPluginOutputAsBufferedRetryTest::DummyFullFeatureOutput2) + end + + test 'warning logs are generated if secondary section is configured' do chunk_key = 'tag' hash = { 'retry_forever' => true, 'retry_randomize' => false, } i = create_output() - assert_raise Fluent::ConfigError do - i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash),config_element('secondary','')])) - end + i.configure(config_element('ROOT','',{},[config_element('buffer',chunk_key,hash),config_element('secondary','', {'@type' => 'output_retries_secondary_test'})])) + logs = i.log.out.logs + assert { logs.any? { |l| l.include?("<secondary> with 'retry_forever', only unrecoverable errors are moved to secondary") } } end test 'retry_timeout and retry_max_times will be ignored if retry_forever is true for exponential backoff' do diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_parser_regexp.rb new/test/plugin/test_parser_regexp.rb --- old/test/plugin/test_parser_regexp.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_parser_regexp.rb 2019-03-19 14:48:00.000000000 +0100 @@ -153,6 +153,15 @@ end sub_test_case "configure" do + def test_bad_expression + conf = { + 'expression' => %q!/.*/!, + } + assert_raise Fluent::ConfigError do + create_driver(conf) + end + end + def test_default_options conf = { 'expression' => %q!/^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] \[(?<date>[^\]]*)\] "(?<flag>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)$/!, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_parser_syslog.rb new/test/plugin/test_parser_syslog.rb --- old/test/plugin/test_parser_syslog.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin/test_parser_syslog.rb 2019-03-19 14:48:00.000000000 +0100 @@ -276,7 +276,7 @@ def test_auto_with_legacy_syslog_message @parser.configure( 'time_format' => '%b %d %M:%S:%H', - 'mseeage_format' => 'auto', + 'message_format' => 'auto', ) text = 'Feb 28 00:00:12 192.168.0.1 fluentd[11111]: [error] Syslog test' @parser.instance.parse(text) do |time, record| @@ -290,7 +290,7 @@ @parser.configure( 'time_format' => '%b %d %M:%S:%H', 'with_priority' => true, - 'mseeage_format' => 'auto', + 'message_format' => 'auto', ) text = '<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test' @parser.instance.parse(text) do |time, record| diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin_helper/test_record_accessor.rb new/test/plugin_helper/test_record_accessor.rb --- old/test/plugin_helper/test_record_accessor.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin_helper/test_record_accessor.rb 2019-03-19 14:48:00.000000000 +0100 @@ -48,7 +48,7 @@ data("missing ']'" => "$['key1'", "missing array index with dot" => "$.hello[]", - "missing array index with braket" => "$[]", + "missing array index with bracket" => "$[]", "more chars" => "$.key1[0]foo", "whitespace char included key in dot notation" => "$.key[0].ke y", "empty keys with dot" => "$.", diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin_helper/test_server.rb new/test/plugin_helper/test_server.rb --- old/test/plugin_helper/test_server.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/plugin_helper/test_server.rb 2019-03-19 14:48:00.000000000 +0100 @@ -767,7 +767,7 @@ def write_cert_and_key(cert_path, cert, key_path, key, passphrase) File.open(cert_path, "w"){|f| f.write(cert.to_pem) } - # Write the secret key (raw or ecnrypted by AES256) in PEM format + # Write the secret key (raw or encrypted by AES256) in PEM format key_str = passphrase ? key.export(OpenSSL::Cipher.new("AES-256-CBC"), passphrase) : key.export File.open(key_path, "w"){|f| f.write(key_str) } File.chmod(0600, cert_path, key_path) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_output.rb new/test/test_output.rb --- old/test/test_output.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/test_output.rb 2019-03-19 14:48:00.000000000 +0100 @@ -129,7 +129,7 @@ end def test_secondary_with_no_warn_log - # ObjectBufferedOutput doesn't implemnt `custom_filter` + # ObjectBufferedOutput doesn't implement `custom_filter` d = Fluent::Test::BufferedOutputTestDriver.new(Fluent::ObjectBufferedOutput) mock(d.instance.log).warn("secondary type should be same with primary one", diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_plugin.rb new/test/test_plugin.rb --- old/test/test_plugin.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/test_plugin.rb 2019-03-19 14:48:00.000000000 +0100 @@ -97,7 +97,7 @@ output1: ['plugin_test_dummy1', Dummy1Output, :new_output], output2: ['plugin_test_dummy2', Dummy2Output, :new_output], ) - test 'retruns plugin instances of registered plugin classes' do |(type, klass, m)| + test 'returns plugin instances of registered plugin classes' do |(type, klass, m)| instance = Fluent::Plugin.__send__(m, type) assert_kind_of klass, instance end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_root_agent.rb new/test/test_root_agent.rb --- old/test/test_root_agent.rb 2019-01-07 04:07:43.000000000 +0100 +++ new/test/test_root_agent.rb 2019-03-19 14:48:00.000000000 +0100 @@ -686,6 +686,54 @@ end end + test 'raises configuration error for too big worker id on multi workers syntax' do + errmsg = "worker id 4 specified by <worker> directive is not allowed. Available worker id is between 0 and 3" + assert_raise Fluent::ConfigError.new(errmsg) do + conf = <<-EOC +<worker 1-4> +</worker> +EOC + configure_ra(conf) + end + end + + test 'raises configuration error for worker id collisions on multi workers syntax' do + errmsg = "specified worker_id<2> collisions is detected on <worker> directive. Available worker id(s): [3]" + assert_raise Fluent::ConfigError.new(errmsg) do + conf = <<-EOC +<worker 0-2> +</worker> +<worker 2-4> +</worker> +EOC + configure_ra(conf) + end + end + + test 'raises configuration error for worker id collisions on multi workers syntax when multi avaliable worker_ids are left' do + errmsg = "specified worker_id<1> collisions is detected on <worker> directive. Available worker id(s): [2, 3]" + assert_raise Fluent::ConfigError.new(errmsg) do + conf = <<-EOC +<worker 0-1> +</worker> +<worker 1-3> +</worker> +EOC + configure_ra(conf) + end + end + + test 'raises configuration error for too big worker id on invalid reversed multi workers syntax' do + errmsg = "greater first_worker_id<3> than last_worker_id<0> specified by <worker> directive is not allowed. Available multi worker assign syntax is <smaller_worker_id>-<greater_worker_id>" + assert_raise Fluent::ConfigError.new(errmsg) do + conf = <<-EOC +<worker 3-0> +</worker> +EOC + configure_ra(conf) + end + end + test 'raises configuration error for invalid elements as a child of worker section' do errmsg = '<worker> section cannot have <system> directive' assert_raise Fluent::ConfigError.new(errmsg) do @@ -844,5 +892,33 @@ assert_equal 0, ra.labels.size refute ra.error_collector end + + test 'with plugins for workers syntax should match worker_id equals to 2' do + conf = <<-EOC +<worker 0-2> + <source> + @type forward + </source> + <filter **> + @type test_filter + @id test_filter + </filter> + <match pattern> + @type stdout + </match> + <label @ERROR> + <match> + @type null + </match> + </label> +</worker> +EOC + + ra = configure_ra(conf) + assert_kind_of Fluent::Plugin::ForwardInput, ra.inputs.first + assert_kind_of Fluent::Plugin::StdoutOutput, ra.outputs.first + assert_kind_of FluentTestFilter, ra.filters.first + assert ra.error_collector + end end end