Hello community, here is the log from the commit of package rubygem-fluentd for openSUSE:Factory checked in at 2017-04-11 09:33:34 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/rubygem-fluentd (Old) and /work/SRC/openSUSE:Factory/.rubygem-fluentd.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "rubygem-fluentd" Tue Apr 11 09:33:34 2017 rev:2 rq:482700 version:0.14.14 Changes: -------- --- /work/SRC/openSUSE:Factory/rubygem-fluentd/rubygem-fluentd.changes 2017-03-09 02:01:50.608409683 +0100 +++ /work/SRC/openSUSE:Factory/.rubygem-fluentd.new/rubygem-fluentd.changes 2017-04-11 09:33:39.162641313 +0200 @@ -1,0 +2,44 @@ +Fri Mar 24 05:29:11 UTC 2017 - co...@suse.com + +- updated to version 0.14.14 + see installed ChangeLog + + ## Release v0.14.14 - 2017/03/23 + + ### New features / Enhancements + + * in_http: Support 'application/msgpack` header + https://github.com/fluent/fluentd/pull/1498 + * in_udp: Add message_length_limit parameter for parameter name consistency with in_syslog + https://github.com/fluent/fluentd/pull/1515 + * in_monitor_agent: Start one HTTP server per worker on sequential port numbers + https://github.com/fluent/fluentd/pull/1493 + * in_tail: Skip the refresh of watching list on startup + https://github.com/fluent/fluentd/pull/1487 + * filter_parser: filter_parser: Add emit_invalid_record_to_error parameter + https://github.com/fluent/fluentd/pull/1494 + * parser_syslog: Support RFC5424 syslog format + https://github.com/fluent/fluentd/pull/1492 + * parser: Allow escape sequence in Apache access log + https://github.com/fluent/fluentd/pull/1479 + * config: Add actual value in the placholder error message + https://github.com/fluent/fluentd/pull/1497 + * log: Add Fluent::Log#<< to support some SDKs + https://github.com/fluent/fluentd/pull/1478 + + ### Bug fixes + + * Fix cleanup resource + https://github.com/fluent/fluentd/pull/1483 + * config: Set encoding forcefully to avoid UndefinedConversionError + https://github.com/fluent/fluentd/pull/1477 + * Fix Input and Output deadlock when buffer is full during startup + https://github.com/fluent/fluentd/pull/1502 + * config: Fix log_level handling in <system> + https://github.com/fluent/fluentd/pull/1501 + * Fix typo in root agent error log + https://github.com/fluent/fluentd/pull/1491 + * storage: Fix a bug storage_create cannot accept hash as `conf` keyword argument + https://github.com/fluent/fluentd/pull/1482 + +------------------------------------------------------------------- Old: ---- fluentd-0.14.13.gem New: ---- fluentd-0.14.14.gem ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ rubygem-fluentd.spec ++++++ --- /var/tmp/diff_new_pack.WZBnrL/_old 2017-04-11 09:33:39.902536792 +0200 +++ /var/tmp/diff_new_pack.WZBnrL/_new 2017-04-11 09:33:39.906536228 +0200 @@ -24,7 +24,7 @@ # Name: rubygem-fluentd -Version: 0.14.13 +Version: 0.14.14 Release: 0 %define mod_name fluentd %define mod_full_name %{mod_name}-%{version} ++++++ fluentd-0.14.13.gem -> fluentd-0.14.14.gem ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ChangeLog new/ChangeLog --- old/ChangeLog 2017-02-17 22:22:06.000000000 +0100 +++ new/ChangeLog 2017-03-24 05:16:16.000000000 +0100 @@ -1,5 +1,43 @@ # v0.14 +## Release v0.14.14 - 2017/03/23 + +### New features / Enhancements + +* in_http: Support 'application/msgpack` header + https://github.com/fluent/fluentd/pull/1498 +* in_udp: Add message_length_limit parameter for parameter name consistency with in_syslog + https://github.com/fluent/fluentd/pull/1515 +* in_monitor_agent: Start one HTTP server per worker on sequential port numbers + https://github.com/fluent/fluentd/pull/1493 +* in_tail: Skip the refresh of watching list on startup + https://github.com/fluent/fluentd/pull/1487 +* filter_parser: filter_parser: Add emit_invalid_record_to_error parameter + https://github.com/fluent/fluentd/pull/1494 +* parser_syslog: Support RFC5424 syslog format + https://github.com/fluent/fluentd/pull/1492 +* parser: Allow escape sequence in Apache access log + https://github.com/fluent/fluentd/pull/1479 +* config: Add actual value in the placholder error message + https://github.com/fluent/fluentd/pull/1497 +* log: Add Fluent::Log#<< to support some SDKs + https://github.com/fluent/fluentd/pull/1478 + +### Bug fixes + +* Fix cleanup resource + https://github.com/fluent/fluentd/pull/1483 +* config: Set encoding forcefully to avoid UndefinedConversionError + https://github.com/fluent/fluentd/pull/1477 +* Fix Input and Output deadlock when buffer is full during startup + https://github.com/fluent/fluentd/pull/1502 +* config: Fix log_level handling in <system> + https://github.com/fluent/fluentd/pull/1501 +* Fix typo in root agent error log + https://github.com/fluent/fluentd/pull/1491 +* storage: Fix a bug storage_create cannot accept hash as `conf` keyword argument + https://github.com/fluent/fluentd/pull/1482 + ## Release v0.14.13 - 2017/02/17 ### New features / Enhancements diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/README.md new/README.md --- old/README.md 2017-02-17 22:22:06.000000000 +0100 +++ new/README.md 2017-03-24 05:16:16.000000000 +0100 @@ -26,6 +26,31 @@ $ fluentd -c conf/fluent.conf & $ echo '{"json":"message"}' | fluent-cat debug.test +## Development + +### Prerequisites + +- Ruby 2.1 or later +- git + +`git` should be in `PATH`. On Windows, you can use `Github for Windows` and `GitShell` for easy setup. + +### Install dependent gems + +Use bundler: + + $ gem install bundler + $ bundle install --path vendor/bundle + +### Run test + + $ bundle exec rake test + +You can run specified test via `TEST` environment variable: + + $ bundle exec rake test TEST=test/test_specified_path.rb + $ bundle exec rake test TEST=test/test_*.rb + ## Fluentd UI: Admin GUI [Fluentd UI](https://github.com/fluent/fluentd-ui) is a graphical user interface to start/stop/configure Fluentd. Binary files old/checksums.yaml.gz and new/checksums.yaml.gz differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/config/types.rb new/lib/fluent/config/types.rb --- old/lib/fluent/config/types.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/config/types.rb 2017-03-24 05:16:16.000000000 +0100 @@ -64,7 +64,7 @@ end end - STRING_TYPE = Proc.new { |val, opts| val.to_s.encode(Encoding::UTF_8) } + STRING_TYPE = Proc.new { |val, opts| val.to_s.force_encoding(Encoding::UTF_8) } ENUM_TYPE = Proc.new { |val, opts| s = val.to_sym list = opts[:list] @@ -85,7 +85,7 @@ value else case type - when :string then value.to_s.encode(Encoding::UTF_8) + when :string then value.to_s.force_encoding(Encoding::UTF_8) when :integer then value.to_i when :float then value.to_f when :size then Config.size_value(value) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/env.rb new/lib/fluent/env.rb --- old/lib/fluent/env.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/env.rb 2017-03-24 05:16:16.000000000 +0100 @@ -14,15 +14,15 @@ # limitations under the License. # +require 'serverengine/utils' + module Fluent DEFAULT_CONFIG_PATH = ENV['FLUENT_CONF'] || '/etc/fluent/fluent.conf' DEFAULT_PLUGIN_DIR = ENV['FLUENT_PLUGIN'] || '/etc/fluent/plugin' DEFAULT_SOCKET_PATH = ENV['FLUENT_SOCKET'] || '/var/run/fluent/fluent.sock' DEFAULT_OJ_OPTIONS = {bigdecimal_load: :float, mode: :compat, use_to_json: true} - IS_WINDOWS = /mswin|mingw/ === RUBY_PLATFORM - private_constant :IS_WINDOWS def self.windows? - IS_WINDOWS + ServerEngine.windows? end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/log.rb new/lib/fluent/log.rb --- old/lib/fluent/log.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/log.rb 2017-03-24 05:16:16.000000000 +0100 @@ -357,6 +357,9 @@ def write(data) @out.write(data) end + # We need `#<<` method to use this logger class with other + # libraries such as aws-sdk + alias << write def flush @out.flush diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/filter_parser.rb new/lib/fluent/plugin/filter_parser.rb --- old/lib/fluent/plugin/filter_parser.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/filter_parser.rb 2017-03-24 05:16:16.000000000 +0100 @@ -32,6 +32,7 @@ config_param :inject_key_prefix, :string, default: nil config_param :replace_invalid_sequence, :bool, default: false config_param :hash_value_field, :string, default: nil + config_param :emit_invalid_record_to_error, :bool, default: true attr_reader :parser @@ -49,7 +50,9 @@ def filter_with_time(tag, time, record) raw_value = record[@key_name] if raw_value.nil? - router.emit_error_event(tag, time, record, ArgumentError.new("#{@key_name} does not exist")) + if @emit_invalid_record_to_error + router.emit_error_event(tag, time, record, ArgumentError.new("#{@key_name} does not exist")) + end if @reserve_data return time, handle_parsed(tag, record, time, {}) else @@ -67,7 +70,9 @@ r = handle_parsed(tag, record, t, values) return t, r else - router.emit_error_event(tag, time, record, Fluent::Plugin::Parser::ParserError.new("pattern not match with data '#{raw_value}'")) + if @emit_invalid_record_to_error + router.emit_error_event(tag, time, record, Fluent::Plugin::Parser::ParserError.new("pattern not match with data '#{raw_value}'")) + end if @reserve_data t = time r = handle_parsed(tag, record, time, {}) @@ -78,8 +83,11 @@ end end rescue Fluent::Plugin::Parser::ParserError => e - router.emit_error_event(tag, time, record, e) - return FAILED_RESULT + if @emit_invalid_record_to_error + raise e + else + return FAILED_RESULT + end rescue ArgumentError => e raise unless @replace_invalid_sequence raise unless e.message.index("invalid byte sequence in") == 0 @@ -87,8 +95,11 @@ raw_value = raw_value.scrub(REPLACE_CHAR) retry rescue => e - router.emit_error_event(tag, time, record, Fluent::Plugin::Parser::ParserError.new("parse failed #{e.message}")) - return FAILED_RESULT + if @emit_invalid_record_to_error + raise Fluent::Plugin::Parser::ParserError, "parse failed #{e.message}" + else + return FAILED_RESULT + end end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_http.rb new/lib/fluent/plugin/in_http.rb --- old/lib/fluent/plugin/in_http.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/in_http.rb 2017-03-24 05:16:16.000000000 +0100 @@ -373,6 +373,8 @@ params.update WEBrick::HTTPUtils.parse_form_data(@body, boundary) elsif @content_type =~ /^application\/json/ params['json'] = @body + elsif @content_type =~ /^application\/msgpack/ + params['msgpack'] = @body end path_info = uri.path diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_monitor_agent.rb new/lib/fluent/plugin/in_monitor_agent.rb --- old/lib/fluent/plugin/in_monitor_agent.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/in_monitor_agent.rb 2017-03-24 05:16:16.000000000 +0100 @@ -226,6 +226,11 @@ @first_warn = false end + def configure(conf) + super + @port += fluentd_worker_id + end + def multi_workers_ready? true end @@ -233,7 +238,7 @@ def start super - log.debug "listening monitoring http server on http://#{@bind}:#{@port}/api/plugins"; + log.debug "listening monitoring http server on http://#{@bind}:#{@port}/api/plugins for worker#{fluentd_worker_id}" @srv = WEBrick::HTTPServer.new({ BindAddress: @bind, Port: @port, diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_syslog.rb new/lib/fluent/plugin/in_syslog.rb --- old/lib/fluent/plugin/in_syslog.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/in_syslog.rb 2017-03-24 05:16:16.000000000 +0100 @@ -85,15 +85,16 @@ config_param :source_hostname_key, :string, default: nil desc 'The field name of source address of sender.' config_param :source_address_key, :string, default: nil - desc 'The field name of the priority.' config_param :priority_key, :string, default: nil desc 'The field name of the facility.' config_param :facility_key, :string, default: nil - config_param :blocking_timeout, :time, default: 0.5 + desc "The max bytes of message" config_param :message_length_limit, :size, default: 2048 + config_param :blocking_timeout, :time, default: 0.5 + config_section :parse do config_set_default :@type, DEFAULT_PARSER config_param :with_priority, :bool, default: true diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tail.rb new/lib/fluent/plugin/in_tail.rb --- old/lib/fluent/plugin/in_tail.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/in_tail.rb 2017-03-24 05:16:16.000000000 +0100 @@ -79,6 +79,8 @@ config_param :open_on_every_update, :bool, default: false desc 'Limit the watching files that the modification time is within the specified time range (when use \'*\' in path).' config_param :limit_recently_modified, :time, default: nil + desc 'Enable the option to skip the refresh of watching list on startup.' + config_param :skip_refresh_on_startup, :bool, default: false attr_reader :paths @@ -160,7 +162,7 @@ @pf = PositionFile.parse(@pf_file) end - refresh_watchers + refresh_watchers unless @skip_refresh_on_startup timer_execute(:in_tail_refresh_watchers, @refresh_interval, &method(:refresh_watchers)) end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_udp.rb new/lib/fluent/plugin/in_udp.rb --- old/lib/fluent/plugin/in_udp.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/in_udp.rb 2017-03-24 05:16:16.000000000 +0100 @@ -34,7 +34,10 @@ desc "The field name of the client's hostname." config_param :source_hostname_key, :string, default: nil - config_param :body_size_limit, :size, default: 4096 + desc "Deprecated parameter. Use message_length_limit instead" + config_param :body_size_limit, :size, default: nil, deprecated: "use message_length_limit instead." + desc "The max bytes of message" + config_param :message_length_limit, :size, default: 4096 config_param :blocking_timeout, :time, default: 0.5 @@ -43,6 +46,7 @@ super @_event_loop_blocking_timeout = @blocking_timeout @source_hostname_key ||= @source_host_key if @source_host_key + @message_length_limit = @body_size_limit if @body_size_limit @parser = parser_create end @@ -55,7 +59,7 @@ super log.info "listening udp socket", bind: @bind, port: @port - server_create(:in_udp_server, @port, proto: :udp, bind: @bind, max_bytes: @body_size_limit) do |data, sock| + server_create(:in_udp_server, @port, proto: :udp, bind: @bind, max_bytes: @message_length_limit) do |data, sock| data.chomp! begin @parser.parse(data) do |time, record| diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/output.rb new/lib/fluent/plugin/output.rb --- old/lib/fluent/plugin/output.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/output.rb 2017-03-24 05:16:16.000000000 +0100 @@ -589,13 +589,13 @@ example = @argument[:example] timekey = @argument[:timekey] if !sec && timekey - raise Fluent::ConfigError, "Parameter '#{name}' doesn't have timestamp placeholders for timekey #{timekey.to_i}" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' doesn't have timestamp placeholders for timekey #{timekey.to_i}" end if sec && !timekey - raise Fluent::ConfigError, "Parameter '#{name}' has timestamp placeholders, but chunk key 'time' is not configured" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' has timestamp placeholders, but chunk key 'time' is not configured" end if sec && timekey && timekey < sec - raise Fluent::ConfigError, "Parameter '#{@name}' doesn't have timestamp placeholder for #{title}('#{example}') for timekey #{timekey.to_i}" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' doesn't have timestamp placeholder for #{title}('#{example}') for timekey #{timekey.to_i}" end end @@ -603,10 +603,10 @@ parts = @argument[:parts] tagkey = @argument[:tagkey] if tagkey && parts.empty? - raise Fluent::ConfigError, "Parameter '#{@name}' doesn't have tag placeholder" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' doesn't have tag placeholder" end if !tagkey && !parts.empty? - raise Fluent::ConfigError, "Parameter '#{@name}' has tag placeholders, but chunk key 'tag' is not configured" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' has tag placeholders, but chunk key 'tag' is not configured" end end @@ -615,11 +615,11 @@ chunk_keys = @argument[:chunkkeys] if (chunk_keys - keys).size > 0 not_specified = (chunk_keys - keys).sort - raise Fluent::ConfigError, "Parameter '#{@name}' doesn't have enough placeholders for keys #{not_specified.join(',')}" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' doesn't have enough placeholders for keys #{not_specified.join(',')}" end if (keys - chunk_keys).size > 0 not_satisfied = (keys - chunk_keys).sort - raise Fluent::ConfigError, "Parameter '#{@name}' has placeholders, but chunk keys doesn't have keys #{not_satisfied.join(',')}" + raise Fluent::ConfigError, "Parameter '#{name}: #{string}' has placeholders, but chunk keys doesn't have keys #{not_satisfied.join(',')}" end end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/parser_apache2.rb new/lib/fluent/plugin/parser_apache2.rb --- old/lib/fluent/plugin/parser_apache2.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/parser_apache2.rb 2017-03-24 05:16:16.000000000 +0100 @@ -21,7 +21,7 @@ class Apache2Parser < Parser Plugin.register_parser('apache2', self) - REGEXP = /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$/ + REGEXP = /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")?$/ TIME_FORMAT = "%d/%b/%Y:%H:%M:%S %z" def initialize diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/parser_syslog.rb new/lib/fluent/plugin/parser_syslog.rb --- old/lib/fluent/plugin/parser_syslog.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin/parser_syslog.rb 2017-03-24 05:16:16.000000000 +0100 @@ -27,9 +27,13 @@ REGEXP = /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/ # From in_syslog default pattern REGEXP_WITH_PRI = /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/ + REGEXP_RFC5424 = /\A^\<(?<pri>[0-9]{1,3})\>[1-9]\d{0,2} (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|[^ ])) (?<message>.+)$\z/ + REGEXP_DETECT_RFC5424 = /^\<.*\>[1-9]\d{0,2}/ config_set_default :time_format, "%b %d %H:%M:%S" config_param :with_priority, :bool, default: false + config_param :message_format, :enum, list: [:rfc3164, :rfc5424, :auto], default: :rfc3164 + config_param :rfc5424_time_format, :string, default: "%Y-%m-%dT%H:%M:%S.%L%z" def initialize super @@ -39,7 +43,27 @@ def configure(conf) super - @regexp = @with_priority ? REGEXP_WITH_PRI : REGEXP + @time_parser_rfc3164 = @time_parser_rfc5424 = nil + @regexp = case @message_format + when :rfc3164 + class << self + alias_method :parse, :parse_plain + end + @with_priority ? REGEXP_WITH_PRI : REGEXP + when :rfc5424 + class << self + alias_method :parse, :parse_plain + end + @time_format = @rfc5424_time_format unless conf.has_key?('time_format') + REGEXP_RFC5424 + when :auto + class << self + alias_method :parse, :parse_auto + end + @time_parser_rfc3164 = time_parser_create(format: @time_format) + @time_parser_rfc5424 = time_parser_create(format: @rfc5424_time_format) + nil + end @time_parser = time_parser_create end @@ -48,6 +72,21 @@ end def parse(text) + # This is overwritten in configure + end + + def parse_auto(text, &block) + if REGEXP_DETECT_RFC5424.match(text) + @regexp = REGEXP_RFC5424 + @time_parser = @time_parser_rfc5424 + else + @regexp = @with_priority ? REGEXP_WITH_PRI : REGEXP + @time_parser = @time_parser_rfc3164 + end + parse_plain(text, &block) + end + + def parse_plain(text, &block) m = @regexp.match(text) unless m yield nil, nil diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin_helper/storage.rb new/lib/fluent/plugin_helper/storage.rb --- old/lib/fluent/plugin_helper/storage.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/plugin_helper/storage.rb 2017-03-24 05:16:16.000000000 +0100 @@ -30,7 +30,7 @@ StorageState = Struct.new(:storage, :running) def storage_create(usage: '', type: nil, conf: nil, default_type: nil) - if conf && !conf.arg.empty? + if conf && conf.respond_to?(:arg) && !conf.arg.empty? usage = conf.arg end if !usage.empty? && usage !~ /^[a-zA-Z][-_.a-zA-Z0-9]*$/ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/root_agent.rb new/lib/fluent/root_agent.rb --- old/lib/fluent/root_agent.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/root_agent.rb 2017-03-24 05:16:16.000000000 +0100 @@ -136,8 +136,12 @@ def start lifecycle(desc: true) do |i| # instance i.start unless i.started? - end - lifecycle(desc: true) do |i| + # Input#start sometimes emits lots of evetns with in_tail/`read_from_head true` case + # and it causes deadlock for small buffer/queue output. To avoid such problem, + # buffer related output threads should be run before `Input#start`. + # This is why after_start should be called immediately after start call. + # This depends on `desc: true` because calling plugin order of `desc: true` is + # Output, Filter, Label, Output with Router, then Input. i.after_start unless i.after_started? end end @@ -173,7 +177,7 @@ log.debug "calling #{method} on #{kind} plugin", type: Plugin.lookup_type_from_class(instance.class), plugin_id: instance.plugin_id instance.send(method) unless instance.send(checker) rescue Exception => e - log.warn "unexpected error while calling #{method} on #{kind} plugin", pluguin: instance.class, plugin_id: instance.plugin_id, error: e + log.warn "unexpected error while calling #{method} on #{kind} plugin", plugin: instance.class, plugin_id: instance.plugin_id, error: e log.warn_backtrace end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/supervisor.rb new/lib/fluent/supervisor.rb --- old/lib/fluent/supervisor.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/supervisor.rb 2017-03-24 05:16:16.000000000 +0100 @@ -61,6 +61,7 @@ def after_run stop_rpc_server if @rpc_endpoint + Fluent::Supervisor.cleanup_resources end def run_rpc_server @@ -387,6 +388,14 @@ } end + def self.cleanup_resources + unless Fluent.windows? + if ENV.has_key?('SERVERENGINE_SOCKETMANAGER_PATH') + FileUtils.rm_f(ENV['SERVERENGINE_SOCKETMANAGER_PATH']) + end + end + end + def initialize(opt) @daemonize = opt[:daemonize] @supervise = opt[:supervise] @@ -493,6 +502,7 @@ init_engine run_configure run_engine + self.class.cleanup_resources if @standalone_worker exit 0 end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/system_config.rb new/lib/fluent/system_config.rb --- old/lib/fluent/system_config.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/system_config.rb 2017-03-24 05:16:16.000000000 +0100 @@ -98,6 +98,13 @@ next # doesn't exist in command line options when :emit_error_log_interval system.emit_error_log_interval = @suppress_interval if @suppress_interval + when :log_level + ll_value = instance_variable_get("@log_level") + # info level can't be specified via command line option. + # log_level is info here, it is default value and <system>'s log_level should be applied if exists. + if ll_value != Fluent::Log::LEVEL_INFO + system.log_level = ll_value + end else next unless instance_variable_defined?("@#{param}") supervisor_value = instance_variable_get("@#{param}") diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/version.rb new/lib/fluent/version.rb --- old/lib/fluent/version.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/lib/fluent/version.rb 2017-03-24 05:16:16.000000000 +0100 @@ -16,6 +16,6 @@ module Fluent - VERSION = '0.14.13' + VERSION = '0.14.14' end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/metadata new/metadata --- old/metadata 2017-02-17 22:22:06.000000000 +0100 +++ new/metadata 2017-03-24 05:16:16.000000000 +0100 @@ -1,14 +1,14 @@ --- !ruby/object:Gem::Specification name: fluentd version: !ruby/object:Gem::Version - version: 0.14.13 + version: 0.14.14 platform: ruby authors: - Sadayuki Furuhashi autorequire: bindir: bin cert_chain: [] -date: 2017-02-17 00:00:00.000000000 Z +date: 2017-03-24 00:00:00.000000000 Z dependencies: - !ruby/object:Gem::Dependency name: msgpack diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_system_config.rb new/test/config/test_system_config.rb --- old/test/config/test_system_config.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/config/test_system_config.rb 2017-03-24 05:16:16.000000000 +0100 @@ -13,11 +13,13 @@ end class FakeSupervisor + attr_writer :log_level + def initialize @workers = nil @root_dir = nil @log = FakeLoggerInitializer.new - @log_level = nil + @log_level = Fluent::Log::LEVEL_INFO @suppress_interval = nil @suppress_config_dump = nil @suppress_repeated_stacktrace = nil @@ -55,7 +57,7 @@ assert_nil(sc.without_source) assert_equal(1, s.instance_variable_get(:@workers)) assert_nil(s.instance_variable_get(:@root_dir)) - assert_nil(s.instance_variable_get(:@log_level)) + assert_equal(Fluent::Log::LEVEL_INFO, s.instance_variable_get(:@log_level)) assert_nil(s.instance_variable_get(:@suppress_repeated_stacktrace)) assert_nil(s.instance_variable_get(:@emit_error_log_interval)) assert_nil(s.instance_variable_get(:@suppress_config_dump)) @@ -99,7 +101,7 @@ sc.apply(s) assert_equal(1, s.instance_variable_get(:@workers)) assert_nil(s.instance_variable_get(:@root_dir)) - assert_nil(s.instance_variable_get(:@log_level)) + assert_equal(Fluent::Log::LEVEL_INFO, s.instance_variable_get(:@log_level)) assert_nil(s.instance_variable_get(:@suppress_repeated_stacktrace)) assert_nil(s.instance_variable_get(:@emit_error_log_interval)) assert_nil(s.instance_variable_get(:@suppress_config_dump)) @@ -109,16 +111,43 @@ assert_nil(s.instance_variable_get(:@dir_permission)) end - test 'log_level' do + data('trace' => Fluent::Log::LEVEL_TRACE, + 'debug' => Fluent::Log::LEVEL_DEBUG, + 'info' => Fluent::Log::LEVEL_INFO, + 'warn' => Fluent::Log::LEVEL_WARN, + 'error' => Fluent::Log::LEVEL_ERROR, + 'fatal' => Fluent::Log::LEVEL_FATAL) + test 'log_level is applied when log_level related command line option is not passed' do |level| + conf = parse_text(<<-EOS) + <system> + log_level #{Fluent::Log::LEVEL_TEXT[level]} + </system> + EOS + s = FakeSupervisor.new + sc = Fluent::SystemConfig.new(conf) + sc.attach(s) + sc.apply(s) + assert_equal(level, s.instance_variable_get("@log").level) + end + + # info is removed because info level can't be specified via command line + data('trace' => Fluent::Log::LEVEL_TRACE, + 'debug' => Fluent::Log::LEVEL_DEBUG, + 'warn' => Fluent::Log::LEVEL_WARN, + 'error' => Fluent::Log::LEVEL_ERROR, + 'fatal' => Fluent::Log::LEVEL_FATAL) + test 'log_level is ignored when log_level related command line option is passed' do |level| conf = parse_text(<<-EOS) <system> - log_level warn + log_level info </system> EOS s = FakeSupervisor.new + s.log_level = level sc = Fluent::SystemConfig.new(conf) + sc.attach(s) sc.apply(s) - assert_equal(Fluent::Log::LEVEL_WARN, s.instance_variable_get("@log").level) + assert_equal(level, s.instance_variable_get("@log").level) end test 'process global overridable variables' do @@ -130,6 +159,7 @@ EOS s = FakeSupervisor.new sc = Fluent::SystemConfig.new(conf) + sc.attach(s) sc.apply(s) assert_equal(0655, s.instance_variable_get(:@file_permission)) assert_equal(0765, s.instance_variable_get(:@dir_permission)) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/config/test_types.rb new/test/config/test_types.rb --- old/test/config/test_types.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/config/test_types.rb 2017-03-24 05:16:16.000000000 +0100 @@ -69,6 +69,17 @@ assert_equal Encoding::UTF_8, Config::STRING_TYPE.call('test', {}).encoding end + data('latin' => 'Märch', + 'ascii' => 'ascii', + 'space' => ' ', + 'number' => '1', + 'Hiragana' => 'あいうえお') + test 'string w/ binary' do |str| + actual = Config::STRING_TYPE.call(str.b, {}) + assert_equal str, actual + assert_equal Encoding::UTF_8, actual.encoding + end + test 'enum' do assert_equal :val, Config::ENUM_TYPE.call('val', {list: [:val, :value, :v]}) assert_equal :v, Config::ENUM_TYPE.call('v', {list: [:val, :value, :v]}) @@ -143,6 +154,14 @@ assert_raise(RuntimeError.new("unknown type in REFORMAT: foo")){ Config::HASH_TYPE.call("x:1,y:2", {value_type: :foo}) } end + data('latin' => ['3:Märch', {"3"=>"Märch"}], + 'ascii' => ['ascii:ascii', {"ascii"=>"ascii"}], + 'number' => ['number:1', {"number"=>"1"}], + 'Hiragana' => ['hiragana:あいうえお', {"hiragana"=>"あいうえお"}]) + test 'hash w/ binary' do |(target, expected)| + assert_equal(expected, Config::HASH_TYPE.call(target.b, { value_type: :string })) + end + test 'array' do assert_equal(["1","2",1], Config::ARRAY_TYPE.call('["1","2",1]', {})) assert_equal(["1","2","1"], Config::ARRAY_TYPE.call('1,2,1', {})) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_filter_parser.rb new/test/plugin/test_filter_parser.rb --- old/test/plugin/test_filter_parser.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_filter_parser.rb 2017-03-24 05:16:16.000000000 +0100 @@ -662,4 +662,39 @@ end end end + + class EmitInvalidRecordToErrorTest < self + def test_pattern_is_mismached_with_emit_invalid_record_to_error + d = create_driver(CONFIG_UNMATCHED_PATTERN_LOG + "emit_invalid_record_to_error false") + flexmock(d.instance.router).should_receive(:emit_error_event).never + assert_nothing_raised { + d.run do + d.feed(@tag, Fluent::EventTime.now.to_i, {'message' => INVALID_MESSAGE}) + end + } + assert_equal 0, d.filtered.length + end + + def test_parser_error_with_emit_invalid_record_to_error + d = create_driver(CONFIG_INVALID_TIME_VALUE + "emit_invalid_record_to_error false") + flexmock(d.instance.router).should_receive(:emit_error_event).never + assert_nothing_raised { + d.run do + d.feed(@tag, Fluent::EventTime.now.to_i, {'data' => '{"time":[], "f1":"v1"}'}) + end + } + assert_equal 0, d.filtered.length + end + + def test_key_not_exist_with_emit_invalid_record_to_error + d = create_driver(CONFIG_NOT_IGNORE + "emit_invalid_record_to_error false") + flexmock(d.instance.router).should_receive(:emit_error_event).never + assert_nothing_raised { + d.run do + d.feed(@tag, Fluent::EventTime.now.to_i, {'foo' => 'bar'}) + end + } + assert_equal 0, d.filtered.length + end + end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_http.rb new/test/plugin/test_in_http.rb --- old/test/plugin/test_in_http.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_in_http.rb 2017-03-24 05:16:16.000000000 +0100 @@ -313,6 +313,28 @@ assert_equal_event_time time, d.events[1][1] end + def test_application_msgpack + d = create_driver + time = event_time("2011-01-02 13:14:15 UTC") + time_i = time.to_i + events = [ + ["tag1", time, {"a"=>1}], + ["tag2", time, {"a"=>2}], + ] + res_codes = [] + + d.run(expect_records: 2) do + events.each do |tag, t, record| + res = post("/#{tag}?time=#{time_i.to_s}", record.to_msgpack, {"Content-Type"=>"application/msgpack"}) + res_codes << res.code + end + end + assert_equal ["200", "200"], res_codes + assert_equal events, d.events + assert_equal_event_time time, d.events[0][1] + assert_equal_event_time time, d.events[1][1] + end + def test_msgpack d = create_driver time = event_time("2011-01-02 13:14:15 UTC") diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_monitor_agent.rb new/test/plugin/test_in_monitor_agent.rb --- old/test/plugin/test_in_monitor_agent.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_in_monitor_agent.rb 2017-03-24 05:16:16.000000000 +0100 @@ -204,10 +204,9 @@ unless header.has_key?('Content-Type') header['Content-Type'] = 'application/octet-stream' end - res = Net::HTTP.start(url.host, url.port) {|http| + Net::HTTP.start(url.host, url.port) {|http| http.request(req) } - res.body end sub_test_case "servlets" do @@ -268,7 +267,7 @@ expected_test_filter_response = "\ plugin_id:test_filter\tplugin_category:filter\ttype:test_filter\toutput_plugin:false\tretry_count:" - response = get("http://127.0.0.1:#{@port}/api/plugins";) + response = get("http://127.0.0.1:#{@port}/api/plugins";).body test_in = response.split("\n")[0] test_filter = response.split("\n")[3] assert_equal(expected_test_in_response, test_in) @@ -306,7 +305,7 @@ } expected_null_response.merge!("config" => {"@id" => "null", "@type" => "null"}) if with_config expected_null_response.merge!("retry" => {}) if with_retry - response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json";)) + response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json";).body) test_in_response = response["plugins"][0] null_response = response["plugins"][5] assert_equal(expected_test_in_response, test_in_response) @@ -343,7 +342,7 @@ } expected_null_response.merge!("config" => {"@id" => "null", "@type" => "null"}) if with_config expected_null_response.merge!("retry" => {}) if with_retry - response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json#{query_param}";)) + response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json#{query_param}";).body) test_in_response = response["plugins"][0] null_response = response["plugins"][5] assert_equal(expected_test_in_response, test_in_response) @@ -376,7 +375,7 @@ "type" => "null", "instance_variables" => {"id" => "null", "num_errors" => 0} } - response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json?with_config=no&with_retry=no&with_ivars=id,num_errors";)) + response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json?with_config=no&with_retry=no&with_ivars=id,num_errors";).body) test_in_response = response["plugins"][0] null_response = response["plugins"][5] assert_equal(expected_test_in_response, test_in_response) @@ -394,7 +393,7 @@ expected_response_regex = /pid:\d+\tppid:\d+\tconfig_path:\/etc\/fluent\/fluent.conf\tpid_file:\tplugin_dirs:\[\"\/etc\/fluent\/plugin\"\]\tlog_path:/ assert_match(expected_response_regex, - get("http://127.0.0.1:#{@port}/api/config";)) + get("http://127.0.0.1:#{@port}/api/config";).body) end test "/api/config.json" do @@ -405,7 +404,7 @@ tag monitor ") d.instance.start - res = JSON.parse(get("http://127.0.0.1:#{@port}/api/config.json";)) + res = JSON.parse(get("http://127.0.0.1:#{@port}/api/config.json";).body) assert_equal("/etc/fluent/fluent.conf", res["config_path"]) assert_nil(res["pid_file"]) assert_equal(["/etc/fluent/plugin"], res["plugin_dirs"]) @@ -474,7 +473,7 @@ output.submit_flush_once sleep 0.1 until output.buffer.queued? end - response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json";)) + response = JSON.parse(get("http://127.0.0.1:#{@port}/api/plugins.json";).body) test_out_fail_write_response = response["plugins"][1] # remove dynamic keys response_retry_count = test_out_fail_write_response.delete("retry_count") @@ -486,4 +485,32 @@ assert{ response_retry_count == response_retry["steps"] + 1 } end end + + sub_test_case "check the port number of http server" do + test "on single worker environment" do + port = unused_port + d = create_driver(" + @type monitor_agent + bind '127.0.0.1' + port #{port} +") + d.instance.start + assert_equal("200", get("http://127.0.0.1:#{port}/api/plugins";).code) + end + + test "worker_id = 2 on multi worker environment" do + port = unused_port + Fluent::SystemConfig.overwrite_system_config('workers' => 4) do + d = Fluent::Test::Driver::Input.new(Fluent::Plugin::MonitorAgentInput) + d.instance.instance_eval{ @_fluentd_worker_id = 2 } + d.configure(" + @type monitor_agent + bind '127.0.0.1' + port #{port - 2} +") + d.instance.start + end + assert_equal("200", get("http://127.0.0.1:#{port}/api/plugins";).code) + end + end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_tail.rb new/test/plugin/test_in_tail.rb --- old/test/plugin/test_in_tail.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_in_tail.rb 2017-03-24 05:16:16.000000000 +0100 @@ -1198,4 +1198,19 @@ assert_equal expected_files, plugin.expand_paths.sort end end + + def test_skip_refresh_on_startup + FileUtils.touch("#{TMP_DIR}/tail.txt") + config = config_element('', '', { + 'format' => 'none', + 'refresh_interval' => 1, + 'skip_refresh_on_startup' => true + }) + d = create_driver(config) + d.run(shutdown: false) {} + assert_equal 0, d.instance.instance_variable_get(:@tails).keys.size + # detect a file at first execution of in_tail_refresh_watchers timer + waiting(5) { sleep 0.1 until d.instance.instance_variable_get(:@tails).keys.size == 1 } + d.instance_shutdown + end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_udp.rb new/test/plugin/test_in_udp.rb --- old/test/plugin/test_in_udp.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_in_udp.rb 2017-03-24 05:16:16.000000000 +0100 @@ -54,7 +54,7 @@ d = create_driver(conf) assert_equal PORT, d.instance.port assert_equal bind, d.instance.bind - assert_equal 4096, d.instance.body_size_limit + assert_equal 4096, d.instance.message_length_limit end data( @@ -87,6 +87,16 @@ end data( + 'message_length_limit' => 'message_length_limit 2048', + 'body_size_limit' => 'body_size_limit 2048' + ) + test 'message_length_limit/body_size_limit compatibility' do |param| + + d = create_driver(CONFIG + param) + assert_equal 2048, d.instance.message_length_limit + end + + data( 'none' => { 'format' => 'none', 'payloads' => ["tcptest1\n", "tcptest2\n"], diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_output.rb new/test/plugin/test_output.rb --- old/test/plugin/test_output.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_output.rb 2017-03-24 05:16:16.000000000 +0100 @@ -315,7 +315,7 @@ validators = @i.placeholder_validators(:path, "/my/path/file.%Y-%m-%d.log") assert_equal 1, validators.size assert_equal 1, validators.select(&:time?).size - assert_raise Fluent::ConfigError.new("Parameter 'path' has timestamp placeholders, but chunk key 'time' is not configured") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.%Y-%m-%d.log' has timestamp placeholders, but chunk key 'time' is not configured") do validators.first.validate! end end @@ -325,7 +325,7 @@ validators = @i.placeholder_validators(:path, "/my/path/to/file.log") assert_equal 1, validators.size assert_equal 1, validators.select(&:time?).size - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have timestamp placeholders for timekey 30") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/to/file.log' doesn't have timestamp placeholders for timekey 30") do validators.first.validate! end end @@ -335,7 +335,7 @@ validators = @i.placeholder_validators(:path, "/my/path/${tag}/file.log") assert_equal 1, validators.size assert_equal 1, validators.select(&:tag?).size - assert_raise Fluent::ConfigError.new("Parameter 'path' has tag placeholders, but chunk key 'tag' is not configured") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/${tag}/file.log' has tag placeholders, but chunk key 'tag' is not configured") do validators.first.validate! end end @@ -345,7 +345,7 @@ validators = @i.placeholder_validators(:path, "/my/path/file.log") assert_equal 1, validators.size assert_equal 1, validators.select(&:tag?).size - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have tag placeholder") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.log' doesn't have tag placeholder") do validators.first.validate! end end @@ -355,7 +355,7 @@ validators = @i.placeholder_validators(:path, "/my/path/${username}/file.${group}.log") assert_equal 1, validators.size assert_equal 1, validators.select(&:keys?).size - assert_raise Fluent::ConfigError.new("Parameter 'path' has placeholders, but chunk keys doesn't have keys group,username") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/${username}/file.${group}.log' has placeholders, but chunk keys doesn't have keys group,username") do validators.first.validate! end end @@ -365,7 +365,7 @@ validators = @i.placeholder_validators(:path, "/my/path/file.log") assert_equal 1, validators.size assert_equal 1, validators.select(&:keys?).size - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have enough placeholders for keys group,username") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.log' doesn't have enough placeholders for keys group,username") do validators.first.validate! end end @@ -374,14 +374,14 @@ sub_test_case '#placeholder_validate!' do test 'raises configuration error for a templace when timestamp placeholders exist but time key is missing' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', '')])) - assert_raise Fluent::ConfigError.new("Parameter 'path' has timestamp placeholders, but chunk key 'time' is not configured") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /path/without/timestamp/file.%Y%m%d-%H%M.log' has timestamp placeholders, but chunk key 'time' is not configured") do @i.placeholder_validate!(:path, "/path/without/timestamp/file.%Y%m%d-%H%M.log") end end test 'raises configuration error for a template without timestamp placeholders when timekey is configured' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'time', {"timekey" => 180})])) - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have timestamp placeholders for timekey 180") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.log' doesn't have timestamp placeholders for timekey 180") do @i.placeholder_validate!(:path, "/my/path/file.log") end assert_nothing_raised do @@ -391,7 +391,7 @@ test 'raises configuration error for a template with timestamp placeholders when plugin is configured more fine timekey' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'time', {"timekey" => 180})])) - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have timestamp placeholder for hour('%H') for timekey 180") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.%Y%m%d_%H.log' doesn't have timestamp placeholder for hour('%H') for timekey 180") do @i.placeholder_validate!(:path, "/my/path/file.%Y%m%d_%H.log") end assert_nothing_raised do @@ -401,14 +401,14 @@ test 'raises configuration error for a template when tag placeholders exist but tag key is missing' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', '')])) - assert_raise Fluent::ConfigError.new("Parameter 'path' has tag placeholders, but chunk key 'tag' is not configured") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/${tag}/file.${tag[2]}.log' has tag placeholders, but chunk key 'tag' is not configured") do @i.placeholder_validate!(:path, "/my/path/${tag}/file.${tag[2]}.log") end end test 'raises configuration error for a template without tag placeholders when tagkey is configured' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'tag')])) - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have tag placeholder") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.log' doesn't have tag placeholder") do @i.placeholder_validate!(:path, "/my/path/file.log") end assert_nothing_raised do @@ -418,14 +418,14 @@ test 'raises configuration error for a template when variable key placeholders exist but chunk keys are missing' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', '')])) - assert_raise Fluent::ConfigError.new("Parameter 'path' has placeholders, but chunk keys doesn't have keys service,username") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/${service}/file.${username}.log' has placeholders, but chunk keys doesn't have keys service,username") do @i.placeholder_validate!(:path, "/my/path/${service}/file.${username}.log") end end test 'raises configuration error for a template without variable key placeholders when chunk keys are configured' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'username,service')])) - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have enough placeholders for keys service,username") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.log' doesn't have enough placeholders for keys service,username") do @i.placeholder_validate!(:path, "/my/path/file.log") end assert_nothing_raised do @@ -435,10 +435,10 @@ test 'raise configuration error for a template and configuration with keys mismatch' do @i.configure(config_element('ROOT', '', {}, [config_element('buffer', 'username,service')])) - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have enough placeholders for keys service") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/file.${username}.log' doesn't have enough placeholders for keys service") do @i.placeholder_validate!(:path, "/my/path/file.${username}.log") end - assert_raise Fluent::ConfigError.new("Parameter 'path' doesn't have enough placeholders for keys username") do + assert_raise Fluent::ConfigError.new("Parameter 'path: /my/path/${service}/file.log' doesn't have enough placeholders for keys username") do @i.placeholder_validate!(:path, "/my/path/${service}/file.log") end assert_nothing_raised do diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_parser_apache2.rb new/test/plugin/test_parser_apache2.rb --- old/test/plugin/test_parser_apache2.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_parser_apache2.rb 2017-03-24 05:16:16.000000000 +0100 @@ -35,4 +35,12 @@ assert_equal(@expected, record) } end + + def test_parse_with_escape_sequence + @parser.instance.parse('192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET /\" HTTP/1.1" 200 777 "referer \\\ \"" "user agent \\\ \""') { |_, record| + assert_equal('/\"', record['path']) + assert_equal('referer \\\ \"', record['referer']) + assert_equal('user agent \\\ \"', record['agent']) + } + end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_parser_syslog.rb new/test/plugin/test_parser_syslog.rb --- old/test/plugin/test_parser_syslog.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin/test_parser_syslog.rb 2017-03-24 05:16:16.000000000 +0100 @@ -63,4 +63,180 @@ assert_equal "Feb 28 00:00:12", record['time'] end end + + class TestRFC5424Regexp < self + def test_parse_with_rfc5424_message + @parser.configure( + 'time_format' => '%Y-%m-%dT%H:%M:%S.%L%z', + 'message_format' => 'rfc5424', + ) + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "-", record["pid"] + assert_equal "-", record["msgid"] + assert_equal "-", record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + + def test_parse_with_rfc5424_message_without_time_format + @parser.configure( + 'message_format' => 'rfc5424', + ) + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "-", record["pid"] + assert_equal "-", record["msgid"] + assert_equal "-", record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + + def test_parse_with_rfc5424_structured_message + @parser.configure( + 'time_format' => '%Y-%m-%dT%H:%M:%S.%L%z', + 'message_format' => 'rfc5424', + ) + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) +assert_equal "11111", record["pid"] + assert_equal "ID24224", record["msgid"] + assert_equal "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]", + record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + end + + class TestAutoRegexp < self + def test_auto_with_legacy_syslog_message + @parser.configure( + 'time_format' => '%b %d %M:%S:%H', + 'mseeage_format' => 'auto', + ) + text = 'Feb 28 00:00:12 192.168.0.1 fluentd[11111]: [error] Syslog test' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("Feb 28 00:00:12", format: '%b %d %M:%S:%H'), time) + assert_equal(@expected, record) + end + end + + def test_auto_with_legacy_syslog_priority_message + @parser.configure( + 'time_format' => '%b %d %M:%S:%H', + 'with_priority' => true, + 'mseeage_format' => 'auto', + ) + text = '<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("Feb 28 12:00:00", format: '%b %d %M:%S:%H'), time) + assert_equal(@expected.merge('pri' => 6), record) + end + end + + def test_parse_with_rfc5424_message + @parser.configure( + 'time_format' => '%Y-%m-%dT%H:%M:%S.%L%z', + 'message_format' => 'auto', + ) + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "-", record["pid"] + assert_equal "-", record["msgid"] + assert_equal "-", record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + + def test_parse_with_rfc5424_structured_message + @parser.configure( + 'time_format' => '%Y-%m-%dT%H:%M:%S.%L%z', + 'message_format' => 'auto', + ) + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "11111", record["pid"] + assert_equal "ID24224", record["msgid"] + assert_equal "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]", + record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + + def test_parse_with_both_message_type + @parser.configure( + 'time_format' => '%b %d %M:%S:%H', + 'rfc5424_time_format' => '%Y-%m-%dT%H:%M:%S.%L%z', + 'message_format' => 'auto', + ) + text = 'Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("Feb 28 12:00:00", format: '%b %d %M:%S:%H'), time) + assert_equal(@expected, record) + end + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "11111", record["pid"] + assert_equal "ID24224", record["msgid"] + assert_equal "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]", + record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + text = 'Feb 28 12:00:02 192.168.0.1 fluentd[11111]: [error] Syslog test' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("Feb 28 12:00:02", format: '%b %d %M:%S:%H'), time) + assert_equal(@expected, record) + end + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "-", record["pid"] + assert_equal "-", record["msgid"] + assert_equal "-", record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + + def test_parse_with_both_message_type_and_priority + @parser.configure( + 'time_format' => '%b %d %M:%S:%H', + 'rfc5424_time_format' => '%Y-%m-%dT%H:%M:%S.%L%z', + 'with_priority' => true, + 'message_format' => 'auto', + ) + text = '<6>Feb 28 12:00:00 192.168.0.1 fluentd[11111]: [error] Syslog test' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("Feb 28 12:00:00", format: '%b %d %M:%S:%H'), time) + assert_equal(@expected.merge('pri' => 6), record) + end + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd 11111 ID24224 [exampleSDID@20224 iut="3" eventSource="Application" eventID="11211"] Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "11111", record["pid"] + assert_equal "ID24224", record["msgid"] + assert_equal "[exampleSDID@20224 iut=\"3\" eventSource=\"Application\" eventID=\"11211\"]", + record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + text = '<16>Feb 28 12:00:02 192.168.0.1 fluentd[11111]: [error] Syslog test' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("Feb 28 12:00:02", format: '%b %d %M:%S:%H'), time) + assert_equal(@expected.merge('pri' => 16), record) + end + text = '<16>1 2017-02-06T13:14:15.003Z 192.168.0.1 fluentd - - - Hi, from Fluentd!' + @parser.instance.parse(text) do |time, record| + assert_equal(event_time("2017-02-06T13:14:15.003Z", format: '%Y-%m-%dT%H:%M:%S.%L%z'), time) + assert_equal "-", record["pid"] + assert_equal "-", record["msgid"] + assert_equal "-", record["extradata"] + assert_equal "Hi, from Fluentd!", record["message"] + end + end + end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin_helper/test_storage.rb new/test/plugin_helper/test_storage.rb --- old/test/plugin_helper/test_storage.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/plugin_helper/test_storage.rb 2017-03-24 05:16:16.000000000 +0100 @@ -99,6 +99,15 @@ assert_equal 0, d._storages.size end + test 'can be configured with hash' do + d = Dummy.new + d.configure(config_element()) + conf = { '@type' => 'example' } + assert_nothing_raised do + d.storage_create(conf: conf) + end + end + test 'can override default configuration parameters, but not overwrite whole definition' do d = Dummy.new d.configure(config_element()) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_log.rb new/test/test_log.rb --- old/test/test_log.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/test_log.rb 2017-03-24 05:16:16.000000000 +0100 @@ -720,6 +720,10 @@ @log.write("log") end + def test_write_alias + assert(@log.respond_to?(:<<)) + end + def test_out assert_equal(@log.out, @logger.out) @log.out = Object.new diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_plugin_classes.rb new/test/test_plugin_classes.rb --- old/test/test_plugin_classes.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/test_plugin_classes.rb 2017-03-24 05:16:16.000000000 +0100 @@ -21,6 +21,28 @@ end end + class FluentTestGenInput < ::Fluent::Plugin::Input + ::Fluent::Plugin.register_input('test_in_gen', self) + + attr_reader :started + + config_param :num, :integer, default: 10000 + + def start + super + @started = true + + @num.times { |i| + router.emit("test.evet", Fluent::EventTime.now, {'message' => 'Hello!', 'key' => "value#{i}", 'num' => i}) + } + end + + def shutdown + @started = false + super + end + end + class FluentTestOutput < ::Fluent::Plugin::Output ::Fluent::Plugin.register_output('test_out', self) @@ -112,6 +134,19 @@ class FluentTestBufferedOutput < ::Fluent::Plugin::Output ::Fluent::Plugin.register_output('test_out_buffered', self) + + attr_reader :started + + def start + super + @started = true + end + + def shutdown + @started = false + super + end + def write(chunk) # drop everything end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_root_agent.rb new/test/test_root_agent.rb --- old/test/test_root_agent.rb 2017-02-17 22:22:06.000000000 +0100 +++ new/test/test_root_agent.rb 2017-03-24 05:16:16.000000000 +0100 @@ -174,10 +174,15 @@ end sub_test_case 'start/shutdown' do - setup do - @ra = RootAgent.new(log: $log) - stub(Engine).root_agent { @ra } - @ra.configure(Config.parse(<<-EOC, "(test)", "(test_dir)", true)) + def setup_root_agent(conf) + ra = RootAgent.new(log: $log) + stub(Engine).root_agent { ra } + ra.configure(Config.parse(conf, "(test)", "(test_dir)", true)) + ra + end + + test 'plugin status' do + ra = setup_root_agent(<<-EOC) <source> @type test_in @id test_in @@ -191,19 +196,41 @@ @id test_out </match> EOC - @ra + ra.start + assert_true ra.inputs.first.started + assert_true ra.filters.first.started + assert_true ra.outputs.first.started + + ra.shutdown + assert_false ra.inputs.first.started + assert_false ra.filters.first.started + assert_false ra.outputs.first.started end - test 'plugin status' do - @ra.start - assert_true @ra.inputs.first.started - assert_true @ra.filters.first.started - assert_true @ra.outputs.first.started - - @ra.shutdown - assert_false @ra.inputs.first.started - assert_false @ra.filters.first.started - assert_false @ra.outputs.first.started + test 'output plugin threads should run before input plugin is blocked with buffer full' do + ra = setup_root_agent(<<-EOC) +<source> + @type test_in_gen + @id test_in_gen +</source> +<match **> + @type test_out_buffered + @id test_out_buffered + <buffer> + chunk_limit_size 1k + queue_limit_length 2 + flush_thread_count 2 + overflow_action block + </buffer> +</match> +EOC + waiting(5) { ra.start } + assert_true ra.inputs.first.started + assert_true ra.outputs.first.started + + ra.shutdown + assert_false ra.inputs.first.started + assert_false ra.outputs.first.started end end
