Hello community, here is the log from the commit of package rubygem-fluentd for openSUSE:Factory checked in at 2019-06-19 20:59:46 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/rubygem-fluentd (Old) and /work/SRC/openSUSE:Factory/.rubygem-fluentd.new.4811 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "rubygem-fluentd" Wed Jun 19 20:59:46 2019 rev:11 rq:705989 version:1.4.2 Changes: -------- --- /work/SRC/openSUSE:Factory/rubygem-fluentd/rubygem-fluentd.changes 2019-04-01 12:37:12.805881247 +0200 +++ /work/SRC/openSUSE:Factory/.rubygem-fluentd.new.4811/rubygem-fluentd.changes 2019-06-19 20:59:48.658056900 +0200 @@ -1,0 +2,30 @@ +Sun May 5 09:25:46 UTC 2019 - Stephan Kulow <co...@suse.com> + +- updated to version 1.4.2 + see installed CHANGELOG.md + + ## Release v1.4.2 - 2019/04/02 + + ### Enhancements + + * in_http: subdomain support in CORS domain + https://github.com/fluent/fluentd/pull/2337 + * in_monitor_agent: Expose current timekey list as a buffer metrics + https://github.com/fluent/fluentd/pull/2343 + * in_tcp/in_udp: Add source_address_key parameter + https://github.com/fluent/fluentd/pull/2347 + * in_forward: Add send_keepalive_packet parameter to check the remote connection is available or not + https://github.com/fluent/fluentd/pull/2352 + + ### Bug fixes + + * out_exec_filter: Fix typo of child_respawn description + https://github.com/fluent/fluentd/pull/2341 + * in_tail: Create parent directories for symlink + https://github.com/fluent/fluentd/pull/2353 + * in_tail: Fix encoding duplication check for non-specified case + https://github.com/fluent/fluentd/pull/2361 + * log: Fix time format handling of plugin logger when log format is JSON + https://github.com/fluent/fluentd/pull/2356 + +------------------------------------------------------------------- Old: ---- fluentd-1.4.1.gem New: ---- fluentd-1.4.2.gem ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ rubygem-fluentd.spec ++++++ --- /var/tmp/diff_new_pack.5OTmVw/_old 2019-06-19 20:59:49.346057516 +0200 +++ /var/tmp/diff_new_pack.5OTmVw/_new 2019-06-19 20:59:49.346057516 +0200 @@ -24,7 +24,7 @@ # Name: rubygem-fluentd -Version: 1.4.1 +Version: 1.4.2 Release: 0 %define mod_name fluentd %define mod_full_name %{mod_name}-%{version} ++++++ fluentd-1.4.1.gem -> fluentd-1.4.2.gem ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/CHANGELOG.md new/CHANGELOG.md --- old/CHANGELOG.md 2019-03-19 14:48:00.000000000 +0100 +++ new/CHANGELOG.md 2019-04-02 17:04:49.000000000 +0200 @@ -1,13 +1,35 @@ # v1.4 +## Release v1.4.2 - 2019/04/02 + +### Enhancements + +* in_http: subdomain support in CORS domain + https://github.com/fluent/fluentd/pull/2337 +* in_monitor_agent: Expose current timekey list as a buffer metrics + https://github.com/fluent/fluentd/pull/2343 +* in_tcp/in_udp: Add source_address_key parameter + https://github.com/fluent/fluentd/pull/2347 +* in_forward: Add send_keepalive_packet parameter to check the remote connection is available or not + https://github.com/fluent/fluentd/pull/2352 + +### Bug fixes + +* out_exec_filter: Fix typo of child_respawn description + https://github.com/fluent/fluentd/pull/2341 +* in_tail: Create parent directories for symlink + https://github.com/fluent/fluentd/pull/2353 +* in_tail: Fix encoding duplication check for non-specified case + https://github.com/fluent/fluentd/pull/2361 +* log: Fix time format handling of plugin logger when log format is JSON + https://github.com/fluent/fluentd/pull/2356 + ## Release v1.4.1 - 2019/03/18 ### Enhancements * system: Add worker_id to process_name when workers is larger than 1 https://github.com/fluent/fluentd/pull/2321 -* in_syslog: Check message length when read from buffer in octet counting - https://github.com/fluent/fluentd/pull/2323 * parser_regexp: Check named captures. When no named captures, configuration error is raised https://github.com/fluent/fluentd/pull/2331 @@ -15,6 +37,8 @@ * out_forward: Make tls_client_private_key_passphrase secret https://github.com/fluent/fluentd/pull/2324 +* in_syslog: Check message length when read from buffer in octet counting + https://github.com/fluent/fluentd/pull/2323 ## Release v1.4.0 - 2019/02/24 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/README.md new/README.md --- old/README.md 2019-03-19 14:48:00.000000000 +0100 +++ new/README.md 2019-04-02 17:04:49.000000000 +0200 @@ -53,11 +53,6 @@ $ bundle exec rake test TEST=test/test_specified_path.rb $ bundle exec rake test TEST=test/test_*.rb -## Running in Production - -Many enterprises run Fluentd in production to handle all of their logging needs. For enterprises requiring Security tested binaries, SLA-based support, architectural guidance, and enhanced plugin connectors see [Fluentd Enterprise](https://www.fluentd.org/enterprise). - - ## Fluentd UI: Admin GUI [Fluentd UI](https://github.com/fluent/fluentd-ui) is a graphical user interface to start/stop/configure Fluentd. Binary files old/checksums.yaml.gz and new/checksums.yaml.gz differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/log.rb new/lib/fluent/log.rb --- old/lib/fluent/log.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/log.rb 2019-04-02 17:04:49.000000000 +0200 @@ -532,6 +532,7 @@ end self.format = @logger.format + self.time_format = @logger.time_format enable_color @logger.enable_color? end @@ -540,6 +541,7 @@ end alias orig_format= format= + alias orig_time_format= time_format= alias orig_enable_color enable_color def format=(fmt) @@ -547,6 +549,11 @@ @logger.format = fmt end + def time_format=(fmt) + self.orig_time_format = fmt + @logger.time_format = fmt + end + def enable_color(b = true) orig_enable_color b @logger.enable_color b @@ -554,9 +561,9 @@ extend Forwardable def_delegators '@logger', :get_worker_id, :enable_color?, :enable_debug, :enable_event, - :disable_events, :log_event_enabled, :log_event_enabled=, :time_format, :time_format=, - :time_formatter, :time_formatter=, :event, :caller_line, :puts, :write, :<<, :flush, - :reset, :out, :out=, :optional_header, :optional_header=, :optional_attrs, :optional_attrs= + :disable_events, :log_event_enabled, :log_event_enabled=, :event, :caller_line, :puts, :write, + :<<, :flush, :reset, :out, :out=, :optional_header, :optional_header=, :optional_attrs, + :optional_attrs= end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/buf_file.rb new/lib/fluent/plugin/buf_file.rb --- old/lib/fluent/plugin/buf_file.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/buf_file.rb 2019-04-02 17:04:49.000000000 +0200 @@ -133,6 +133,8 @@ Dir.glob(patterns) do |path| next unless File.file?(path) + log.debug { "restoring buffer file: path = #{path}" } + m = new_metadata() # this metadata will be overwritten by resuming .meta file content # so it should not added into @metadata_list for now mode = Fluent::Plugin::Buffer::FileChunk.assume_chunk_state(path) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/buffer.rb new/lib/fluent/plugin/buffer.rb --- old/lib/fluent/plugin/buffer.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/buffer.rb 2019-04-02 17:04:49.000000000 +0200 @@ -154,6 +154,7 @@ @dequeued_num = {} # metadata => int (number of dequeued chunks) @stage_size = @queue_size = 0 + @timekeys = Hash.new(0) @metadata_list = [] # keys of @stage end @@ -176,12 +177,14 @@ @stage.each_pair do |metadata, chunk| @metadata_list << metadata unless @metadata_list.include?(metadata) @stage_size += chunk.bytesize + add_timekey(metadata) end @queue.each do |chunk| @metadata_list << chunk.metadata unless @metadata_list.include?(chunk.metadata) @queued_num[chunk.metadata] ||= 0 @queued_num[chunk.metadata] += 1 @queue_size += chunk.bytesize + add_timekey(chunk.metadata) end log.debug "buffer started", instance: self.object_id, stage_size: @stage_size, queue_size: @queue_size end @@ -206,6 +209,7 @@ super @dequeued = @stage = @queue = @queued_num = @metadata_list = nil @stage_size = @queue_size = 0 + @timekeys.clear end def storable? @@ -251,6 +255,7 @@ @metadata_list[i] else @metadata_list << metadata + add_timekey(metadata) metadata end end @@ -261,6 +266,30 @@ add_metadata(meta) end + def add_timekey(metadata) + if t = metadata.timekey + @timekeys[t] += 1 + end + nil + end + private :add_timekey + + def del_timekey(metadata) + if t = metadata.timekey + if @timekeys[t] <= 1 + @timekeys.delete(t) + else + @timekeys[t] -= 1 + end + end + nil + end + private :del_timekey + + def timekeys + @timekeys.keys + end + # metadata MUST have consistent object_id for each variation # data MUST be Array of serialized events, or EventStream # metadata_and_data MUST be a hash of { metadata => data } @@ -506,6 +535,7 @@ @metadata_list.delete(metadata) @queued_num.delete(metadata) @dequeued_num.delete(metadata) + del_timekey(metadata) end log.trace "chunk purged", instance: self.object_id, chunk_id: dump_unique_id_hex(chunk_id), metadata: metadata end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_forward.rb new/lib/fluent/plugin/in_forward.rb --- old/lib/fluent/plugin/in_forward.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/in_forward.rb 2019-04-02 17:04:49.000000000 +0200 @@ -47,6 +47,8 @@ config_param :resolve_hostname, :bool, default: nil desc 'Connections will be disconnected right after receiving first message if this value is true.' config_param :deny_keepalive, :bool, default: false + desc 'Check the remote connection is still available by sending a keepalive packet if this value is true.' + config_param :send_keepalive_packet, :bool, default: false desc 'Log warning if received chunk size is larger than this value.' config_param :chunk_size_warn_limit, :size, default: nil @@ -141,6 +143,10 @@ }) end end + + if @send_keepalive_packet && @deny_keepalive + raise Fluent::ConfigError, "both 'send_keepalive_packet' and 'deny_keepalive' cannot be set to true" + end end def multi_workers_ready? @@ -161,6 +167,7 @@ shared: shared_socket, resolve_name: @resolve_hostname, linger_timeout: @linger_timeout, + send_keepalive_packet: @send_keepalive_packet, backlog: @backlog, &method(:handle_connection) ) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_http.rb new/lib/fluent/plugin/in_http.rb --- old/lib/fluent/plugin/in_http.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/in_http.rb 2019-04-02 17:04:49.000000000 +0200 @@ -394,7 +394,7 @@ if @cors_allow_origins.include?('*') header["Access-Control-Allow-Origin"] = "*" send_response_and_close("200 OK", header, "") - elsif @cors_allow_origins.include?(@origin) + elsif include_cors_allow_origin header["Access-Control-Allow-Origin"] = @origin send_response_and_close("200 OK", header, "") else @@ -414,7 +414,7 @@ # For every incoming request, we check if we have some CORS # restrictions and white listed origins through @cors_allow_origins. unless @cors_allow_origins.nil? - unless @cors_allow_origins.include?('*') or @cors_allow_origins.include?(@origin) + unless @cors_allow_origins.include?('*') or include_cors_allow_origin send_response_and_close("403 Forbidden", {'Connection' => 'close'}, "") return end @@ -464,7 +464,7 @@ unless @cors_allow_origins.nil? if @cors_allow_origins.include?('*') header['Access-Control-Allow-Origin'] = '*' - elsif @cors_allow_origins.include?(@origin) + elsif include_cors_allow_origin header['Access-Control-Allow-Origin'] = @origin end end @@ -512,6 +512,17 @@ data << "\r\n" write data end + + def include_cors_allow_origin + if @cors_allow_origins.include?(@origin) + return true + end + filtered_cors_allow_origins = @cors_allow_origins.select {|origin| origin != ""} + return filtered_cors_allow_origins.find do |origin| + (start_str,end_str) = origin.split("*",2) + @origin.start_with?(start_str) and @origin.end_with?(end_str) + end != nil + end end end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_monitor_agent.rb new/lib/fluent/plugin/in_monitor_agent.rb --- old/lib/fluent/plugin/in_monitor_agent.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/in_monitor_agent.rb 2019-04-02 17:04:49.000000000 +0200 @@ -285,6 +285,7 @@ MONITOR_INFO = { 'output_plugin' => ->(){ is_a?(::Fluent::Plugin::Output) }, 'buffer_queue_length' => ->(){ throw(:skip) unless instance_variable_defined?(:@buffer) && !@buffer.nil? && @buffer.is_a?(::Fluent::Plugin::Buffer); @buffer.queue.size }, + 'buffer_timekeys' => ->(){ throw(:skip) unless instance_variable_defined?(:@buffer) && !@buffer.nil? && @buffer.is_a?(::Fluent::Plugin::Buffer); @buffer.timekeys }, 'buffer_total_queued_size' => ->(){ throw(:skip) unless instance_variable_defined?(:@buffer) && !@buffer.nil? && @buffer.is_a?(::Fluent::Plugin::Buffer); @buffer.stage_size + @buffer.queue_size }, 'retry_count' => ->(){ instance_variable_defined?(:@num_errors) ? @num_errors : nil }, } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tail.rb new/lib/fluent/plugin/in_tail.rb --- old/lib/fluent/plugin/in_tail.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/in_tail.rb 2019-04-02 17:04:50.000000000 +0200 @@ -171,7 +171,7 @@ @encoding = parse_encoding_param(@encoding) if @encoding @from_encoding = parse_encoding_param(@from_encoding) if @from_encoding - if @encoding == @from_encoding + if @encoding && (@encoding == @from_encoding) log.warn "'encoding' and 'from_encoding' are same encoding. No effect" end end @@ -239,6 +239,7 @@ false end rescue Errno::ENOENT + log.debug("#{p} is missing after refresh file list") false end } @@ -260,6 +261,8 @@ target_paths = expand_paths existence_paths = @tails.keys + log.debug { "tailing paths: target = #{target_paths.join(",")} | existing = #{existence_paths.join(",")}" } + unwatched = existence_paths - target_paths added = target_paths - existence_paths @@ -337,7 +340,7 @@ def update_watcher(path, pe) if @pf unless pe.read_inode == @pf[path].read_inode - log.trace "Skip update_watcher because watcher has been already updated by other inotify event" + log.debug "Skip update_watcher because watcher has been already updated by other inotify event" return end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_tcp.rb new/lib/fluent/plugin/in_tcp.rb --- old/lib/fluent/plugin/in_tcp.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/in_tcp.rb 2019-04-02 17:04:50.000000000 +0200 @@ -33,6 +33,8 @@ config_param :source_host_key, :string, default: nil, deprecated: "use source_hostname_key instead." desc "The field name of the client's hostname." config_param :source_hostname_key, :string, default: nil + desc "The field name of the client's address." + config_param :source_address_key, :string, default: nil config_param :blocking_timeout, :time, default: 0.5 @@ -76,6 +78,7 @@ tag = extract_tag_from_record(record) tag ||= @tag time ||= extract_time_from_record(record) || Fluent::EventTime.now + record[@source_address_key] = conn.remote_addr if @source_address_key record[@source_hostname_key] = conn.remote_host if @source_hostname_key router.emit(tag, time, record) end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/in_udp.rb new/lib/fluent/plugin/in_udp.rb --- old/lib/fluent/plugin/in_udp.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/in_udp.rb 2019-04-02 17:04:50.000000000 +0200 @@ -33,6 +33,8 @@ config_param :source_host_key, :string, default: nil, deprecated: "use source_hostname_key instead." desc "The field name of the client's hostname." config_param :source_hostname_key, :string, default: nil + desc "The field name of the client's address." + config_param :source_address_key, :string, default: nil desc "Deprecated parameter. Use message_length_limit instead" config_param :body_size_limit, :size, default: nil, deprecated: "use message_length_limit instead." @@ -79,6 +81,7 @@ tag = extract_tag_from_record(record) tag ||= @tag time ||= extract_time_from_record(record) || Fluent::EventTime.now + record[@source_address_key] = sock.remote_addr if @source_address_key record[@source_hostname_key] = sock.remote_host if @source_hostname_key router.emit(tag, time, record) end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/out_exec_filter.rb new/lib/fluent/plugin/out_exec_filter.rb --- old/lib/fluent/plugin/out_exec_filter.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/out_exec_filter.rb 2019-04-02 17:04:50.000000000 +0200 @@ -69,7 +69,7 @@ desc 'The number of spawned process for command.' config_param :num_children, :integer, default: 1 - desc 'Respawn command when command exit. ["none", "inf" or positive integer for times to respawn (defaut: none)]' + desc 'Respawn command when command exit. ["none", "inf" or positive integer for times to respawn (default: none)]' # nil, 'none' or 0: no respawn, 'inf' or -1: infinite times, positive integer: try to respawn specified times only config_param :child_respawn, :string, default: nil diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin/out_file.rb new/lib/fluent/plugin/out_file.rb --- old/lib/fluent/plugin/out_file.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin/out_file.rb 2019-04-02 17:04:50.000000000 +0200 @@ -68,6 +68,7 @@ config_set_default :timekey, DEFAULT_TIMEKEY end + attr_reader :dir_perm attr_accessor :last_written_path # for tests module SymlinkBufferMixin @@ -87,7 +88,9 @@ # These chunks will be enqueued immediately, and will be flushed soon. latest_metadata = metadata_list.select{|m| m.timekey }.sort_by(&:timekey).last if chunk.metadata == latest_metadata - FileUtils.ln_sf(chunk.path, @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk)) + sym_path = @_output_plugin_for_symlink.extract_placeholders(@_symlink_path, chunk) + FileUtils.mkdir_p(File.dirname(sym_path), mode: @_output_plugin_for_symlink.dir_perm) + FileUtils.ln_sf(chunk.path, sym_path) end chunk end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/plugin_helper/socket_option.rb new/lib/fluent/plugin_helper/socket_option.rb --- old/lib/fluent/plugin_helper/socket_option.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/plugin_helper/socket_option.rb 2019-04-02 17:04:50.000000000 +0200 @@ -24,7 +24,7 @@ FORMAT_STRUCT_LINGER = 'I!I!' # { int l_onoff; int l_linger; } FORMAT_STRUCT_TIMEVAL = 'L!L!' # { time_t tv_sec; suseconds_t tv_usec; } - def socket_option_validate!(protocol, resolve_name: nil, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil) + def socket_option_validate!(protocol, resolve_name: nil, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil, send_keepalive_packet: nil) unless resolve_name.nil? if protocol != :tcp && protocol != :udp && protocol != :tls raise ArgumentError, "BUG: resolve_name in available for tcp/udp/tls" @@ -35,9 +35,14 @@ raise ArgumentError, "BUG: linger_timeout is available for tcp/tls" end end + if send_keepalive_packet + if protocol != :tcp + raise ArgumentError, "BUG: send_keepalive_packet is available for tcp" + end + end end - def socket_option_set(sock, resolve_name: nil, nonblock: false, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil) + def socket_option_set(sock, resolve_name: nil, nonblock: false, linger_timeout: nil, recv_timeout: nil, send_timeout: nil, receive_buffer_size: nil, send_keepalive_packet: nil) unless resolve_name.nil? sock.do_not_reverse_lookup = !resolve_name end @@ -59,6 +64,9 @@ if receive_buffer_size socket_option_set_one(sock, :SO_RCVBUF, receive_buffer_size.to_i) end + if send_keepalive_packet + socket_option_set_one(sock, :SO_KEEPALIVE, true) + end sock end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/lib/fluent/version.rb new/lib/fluent/version.rb --- old/lib/fluent/version.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/lib/fluent/version.rb 2019-04-02 17:04:50.000000000 +0200 @@ -16,6 +16,6 @@ module Fluent - VERSION = '1.4.1' + VERSION = '1.4.2' end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/metadata new/metadata --- old/metadata 2019-03-19 14:48:00.000000000 +0100 +++ new/metadata 2019-04-02 17:04:49.000000000 +0200 @@ -1,14 +1,14 @@ --- !ruby/object:Gem::Specification name: fluentd version: !ruby/object:Gem::Version - version: 1.4.1 + version: 1.4.2 platform: ruby authors: - Sadayuki Furuhashi autorequire: bindir: bin cert_chain: [] -date: 2019-03-19 00:00:00.000000000 Z +date: 2019-04-02 00:00:00.000000000 Z dependencies: - !ruby/object:Gem::Dependency name: msgpack diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_buffer.rb new/test/plugin/test_buffer.rb --- old/test/plugin/test_buffer.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_buffer.rb 2019-04-02 17:04:50.000000000 +0200 @@ -189,6 +189,7 @@ assert_equal 0, plugin.stage_size assert_equal 0, plugin.queue_size + assert_equal [], plugin.timekeys # @p is started plugin @@ -242,6 +243,7 @@ assert_nil @p.instance_eval{ @metadata_list } # #metadata_list does #dup for @metadata_list assert_equal 0, @p.stage_size assert_equal 0, @p.queue_size + assert_equal [], @p.timekeys end test '#metadata_list returns list of metadata on stage or in queue' do @@ -569,9 +571,12 @@ assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata) assert_equal [@dm2,@dm3], @p.stage.keys - prev_stage_size = @p.stage_size + timekey = Time.parse('2016-04-11 16:40:00 +0000').to_i + assert !@p.timekeys.include?(timekey) - m = @p.metadata(timekey: Time.parse('2016-04-11 16:40:00 +0000').to_i) + prev_stage_size = @p.stage_size + + m = @p.metadata(timekey: timekey) @p.write({m => ["x" * 256, "y" * 256, "z" * 256]}) @@ -581,6 +586,8 @@ assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata) assert_equal [@dm2,@dm3,m], @p.stage.keys + + assert @p.timekeys.include?(timekey) end test '#write tries to enqueue and store data into a new chunk if existing chunk is full' do @@ -688,8 +695,11 @@ assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata) assert_equal [@dm2,@dm3], @p.stage.keys + + timekey = Time.parse('2016-04-11 16:40:00 +0000').to_i + assert !@p.timekeys.include?(timekey) - m = @p.metadata(timekey: Time.parse('2016-04-11 16:40:00 +0000').to_i) + m = @p.metadata(timekey: timekey) es = Fluent::ArrayEventStream.new( [ @@ -708,6 +718,8 @@ assert_equal [@dm0,@dm1,@dm1], @p.queue.map(&:metadata) assert_equal [@dm2,@dm3,m], @p.stage.keys assert_equal 1, @p.stage[m].append_count + + assert @p.timekeys.include?(timekey) end test '#write w/ format tries to enqueue and store data into a new chunk if existing chunk does not have enough space' do diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_forward.rb new/test/plugin/test_in_forward.rb --- old/test/plugin/test_in_forward.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_in_forward.rb 2019-04-02 17:04:50.000000000 +0200 @@ -81,6 +81,27 @@ assert_equal 1, d.instance.security.users.size assert_equal 1, d.instance.security.clients.size end + + test 'send_keepalive_packet is disabled by default' do + @d = d = create_driver(CONFIG_AUTH) + assert_false d.instance.send_keepalive_packet + end + + test 'send_keepalive_packet can be enabled' do + @d = d = create_driver(CONFIG_AUTH + %[ + send_keepalive_packet true + ]) + assert_true d.instance.send_keepalive_packet + end + + test 'both send_keepalive_packet and deny_keepalive cannot be enabled' do + assert_raise(Fluent::ConfigError.new("both 'send_keepalive_packet' and 'deny_keepalive' cannot be set to true")) do + create_driver(CONFIG_AUTH + %[ + send_keepalive_packet true + deny_keepalive true + ]) + end + end end sub_test_case 'message' do diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_http.rb new/test/plugin/test_in_http.rb --- old/test/plugin/test_in_http.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_in_http.rb 2019-04-02 17:04:50.000000000 +0200 @@ -641,6 +641,63 @@ end end + def test_cors_allowed_wildcard_for_subdomain + d = create_driver(CONFIG + 'cors_allow_origins ["http://*.foo.com"]') + + time = event_time("2011-01-02 13:14:15 UTC") + events = [ + ["tag1", time, {"a"=>1}], + ] + + d.run do + events.each do |tag, time, record| + headers = {"Origin" => "http://subdomain.foo.com"} + + res = post("/#{tag}", {"json" => record.to_json, "time" => time.to_i}, headers) + + assert_equal "200", res.code + assert_equal "http://subdomain.foo.com", res["Access-Control-Allow-Origin"] + end + end + end + + def test_cors_allowed_exclude_empty_string + d = create_driver(CONFIG + 'cors_allow_origins ["", "http://*.foo.com"]') + + time = event_time("2011-01-02 13:14:15 UTC") + events = [ + ["tag1", time, {"a"=>1}], + ] + + d.run do + events.each do |tag, time, record| + headers = {"Origin" => "http://subdomain.foo.com"} + + res = post("/#{tag}", {"json" => record.to_json, "time" => time.to_i}, headers) + + assert_equal "200", res.code + assert_equal "http://subdomain.foo.com", res["Access-Control-Allow-Origin"] + end + end + end + + def test_cors_allowed_wildcard_preflight_for_subdomain + d = create_driver(CONFIG + 'cors_allow_origins ["http://*.foo.com"]') + + d.run do + header = { + "Origin" => "http://subdomain.foo.com", + "Access-Control-Request-Method" => "POST", + "Access-Control-Request-Headers" => "Content-Type", + } + res = options("/cors.test", {}, header) + + assert_equal "200", res.code + assert_equal "http://subdomain.foo.com", res["Access-Control-Allow-Origin"] + assert_equal "POST", res["Access-Control-Allow-Methods"] + end + end + def test_content_encoding_gzip d = create_driver diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_monitor_agent.rb new/test/plugin/test_in_monitor_agent.rb --- old/test/plugin/test_in_monitor_agent.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_in_monitor_agent.rb 2019-04-02 17:04:50.000000000 +0200 @@ -123,6 +123,7 @@ output_info.merge!("config" => {"@id" => "test_out", "@type" => "test_out"}) if with_config error_label_info = { "buffer_queue_length" => 0, + "buffer_timekeys" => [], "buffer_total_queued_size" => 0, "output_plugin" => true, "plugin_category" => "output", @@ -296,6 +297,7 @@ expected_test_in_response.merge!("config" => {"@id" => "test_in", "@type" => "test_in"}) if with_config expected_null_response = { "buffer_queue_length" => 0, + "buffer_timekeys" => [], "buffer_total_queued_size" => 0, "output_plugin" => true, "plugin_category" => "output", @@ -333,6 +335,7 @@ expected_test_in_response.merge!("config" => {"@id" => "test_in", "@type" => "test_in"}) if with_config expected_null_response = { "buffer_queue_length" => 0, + "buffer_timekeys" => [], "buffer_total_queued_size" => 0, "output_plugin" => true, "plugin_category" => "output", @@ -367,6 +370,7 @@ } expected_null_response = { "buffer_queue_length" => 0, + "buffer_timekeys" => [], "buffer_total_queued_size" => 0, "output_plugin" => true, "plugin_category" => "output", @@ -434,7 +438,8 @@ <match **> @type test_out_fail_write @id test_out_fail_write - <buffer> + <buffer time> + timekey 1m flush_mode immediate </buffer> </match> @@ -453,17 +458,18 @@ include_config no ") d.instance.start + output = @ra.outputs[0] + output.start + output.after_start expected_test_out_fail_write_response = { "buffer_queue_length" => 1, + "buffer_timekeys" => [output.calculate_timekey(event_time)], "buffer_total_queued_size" => 40, "output_plugin" => true, "plugin_category" => "output", "plugin_id" => "test_out_fail_write", "type" => "test_out_fail_write", } - output = @ra.outputs[0] - output.start - output.after_start output.emit_events('test.tag', Fluent::ArrayEventStream.new([[event_time, {"message" => "test failed flush 1"}]])) # flush few times to check steps 2.times do diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_tcp.rb new/test/plugin/test_in_tcp.rb --- old/test/plugin/test_in_tcp.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_in_tcp.rb 2019-04-02 17:04:50.000000000 +0200 @@ -142,4 +142,24 @@ assert event[1].is_a?(Fluent::EventTime) assert_equal hostname, event[2]['host'] end + + test 'source_address_key' do + d = create_driver(BASE_CONFIG + %! + format none + source_address_key addr + !) + address = nil + d.run(expect_records: 1) do + create_tcp_socket('127.0.0.1', PORT) do |sock| + address = sock.peeraddr[3] + sock.send("test\n", 0) + end + end + + assert_equal 1, d.events.size + event = d.events[0] + assert_equal "tcp", event[0] + assert event[1].is_a?(Fluent::EventTime) + assert_equal address, event[2]['addr'] + end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_in_udp.rb new/test/plugin/test_in_udp.rb --- old/test/plugin/test_in_udp.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_in_udp.rb 2019-04-02 17:04:50.000000000 +0200 @@ -201,6 +201,26 @@ assert_equal hostname, d.events[0][2]['host'] end + test 'source_address_key' do + d = create_driver(BASE_CONFIG + %! + format none + source_address_key addr + !) + address = nil + d.run(expect_records: 1) do + create_udp_socket('127.0.0.1', PORT) do |u| + u.send("test", 0) + address = u.peeraddr[3] + end + end + + expected = {'message' => 'test'} + assert_equal 1, d.events.size + assert_equal "udp", d.events[0][0] + assert d.events[0][1].is_a?(Fluent::EventTime) + assert_equal address, d.events[0][2]['addr'] + end + test 'receive_buffer_size' do # doesn't check exact value because it depends on platform and condition diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin/test_out_file.rb new/test/plugin/test_out_file.rb --- old/test/plugin/test_out_file.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin/test_out_file.rb 2019-04-02 17:04:50.000000000 +0200 @@ -684,11 +684,11 @@ omit "Windows doesn't support symlink" if Fluent.windows? conf = %[ path #{TMP_DIR}/${tag}/out_file_test - symlink_path #{SYMLINK_PATH}-${tag} + symlink_path #{SYMLINK_PATH}/foo/${tag} <buffer tag,time> </buffer> ] - symlink_path = "#{SYMLINK_PATH}-tag" + symlink_path = "#{SYMLINK_PATH}/foo/tag" d = create_driver(conf) begin diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/plugin_helper/test_server.rb new/test/plugin_helper/test_server.rb --- old/test/plugin_helper/test_server.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/plugin_helper/test_server.rb 2019-04-02 17:04:50.000000000 +0200 @@ -237,6 +237,15 @@ end data( + 'server_create udp' => [:server_create, :udp], + ) + test 'raise error if tcp/tls/unix options specified for udp' do |(m, proto)| + assert_raise(ArgumentError.new("BUG: send_keepalive_packet is available for tcp")) do + @d.__send__(m, :myserver, PORT, proto: proto, send_keepalive_packet: true){|x| x } + end + end + + data( 'server_create tcp' => [:server_create, :tcp, {}], 'server_create udp' => [:server_create, :udp, {max_bytes: 128}], # 'server_create unix' => [:server_create, :unix, {}], @@ -352,7 +361,7 @@ sub_test_case '#server_create_tcp' do test 'can accept all keyword arguments valid for tcp server' do assert_nothing_raised do - @d.server_create_tcp(:s, PORT, bind: '127.0.0.1', shared: false, resolve_name: true, linger_timeout: 10, backlog: 500) do |data, conn| + @d.server_create_tcp(:s, PORT, bind: '127.0.0.1', shared: false, resolve_name: true, linger_timeout: 10, backlog: 500, send_keepalive_packet: true) do |data, conn| # ... end end diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/test/test_log.rb new/test/test_log.rb --- old/test/test_log.rb 2019-03-19 14:48:00.000000000 +0100 +++ new/test/test_log.rb 2019-04-02 17:04:50.000000000 +0200 @@ -744,12 +744,6 @@ @log.disable_events(Thread.current) end - def test_time_format - assert_equal(@log.time_format, @logger.time_format) - @log.time_format = "time_format" - assert_equal(@log.time_format, @logger.time_format) - end - def test_event mock(@logger).event(Fluent::Log::LEVEL_TRACE, { key: "value" }) @log.event(Fluent::Log::LEVEL_TRACE, { key: "value" }) @@ -794,6 +788,36 @@ assert_equal(@log.optional_attrs, @logger.optional_attrs) end end + + sub_test_case "partially delegated" do + def setup + super + @log = Fluent::PluginLogger.new(@logger) + end + + data( + text: [:text, "2016-04-21 11:58:41 +0900 [info]: yaaay\n"], + json: [:json, %Q({"time":"2016-04-21 11:58:41 +0900","level":"info","message":"yaaay"}\n)], + ) + def test_format(data) + fmt, expected_log_line = data + @log.format = fmt + @log.info "yaaay" + assert{ @log_device.logs.include? expected_log_line } + end + + data( + text: [:text, "2016 [info]: yaaay\n"], + json: [:json, %Q({"time":"2016","level":"info","message":"yaaay"}\n)], + ) + def test_time_format(data) + fmt, expected_log_line = data + @log.format = fmt + @log.time_format = "%Y" + @log.info "yaaay" + assert{ @log_device.logs.include? expected_log_line } + end + end end class PluginLoggerMixinTest < Test::Unit::TestCase