Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8368 Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Change By: Josh Cooper Release Notes Summary: Puppet agents on non-English Windows hosts would crash if they encountered a socket error, such as failing to connect to the default server named "puppet". Release Notes: Bug Fix Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Josh Cooper commented on PUP-8368 Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Patch works: $ ./ruby --version ruby 2.4.3p205 (2017-12-14 revision 61247) [x64-mingw32] $ ./ruby tcp.rb 証明書をリクエストできませんでした: getaddrinfo: そのようなホストは不明です。 Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Josh Cooper commented on PUP-8368 Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Ruby merged a fix upstream: https://github.com/ruby/ruby/commit/a41005eb6a99376ea940888bcf97140643d18f61. I'm trying a ruby build with that patch (against 2.4.3) to see if it resolves the issue. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Josh Cooper commented on PUP-8368 Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Same result on our French Windows image: PS C:\work\puppet> Get-WinSystemLocale LCID Name DisplayName --- 1036 fr-FRFrançais (France) PS C:\work\puppet> ruby .\tcp.rb ./tcp.rb:6:in `%': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError) from ./tcp.rb:6:in `rescue in ' from ./tcp.rb:3:in `'
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Kenn Hussey updated an issue Puppet / PUP-8368 Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Change By: Kenn Hussey Flagged: Impediment Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Josh Cooper commented on PUP-8368
Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
This is a smaller reproduction, which more closely matches the agent behavior where the format string is a valid "UTF-8" encoded string:
# coding: utf-8
require 'socket'
begin
TCPSocket.open('nowhere', 80)
rescue => e
puts "証明書をリクエストできませんでした: %s" % e.message
end
Add Comment
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Josh Cooper commented on PUP-8368
Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
I can reproduce the problem using just ruby on non-Windows platforms. It appears ruby will barf if given a non-ascii *format* string and an ASCII_8BIT encoded argument string.
# coding: cp932
#
# TCP.connect raises an exception whose message is ASCII_8BIT
message = "Failed to open TCP connection to nowhere (getaddrinfo: そのようなホストは不明です。 )"
message.force_encoding(Encoding::ASCII_8BIT)
puts "Error: %{message}" % {message: message}
# this doesn't
begin
puts "証明書をリクエストできませんでした: %s" % message
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Josh Cooper updated an issue
Puppet / PUP-8368
Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Change By:
Josh Cooper
On a Windows Japanese box if you run 'puppet agent -t -- sever server no_such_host' the agent prints errors about incompatible encodings:{code}$ puppet.bat agent -t --server no_such_hostWarning: ノード定義を取得できませんが、agent実行を続行します:Warning: Failed to open TCP connection to no_such_host:8140 (getaddrinfo: そのようなホストは不明です。 )Info: pluginfactsを取得しています。Warning: detail.class: SocketErrorWarning: detail.to_s.encoding: ASCII-8BITError: カタログ適用に失敗しました: incompatible character encodings: UTF-8 and ASCII-8BITError: Puppet設定クライアントを実行できません: incompatible character encodings: UTF-8 and ASCII-8BIT{code}The underlying issue appears to be when an error has be rescued, and as the error string is ASCII-8BIT and the formatting of the string blows up:{code} def eval_generate(resource)return false unless resource.respond_to?(:eval_generate)raise Puppet::DevError, _("Depthfirst resources are not supported by eval_generate") if resource.depthfirst?begin generated = replace_duplicates_with_catalog_resources(resource.eval_generate) return false if generated.empty?rescue => detail @resources_failed_to_generate = true #TRANSLATORS eval_generate is a method name and should be left untranslated resource.log_exception(detail, _("Failed to generate additional resources using 'eval_generate': %{detail}") % { detail: detail }){code}If you print out the class and string encoding value:{code}Warning: detail.class: SocketErrorWarning: detail.to_s: Failed to open TCP connection to no_such_host:8140 (getaddrinfo: そのようなホストは不明です。 )Warning: detail.to_s.encoding: ASCII-8BIT{code}Here is the stack trace:{code}C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/gems/2.4.0/gems/fast_gettext-1.1.1/lib/fast_gettext/vendor/string.rb:70:in `%'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/gems/2.4.0/gems/fast_gettext-1.1.1/lib/fast_gettext/vendor/string.rb:70:in `%'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/transaction/additional_resource_generator.rb:64:in `rescue in eval_generate'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/transaction/additional_resource_generator.rb:55:in `eval_generate'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/transaction.rb:112:in `block in evaluate'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:119:in `traverse'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/transaction.rb:169:in `evaluate'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:231:in `block in apply'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/util/log.rb:160:in `with_destination'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/transaction/report.rb:144:in `as_logging_destination'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:230:in `apply'C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/vendor_ruby/puppet/configurer/downloader.rb:13:in `evaluate'C:/Program
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Josh Cooper assigned an issue to Josh Cooper Puppet / PUP-8368 Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Change By: Josh Cooper Assignee: Josh Cooper Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Josh Cooper commented on PUP-8368
Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
A small test case shows that ruby is raising a SocketError whose detail is an ASCII_8BIT (aka binary) string:
require 'socket'
begin
TCPSocket.open('nowhere', 80)
rescue => e
puts e.message
puts e.message.encoding
end
C:\work\puppet>ruby tcp.rb
getaddrinfo: No such host is known.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Eric Delaney updated an issue Puppet / PUP-8368 Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Change By: Eric Delaney Acceptance Criteria: * enable test on Windows JA tests/i18n/modules/puppet_agent_cached_catalog.rb Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Eric Delaney commented on PUP-8368
Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
This works in the case of the first failure:
resource.log_exception(detail, _("Format Failed to generate additional resources using 'eval_generate': % {detail}
") % { detail: Puppet::Util::CharacterEncoding.convert_to_utf_8(detail.to_s) }
)
Probably what we need to do is find all the places that we rescue a raised error and add a the 'convert_to_utf_8()' call on the string detail to ensure we're passing in a UTF-8 string.
Add Comment
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Eric Delaney commented on PUP-8368
Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
We need to change the encoding from the default_external to UTF-8.
def encode_string(string) string.encode('UTF-8', Encoding.default_external) end
Add Comment
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title
Eric Delaney commented on PUP-8368
Re: Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
So the problem appears to be that the library calls are returning a string that is not UTF-8 compatible.
The Ruby on windows is running with Windows-31J as the default external encoding
Warning: Encoding.default_internal: Warning: Encoding.default_external: Windows-31J
If you print out the error string is:
Warning: detail.to_s: Failed to open TCP connection to no_such_server:8140 (getaddrinfo: そのようなホストは不明です。 )
Warning: detail.to_s.encoding: ASCII-8BIT
If you print out the error string in hex you get the following:
Warning: detail as hex:
["46 ", "61 ", "69 ", "6c ", "65 ", "64 ", "20 ", "74 ", "6f ", "20 ", "6f ", "70 ", "65 ", "6e ", "20 ", "54 ", "43 ", "50 ", "20 ", "63 ", "6f ", "6e ", "6e ", "65 ", "63 ", "74 ", "69 ", "6f ", "6e ", "20 ", "74 ", "6f ", "20 ", "6e ", "6f ", "5f ", "73 ", "75 ", "63 ", "68 ", "5f ", "73 ", "65 ", "72 ", "76 ", "65 ", "72 ", "3a ", "38 ", "31 ", "34 ", "30 ", "20 ", "28 ", "67 ", "65 ", "74 ", "61 ", "64 ", "64 ", "72 ", "69 ", "6e ", "66 ", "6f ", "3a ", "20 ", "82 ", "bb ", "82 ", "cc ", "82 ", "e6 ", "82 ", "a4 ", "82 ", "c8 ", "83 ", "7a ", "83 ", "58 ", "83 ", "67 ", "82 ", "cd ", "95 ", "73 ", "96 ", "be ", "82 ", "c5 ", "82 ", "b7 ", "81 ", "42 ", "20 ", "29 "]
If you try to convert the string 'detail_to_s = detail.to_s.encode("Windows-31J")' it fails because the "82" in the strings is not a valid UTF-8 character (this is true for encode to Windows-31J or UTF-8):
Jira (PUP-8368) Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server
Title: Message Title Eric Delaney updated an issue Puppet / PUP-8368 Japanese windows puppet agent run fails with 'incompatible character encodings' when run against non-existent server Change By: Eric Delaney Summary: Japanese windows puppet agent run fails with 'incompatible character encodings' when unknown run against non-existent server given Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
