Here is a series of patches to clean up the Unicode mapping script
business in src/backend/utils/mb/Unicode/. It overlaps with the
perlcritic work that I recently wrote about, except that these pieces
are not strictly related to Perl, but wrong comments, missing makefile
pieces, and such.
I discovered that some of the source files that one is supposed to
download don't exist anymore or are labeled obsolete. Also, running the
scripts produces slight differences in the output. So apparently, the
CJK to Unicode mappings are still evolving and should be updated
occasionally. Next steps would be to commit some or all of these
differences after additional verification, and then update the scripts
to use whatever the non-obsolete mapping sources are supposed to be.
>From 2c96b1154c300325654735984f4268df7cc6efcd Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:12 -0400
Subject: [PATCH 01/10] UCS_to_most.pl: Make executable, for consistency with
other scripts
---
src/backend/utils/mb/Unicode/UCS_to_most.pl | 0
1 file changed, 0 insertions(+), 0 deletions(-)
mode change 100644 => 100755 src/backend/utils/mb/Unicode/UCS_to_most.pl
diff --git a/src/backend/utils/mb/Unicode/UCS_to_most.pl b/src/backend/utils/mb/Unicode/UCS_to_most.pl
old mode 100644
new mode 100755
--
2.5.1
>From 374d2954b31867e7ffe4ed183f9a9cd7a098cb9b Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:12 -0400
Subject: [PATCH 02/10] Fix comments
Some of these comments were copied and pasted without updating them,
some of them were duplicates.
---
src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl | 6 +-----
src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl | 4 ----
src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl | 6 +-----
src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl | 6 +-----
src/backend/utils/mb/Unicode/UCS_to_SJIS.pl | 4 ----
5 files changed, 3 insertions(+), 23 deletions(-)
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl
index bfc9912..643f02b 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_CN.pl
@@ -49,10 +49,6 @@
}
close(FILE);
-#
-# first, generate UTF8 --> EUC_CN table
-#
-
$file = "utf8_to_euc_cn.map";
open(FILE, "> $file") || die("cannot open $file");
print FILE "static const pg_utf_to_local ULmapEUC_CN[ $count ] = {\n";
@@ -75,7 +71,7 @@
close(FILE);
#
-# then generate EUC_JP --> UTF8 table
+# then generate EUC_CN --> UTF8 table
#
reset 'array';
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl
index 79bc05b..687e668 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_JP.pl
@@ -130,10 +130,6 @@
}
close(FILE);
-#
-# first, generate UTF8 --> EUC_JP table
-#
-
$file = "utf8_to_euc_jp.map";
open(FILE, "> $file") || die("cannot open $file");
print FILE "static const pg_utf_to_local ULmapEUC_JP[ $count ] = {\n";
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl
index fa553fd..82490a0 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_KR.pl
@@ -49,10 +49,6 @@
}
close(FILE);
-#
-# first, generate UTF8 --> EUC_KR table
-#
-
$file = "utf8_to_euc_kr.map";
open(FILE, "> $file") || die("cannot open $file");
print FILE "static const pg_utf_to_local ULmapEUC_KR[ $count ] = {\n";
@@ -75,7 +71,7 @@
close(FILE);
#
-# then generate EUC_JP --> UTF8 table
+# then generate EUC_KR --> UTF8 table
#
reset 'array';
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl
index 02414ba..697b6e6 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_TW.pl
@@ -65,10 +65,6 @@
}
close(FILE);
-#
-# first, generate UTF8 --> EUC_TW table
-#
-
$file = "utf8_to_euc_tw.map";
open(FILE, "> $file") || die("cannot open $file");
print FILE "static const pg_utf_to_local ULmapEUC_TW[ $count ] = {\n";
@@ -91,7 +87,7 @@
close(FILE);
#
-# then generate EUC_JP --> UTF8 table
+# then generate EUC_TW --> UTF8 table
#
reset 'array';
diff --git a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
index 74cd7ac..e607e91 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
@@ -66,10 +66,6 @@
close(FILE);
-#
-# first, generate UTF8 --> SJIS table
-#
-
$file = "utf8_to_sjis.map";
open(FILE, "> $file") || die("cannot open $file");
print FILE "static const pg_utf_to_local ULmapSJIS[ $count ] = {\n";
--
2.5.1
>From 8f19caa272880afc1a0857e6478ac3fac8203cd6 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:12 -0400
Subject: [PATCH 03/10] Remove manually added header comments from generated
files
These header comments were added as part of commit f3d99d16, but only a
few of the map files were changed. Since these files are in theory
generated from scripts, we shouldn't manually change them in a way
that's inconsistent with the script output.
---
src/backend/utils/mb/Unicode/euc_cn_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/euc_jp_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/euc_tw_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/gbk_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_10_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_13_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_14_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_15_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_16_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_3_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_4_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_6_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_7_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_8_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/iso8859_9_to_utf8.map | 2 --
src/backend/utils/mb/Unicode/koi8r_to_utf8.map | 2 --
18 files changed, 36 deletions(-)
diff --git a/src/backend/utils/mb/Unicode/euc_cn_to_utf8.map b/src/backend/utils/mb/Unicode/euc_cn_to_utf8.map
index 4052379..17cf7c8 100644
--- a/src/backend/utils/mb/Unicode/euc_cn_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/euc_cn_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/euc_cn_to_utf8.map */
-
static const pg_local_to_utf LUmapEUC_CN[ 7445 ] = {
{0xa1a1, 0xe38080},
{0xa1a2, 0xe38081},
diff --git a/src/backend/utils/mb/Unicode/euc_jp_to_utf8.map b/src/backend/utils/mb/Unicode/euc_jp_to_utf8.map
index db427cb..937b53c 100644
--- a/src/backend/utils/mb/Unicode/euc_jp_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/euc_jp_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/euc_jp_to_utf8.map */
-
static const pg_local_to_utf LUmapEUC_JP[] = {
{0x8ea1, 0xefbda1},
{0x8ea2, 0xefbda2},
diff --git a/src/backend/utils/mb/Unicode/euc_tw_to_utf8.map b/src/backend/utils/mb/Unicode/euc_tw_to_utf8.map
index b430b44..8535b99 100644
--- a/src/backend/utils/mb/Unicode/euc_tw_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/euc_tw_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/euc_tw_to_utf8.map */
-
static const pg_local_to_utf LUmapEUC_TW[ 23575 ] = {
{0xa1a1, 0xe38080},
{0xa1a2, 0xefbc8c},
diff --git a/src/backend/utils/mb/Unicode/gbk_to_utf8.map b/src/backend/utils/mb/Unicode/gbk_to_utf8.map
index fced1f4..964aa52 100644
--- a/src/backend/utils/mb/Unicode/gbk_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/gbk_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/gbk_to_utf8.map */
-
static const pg_local_to_utf LUmapGBK[ 21792 ] = {
{0x0080, 0xe282ac},
{0x8140, 0xe4b882},
diff --git a/src/backend/utils/mb/Unicode/iso8859_10_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_10_to_utf8.map
index 8a650ee..91d3e68 100644
--- a/src/backend/utils/mb/Unicode/iso8859_10_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_10_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_10_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_10[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_13_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_13_to_utf8.map
index 2075706..b641673 100644
--- a/src/backend/utils/mb/Unicode/iso8859_13_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_13_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_13_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_13[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_14_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_14_to_utf8.map
index 49d63d4..e2ca373 100644
--- a/src/backend/utils/mb/Unicode/iso8859_14_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_14_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_14_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_14[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_15_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_15_to_utf8.map
index 349b64c..f9803e8 100644
--- a/src/backend/utils/mb/Unicode/iso8859_15_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_15_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_15_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_15[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_16_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_16_to_utf8.map
index d8e2801..87e9246 100644
--- a/src/backend/utils/mb/Unicode/iso8859_16_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_16_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_16_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_16[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map
index 30d487a..5fedd36 100644
--- a/src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_2_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_2[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_3_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_3_to_utf8.map
index 94b5bc4..c6f824c 100644
--- a/src/backend/utils/mb/Unicode/iso8859_3_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_3_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_3_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_3[ 121 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_4_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_4_to_utf8.map
index f339c19..73228fa 100644
--- a/src/backend/utils/mb/Unicode/iso8859_4_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_4_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_4_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_4[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map
index 601be30..cd832b6 100644
--- a/src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_5_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_5[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_6_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_6_to_utf8.map
index 289f97e..5e7b676 100644
--- a/src/backend/utils/mb/Unicode/iso8859_6_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_6_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_6_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_6[ 83 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_7_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_7_to_utf8.map
index fbbecaa..987c0ad 100644
--- a/src/backend/utils/mb/Unicode/iso8859_7_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_7_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_7_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_7[ 125 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_8_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_8_to_utf8.map
index 4ed316c..9f0a597 100644
--- a/src/backend/utils/mb/Unicode/iso8859_8_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_8_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_8_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_8[ 92 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/iso8859_9_to_utf8.map b/src/backend/utils/mb/Unicode/iso8859_9_to_utf8.map
index f86cc65..93c9cf5 100644
--- a/src/backend/utils/mb/Unicode/iso8859_9_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/iso8859_9_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/iso8859_9_to_utf8.map */
-
static const pg_local_to_utf LUmapISO8859_9[ 128 ] = {
{0x0080, 0xc280},
{0x0081, 0xc281},
diff --git a/src/backend/utils/mb/Unicode/koi8r_to_utf8.map b/src/backend/utils/mb/Unicode/koi8r_to_utf8.map
index 738f160..d8a544c 100644
--- a/src/backend/utils/mb/Unicode/koi8r_to_utf8.map
+++ b/src/backend/utils/mb/Unicode/koi8r_to_utf8.map
@@ -1,5 +1,3 @@
-/* src/backend/utils/mb/Unicode/koi8r_to_utf8.map */
-
static const pg_local_to_utf LUmapKOI8R[ 128 ] = {
{0x0080, 0xe29480},
{0x0081, 0xe29482},
--
2.5.1
>From f970a7b93b815a27d1724a448a36b12e1c87b328 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:13 -0400
Subject: [PATCH 04/10] Add Unicode map generation scripts as rule
prerequisites
That way, the rules will trigger when the scripts change.
---
src/backend/utils/mb/Unicode/Makefile | 32 ++++++++++++++++----------------
1 file changed, 16 insertions(+), 16 deletions(-)
diff --git a/src/backend/utils/mb/Unicode/Makefile b/src/backend/utils/mb/Unicode/Makefile
index 353fc75..b15efce 100644
--- a/src/backend/utils/mb/Unicode/Makefile
+++ b/src/backend/utils/mb/Unicode/Makefile
@@ -68,29 +68,29 @@ GENERICTEXTS = $(ISO8859TEXTS) $(WINTEXTS) \
all: $(MAPS)
-$(GENERICMAPS) : $(GENERICTEXTS)
- $(PERL) $(srcdir)/UCS_to_most.pl
+$(GENERICMAPS): UCS_to_most.pl $(GENERICTEXTS)
+ $(PERL) $<
-euc_jp_to_utf8.map utf8_to_euc_jp.map : JIS0201.TXT JIS0208.TXT JIS0212.TXT
- $(PERL) $(srcdir)/UCS_to_EUC_JP.pl
+euc_jp_to_utf8.map utf8_to_euc_jp.map: UCS_to_EUC_JP.pl JIS0201.TXT JIS0208.TXT JIS0212.TXT
+ $(PERL) $<
-euc_cn_to_utf8.map utf8_to_euc_cn.map : GB2312.TXT
- $(PERL) $(srcdir)/UCS_to_EUC_CN.pl
+euc_cn_to_utf8.map utf8_to_euc_cn.map: UCS_to_EUC_CN.pl GB2312.TXT
+ $(PERL) $<
-euc_kr_to_utf8.map utf8_to_euc_kr.map : KSX1001.TXT
- $(PERL) $(srcdir)/UCS_to_EUC_KR.pl
+euc_kr_to_utf8.map utf8_to_euc_kr.map: UCS_to_EUC_KR.pl KSX1001.TXT
+ $(PERL) $<
-euc_tw_to_utf8.map utf8_to_euc_tw.map : CNS11643.TXT
- $(PERL) $(srcdir)/UCS_to_EUC_TW.pl
+euc_tw_to_utf8.map utf8_to_euc_tw.map: UCS_to_EUC_TW.pl CNS11643.TXT
+ $(PERL) $<
-sjis_to_utf8.map utf8_to_sjis.map : CP932.TXT
- $(PERL) $(srcdir)/UCS_to_SJIS.pl
+sjis_to_utf8.map utf8_to_sjis.map: UCS_to_SJIS.pl CP932.TXT
+ $(PERL) $<
-gb18030_to_utf8.map utf8_to_gb18030.map : gb-18030-2000.xml
- $(PERL) $(srcdir)/UCS_to_GB18030.pl
+gb18030_to_utf8.map utf8_to_gb18030.map: UCS_to_GB18030.pl gb-18030-2000.xml
+ $(PERL) $<
-big5_to_utf8.map utf8_to_big5.map : BIG5.TXT CP950.TXT
- $(PERL) $(srcdir)/UCS_to_BIG5.pl
+big5_to_utf8.map utf8_to_big5.map: UCS_to_BIG5.pl BIG5.TXT CP950.TXT
+ $(PERL) $<
distclean: clean
rm -f $(TEXTS)
--
2.5.1
>From 0931da19b1960525f696a92997e294e9bea30e5c Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:13 -0400
Subject: [PATCH 05/10] Add missing rules related to EUC_JIS_2004 and
SHIFT_JIS_2004 encodings
This was apparently forgotten in commit
75c6519ff68dbb97f73b13e9976fb8075bbde7b8.
---
src/backend/utils/mb/Unicode/Makefile | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/mb/Unicode/Makefile b/src/backend/utils/mb/Unicode/Makefile
index b15efce..1c14b13 100644
--- a/src/backend/utils/mb/Unicode/Makefile
+++ b/src/backend/utils/mb/Unicode/Makefile
@@ -50,7 +50,11 @@ SPECIALMAPS = euc_cn_to_utf8.map utf8_to_euc_cn.map \
euc_tw_to_utf8.map utf8_to_euc_tw.map \
sjis_to_utf8.map utf8_to_sjis.map \
gb18030_to_utf8.map utf8_to_gb18030.map \
- big5_to_utf8.map utf8_to_big5.map
+ big5_to_utf8.map utf8_to_big5.map \
+ euc_jis_2004_to_utf8.map euc_jis_2004_to_utf8_combined.map \
+ utf8_to_euc_jis_2004.map utf8_to_euc_jis_2004_combined.map \
+ shift_jis_2004_to_utf8.map shift_jis_2004_to_utf8_combined.map \
+ utf8_to_shift_jis_2004.map utf8_to_shift_jis_2004_combined.map
MAPS = $(GENERICMAPS) $(SPECIALMAPS)
@@ -92,6 +96,12 @@ gb18030_to_utf8.map utf8_to_gb18030.map: UCS_to_GB18030.pl gb-18030-2000.xml
big5_to_utf8.map utf8_to_big5.map: UCS_to_BIG5.pl BIG5.TXT CP950.TXT
$(PERL) $<
+euc_jis_2004_to_utf8.map euc_jis_2004_to_utf8_combined.map utf8_to_euc_jis_2004.map utf8_to_euc_jis_2004_combined.map: UCS_to_EUC_JIS_2004.pl euc-jis-2004-std.txt
+ $(PERL) $<
+
+shift_jis_2004_to_utf8.map shift_jis_2004_to_utf8_combined.map utf8_to_shift_jis_2004.map utf8_to_shift_jis_2004_combined.map: UCS_to_SHIFT_JIS_2004.pl sjis-0213-2004-std.txt
+ $(PERL) $<
+
distclean: clean
rm -f $(TEXTS)
--
2.5.1
>From c542057ba817c959fbb09f0aa44075d247db60d4 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:13 -0400
Subject: [PATCH 06/10] Make some adjustments in variable assignments
These variables aren't really used for anything interesting, but it
seems the existing grouping was somewhat nonsensical.
---
src/backend/utils/mb/Unicode/Makefile | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/src/backend/utils/mb/Unicode/Makefile b/src/backend/utils/mb/Unicode/Makefile
index 1c14b13..baf15d2 100644
--- a/src/backend/utils/mb/Unicode/Makefile
+++ b/src/backend/utils/mb/Unicode/Makefile
@@ -63,12 +63,13 @@ ISO8859TEXTS = 8859-2.TXT 8859-3.TXT 8859-4.TXT 8859-5.TXT \
8859-10.TXT 8859-13.TXT 8859-14.TXT 8859-15.TXT \
8859-16.TXT
-WINTEXTS = CP866.TXT CP874.TXT CP1250.TXT CP1251.TXT \
+WINTEXTS = CP866.TXT CP874.TXT CP936.TXT CP949.TXT \
+ CP1250.TXT CP1251.TXT \
CP1252.TXT CP1253.TXT CP1254.TXT CP1255.TXT \
CP1256.TXT CP1257.TXT CP1258.TXT
GENERICTEXTS = $(ISO8859TEXTS) $(WINTEXTS) \
- KOI8-R.TXT CP936.TXT CP949.TXT JOHAB.TXT
+ KOI8-R.TXT JOHAB.TXT
all: $(MAPS)
--
2.5.1
>From 62826ff0d14dc8a4a72ed7ef16059060b82d05d1 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:13 -0400
Subject: [PATCH 07/10] Add prerequisite for KOI8-U.TXT
This was missed when the encoding was added.
---
src/backend/utils/mb/Unicode/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/utils/mb/Unicode/Makefile b/src/backend/utils/mb/Unicode/Makefile
index baf15d2..433fb29 100644
--- a/src/backend/utils/mb/Unicode/Makefile
+++ b/src/backend/utils/mb/Unicode/Makefile
@@ -69,7 +69,7 @@ WINTEXTS = CP866.TXT CP874.TXT CP936.TXT CP949.TXT \
CP1256.TXT CP1257.TXT CP1258.TXT
GENERICTEXTS = $(ISO8859TEXTS) $(WINTEXTS) \
- KOI8-R.TXT JOHAB.TXT
+ KOI8-R.TXT KOI8-U.TXT JOHAB.TXT
all: $(MAPS)
--
2.5.1
>From 25d14ecee10d7014ed82527ba13abd93133260d9 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:13 -0400
Subject: [PATCH 08/10] Add rules to download raw mapping files
These are not part of a normal build, like this entire directory.
---
src/backend/utils/mb/Unicode/Makefile | 34 ++++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/src/backend/utils/mb/Unicode/Makefile b/src/backend/utils/mb/Unicode/Makefile
index 433fb29..342d990 100644
--- a/src/backend/utils/mb/Unicode/Makefile
+++ b/src/backend/utils/mb/Unicode/Makefile
@@ -108,3 +108,37 @@ distclean: clean
maintainer-clean: distclean
rm -f $(MAPS)
+
+
+WGET = wget -O $@ --no-use-server-timestamps
+#WGET = curl -o $@
+
+BIG.TXT:
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/$(@F)
+
+CNS11643.TXT:
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/OTHER/$(@F)
+
+GB2312.TXT:
+ $(WGET) 'http://trac.greenstone.org/browser/trunk/gsdl/unicode/MAPPINGS/EASTASIA/GB/GB2312.TXT?rev=1842&format=txt'
+
+JIS0201.TXT JIS0208.TXT JIS0212.TXT:
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/$(@F)
+
+JOHAB.TXT:
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/$(@F)
+
+KOI8-R.TXT KOI8-U.TXT:
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/VENDORS/MISC/$(@F)
+
+KSX1001.TXT:
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/$(@F)
+
+$(ISO8859TEXTS):
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/ISO8859/$(@F)
+
+$(filter-out CP8%,$(WINTEXTS)):
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/$(@F)
+
+$(filter CP8%,$(WINTEXTS)):
+ $(WGET) http://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/$(@F)
--
2.5.1
>From fd4fecaa40bc3e62e102aec577e71dda20f34be4 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:14 -0400
Subject: [PATCH 09/10] Make spacing and punctuation consistent
---
src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl | 4 ++--
src/backend/utils/mb/Unicode/UCS_to_SJIS.pl | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl b/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
index 33d108e..e8f2467 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_SHIFT_JIS_2004.pl
@@ -168,9 +168,9 @@
if ($array{$code} ne "")
{
printf STDERR
- "Warning: duplicate UTF-8: %08x UCS: %04x Shift JIS: %04x\n", $utf,
+ "Warning: duplicate UTF8: %08x UCS: %04x Shift JIS: %04x\n", $utf,
$ucs, $code;
- printf STDERR "Previous value: UTF-8: %08x\n", $array{$utf};
+ printf STDERR "Previous value: UTF8: %08x\n", $array{$utf};
next;
}
$count++;
diff --git a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
index e607e91..5d2a1ca 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_SJIS.pl
@@ -55,7 +55,7 @@
&& ($code <= 0x879c)))
{
printf STDERR
- "Warning: duplicate UTF8 : UCS=0x%04x SJIS=0x%04x\n", $ucs,
+ "Warning: duplicate UTF8: UCS=0x%04x SJIS=0x%04x\n", $ucs,
$code;
next;
}
--
2.5.1
>From 80d0ecb8c6bafab496b4acd4abc3af6495b05c9e Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <pete...@gmx.net>
Date: Mon, 31 Aug 2015 23:59:14 -0400
Subject: [PATCH 10/10] Turn off "test" mode by default
It produces debugging output files that are of no further use, so we
don't need that by default.
---
src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl b/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl
index 7860736..92252a2 100755
--- a/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl
+++ b/src/backend/utils/mb/Unicode/UCS_to_EUC_JIS_2004.pl
@@ -9,7 +9,7 @@
require "ucs2utf.pl";
-$TEST = 1;
+$TEST = 0;
# first generate UTF-8 --> EUC_JIS_2004 table
--
2.5.1
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
