Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Hi Nobert, my email last week was incorrect. > luahbtex should behave better, right, if the font is loaded with the > harf renderer selected? Actually, mode=harf with luaotfload 3.12 uses only 0.3 GB, but \setmainfont in fontspec.sty somehow uses 2 GB... It was reported at https://github.com/wspr/fontspec/issues/414 and https://ja.osdn.net/projects/luatex-ja/ticket/40052 Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
> luahbtex should behave better, right, if the font is loaded with the
> harf renderer selected?
Ah, you seems right. But, when I added [RawFeature={mode=harf}] to
the example given at the beginning of this report and processed the file
with lualatex-dev in TeXLive 2019 of February 14, the memory comsumption
was again mere 2GB... (I did it on last Saturday).
{mode=harf} can possibly behave better as we do not observe the huge
memory comsumption in xelatex, but I have personally not digged into
that possiblitty as the lualatex-ja developer recommends not to use
{mode=harf} with lualatex-ja as
\paragraph{Using \LuaTeX-ja with LuaHB\TeX}
Using \LuaTeX-ja with LuaHB\TeX (\LuaTeX\ integrated with
\href{https://github.com/harfbuzz/harfbuzz}{HarfBuzz}) is not well tested.
Maybe documents can typeset without an error,
but with unwanted results (especially, vertical typesetting and \cs{CID}).
Especially,
\emph{We don't recommend defining a Japanese font with HarfBuzz},
by specifying \texttt{Renderer=Harfbuzz}~etc.\ (\Pkg{fontspec}) or~{mode=harf}
(otherwise).
at
https://ja.osdn.net/projects/luatex-ja/scm/git/luatexja/blobs/master/doc/luatexja.dtx
Best regards, Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
> the memory comsumption was improved from 6GB to mere 2GB! a "mere 2GB" ;-) luahbtex should behave better, right, if the font is loaded with the harf renderer selected? Best Norbert -- PREINING Norbert https://www.preining.info Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
> Your best bet is bringing this up to the luatex-dev mailing list, not > here in Debian, because we will surely not patch anything of that > dimension. I agree. Actually, this Debian bug already has "upstream" tag and marked "forwarded" to https://github.com/u-fischer/luaotfload/issues/49 I think this is recognized by the upstream. If my memory is correct, this comes from the underlying ConTeXt and it was also discussed in the ConTeXt mailing list. Because of that, the memory comsumption was improved from 6GB to mere 2GB! Best regards, Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Hi > lualatex in TeXlive 2020, so I bet this #930292 will remain the same, Yes, probably. > I wonder if this issue (huge memory comsumption with a huge number > (e.g. 65535) of glyphs) can be fixed... Your best bet is bringing this up to the luatex-dev mailing list, not here in Debian, because we will surely not patch anything of that dimension. Best Norbert -- PREINING Norbert https://www.preining.info Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
>> I know. It was by lualatex-dev in TeXLive 2019 as of 14 Februrary. > > Within the next 2 months or so TeX Live 2020 pretest will be uploaded, > which will contain luahbtex and all the required changes. As you know, lualatex-dev in TeXlive 2019 (after Nov. 2019) will almost the same as lualatex in TeXlive 2020, so I bet this #930292 will remain the same, which of couse I do not welcome... I wonder if this issue (huge memory comsumption with a huge number (e.g. 65535) of glyphs) can be fixed... I understand that lualatex has to examine every pair of all the glyphs, which could be 65535 * 65535 paris... Best regards, Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
> I know. It was by lualatex-dev in TeXLive 2019 as of 14 Februrary. Within the next 2 months or so TeX Live 2020 pretest will be uploaded, which will contain luahbtex and all the required changes. Best Norbert -- PREINING Norbert https://www.preining.info Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
> There is no luahbtex in Debian. I know. It was by lualatex-dev in TeXLive 2019 as of 14 Februrary. It suggests that the situation will remain the same in near future, unless packaging modifies the upstream source in significant way. I should have written the background in more detail to avoid your misunderstanding. Ryutaroh From: Norbert Preining Subject: Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts Date: Sat, 15 Feb 2020 12:27:27 +0900 > On Sat, 15 Feb 2020, Ryutaroh Matsumoto wrote: >> This is LuaHBTeX, Version 1.11.2 (TeX Live 2020/dev) > > There is no luahbtex in Debian. > > Norbert > > -- > PREINING Norbert http://www.preining.info > Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev > GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
On Sat, 15 Feb 2020, Ryutaroh Matsumoto wrote: > This is LuaHBTeX, Version 1.11.2 (TeX Live 2020/dev) There is no luahbtex in Debian. Norbert -- PREINING Norbert http://www.preining.info Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
By luaotfload 3.12 and This is LuaHBTeX, Version 1.11.2 (TeX Live 2020/dev) the memory consumption is 2 GB, by texlive 15 February 2020. Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts,Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Hi Hilmar, I forgot to mention one thing. The version of luaotfload was 2.98 (almost the same as your 2.97). What I saw may be the same as what you told. I tend to agree with the upstream developer as he also consideres this issue as an open bug at https://github.com/u-fischer/luaotfload/issues/49 Anyway 1.8 GB is much better (for Japanese) than 6GB. Ryutaroh From: Ryutaroh Matsumoto Subject: Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts,Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts Date: Fri, 19 Jul 2019 10:39:30 +0900 (JST) > Hi Hilmar, > > Thank you for your response. > I have checked this issue under TeXLive 2019 as of July 18 > (installed by "install-tl") under ubuntu 19.04 > (not debian/ubuntu packages) > > The memory consumption of the latex file included in this > bug report was "only" 1.8 GB after rm -rf ~/.texlive2019. > Noto CJK fonts versions are those of ubuntu 19.04 > (not the latest ones). > It was 6 GB when I filed this report. > Maybe 32-bit Linux can now handle the Noto CJK fonts. > > At least we can say significant improvement is in the upstream, > though 1.8 GB seems a bit larger than necessary... > > I again compiled the same latex file by xelatex, > but the processing finished immediately and > I could not see the memory consumption by "top"... > > Ryutaroh > > > From: =?UTF-8?Q?Hilmar_Preu=c3=9fe?= > Subject: Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto > CJK fonts,Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto > CJK fonts > Date: Thu, 18 Jul 2019 16:24:39 +0200 > >> On 18.07.19 13:39, Ryutaroh Matsumoto wrote: >> >> Hi Ryutaroh, >> >>> I believe that the problem is spotted and fixed at >>> https://github.com/u-fischer/luaotfload/issues/55 >>> >>> The above seems to be uploaded to ctan on July 15. >>> Maybe the next release of texlive-luatex package will close >>> this issue without special effort. >>> >> >> 2019-05-18 luaotfload v2.97 >> * fix issue #47 >> * fix whatsits interfering with letterspacing (issue #53) >> * fix luaotfload-tool switches version and find not working >> correctly (PR#59) >> * fix luaotfload-tool support of ttc fonts (PR#58) >> * sync with context files from 2019-05-18 (improves handling of >> large fonts, see e.g. issue #55 and PR#58) >> >> So, I'd expect that v2.97 solves the problem. This version however is in >> Debian unstable, hence I gave it another try. I've noticed a virtual >> memory allocation of luatex having a size of 1,9 GB when building the >> cache. Not sure, if this is an significant improvement over the initial >> situation. >> >> Hilmar >> -- >> sigfault >> #206401 http://counter.li.org >>
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts,Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Hi Hilmar, Thank you for your response. I have checked this issue under TeXLive 2019 as of July 18 (installed by "install-tl") under ubuntu 19.04 (not debian/ubuntu packages) The memory consumption of the latex file included in this bug report was "only" 1.8 GB after rm -rf ~/.texlive2019. Noto CJK fonts versions are those of ubuntu 19.04 (not the latest ones). It was 6 GB when I filed this report. Maybe 32-bit Linux can now handle the Noto CJK fonts. At least we can say significant improvement is in the upstream, though 1.8 GB seems a bit larger than necessary... I again compiled the same latex file by xelatex, but the processing finished immediately and I could not see the memory consumption by "top"... Ryutaroh From: =?UTF-8?Q?Hilmar_Preu=c3=9fe?= Subject: Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts,Re: Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts Date: Thu, 18 Jul 2019 16:24:39 +0200 > On 18.07.19 13:39, Ryutaroh Matsumoto wrote: > > Hi Ryutaroh, > >> I believe that the problem is spotted and fixed at >> https://github.com/u-fischer/luaotfload/issues/55 >> >> The above seems to be uploaded to ctan on July 15. >> Maybe the next release of texlive-luatex package will close >> this issue without special effort. >> > > 2019-05-18 luaotfload v2.97 > * fix issue #47 > * fix whatsits interfering with letterspacing (issue #53) > * fix luaotfload-tool switches version and find not working > correctly (PR#59) > * fix luaotfload-tool support of ttc fonts (PR#58) > * sync with context files from 2019-05-18 (improves handling of > large fonts, see e.g. issue #55 and PR#58) > > So, I'd expect that v2.97 solves the problem. This version however is in > Debian unstable, hence I gave it another try. I've noticed a virtual > memory allocation of luatex having a size of 1,9 GB when building the > cache. Not sure, if this is an significant improvement over the initial > situation. > > Hilmar > -- > sigfault > #206401 http://counter.li.org >
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Control: tags 930292 - fixed-upstream Control: forwarded 930292 https://github.com/u-fischer/luaotfload/issues/49 Sorry I was wrong. The issue in the upstream is still open as above. Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
On 18.07.19 13:39, Ryutaroh Matsumoto wrote: Hi Ryutaroh, > I believe that the problem is spotted and fixed at > https://github.com/u-fischer/luaotfload/issues/55 > > The above seems to be uploaded to ctan on July 15. > Maybe the next release of texlive-luatex package will close > this issue without special effort. > 2019-05-18 luaotfload v2.97 * fix issue #47 * fix whatsits interfering with letterspacing (issue #53) * fix luaotfload-tool switches version and find not working correctly (PR#59) * fix luaotfload-tool support of ttc fonts (PR#58) * sync with context files from 2019-05-18 (improves handling of large fonts, see e.g. issue #55 and PR#58) So, I'd expect that v2.97 solves the problem. This version however is in Debian unstable, hence I gave it another try. I've noticed a virtual memory allocation of luatex having a size of 1,9 GB when building the cache. Not sure, if this is an significant improvement over the initial situation. Hilmar -- sigfault #206401 http://counter.li.org signature.asc Description: OpenPGP digital signature
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Control: tags 930292 + fixed-upstream I believe that the problem is spotted and fixed at https://github.com/u-fischer/luaotfload/issues/55 The above seems to be uploaded to ctan on July 15. Maybe the next release of texlive-luatex package will close this issue without special effort. Ryutaroh
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Am 10.06.2019 um 06:13 teilte Ryutaroh Matsumoto mit: This report contains UTF-8 encoded text in the latex example. lualatex uses 6 GB of real memory with a small input (see below) when the Noto CJK fonts used in the first time. First Link: https://tex.stackexchange.com/questions/453143/noto-cjk-font-not-usable-with-context H. -- #206401 http://counter.li.org
Bug#930292: texlive-luatex: lualatex uses huge memory for Noto CJK fonts
Package: texlive-luatex
Version: 2019.20190605-1
Severity: minor
Tags: upstream l10n
Dear Maintainer,
This report contains UTF-8 encoded text in the latex example.
lualatex uses 6 GB of real memory with a small input (see below)
when the Noto CJK fonts used in the first time.
This is an upstream issue, but I file it here by a suggestion
at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929128#61
A procedure to reproduce this is as follows:
1. Install the Noto CJK fonts, e.g. by apt-get install fonts-noto-cjk
fonts-noto-cjk-extra
2. If one has already used the Noto CJK fonts by luatex, clear the cache by rm
-rf $HOME/.texlive201?
3. Process the following file by lualatex
\documentclass{minimal}
\usepackage{fontspec}
\setmainfont{Noto Serif CJK JP}
\setsansfont{Noto Sans CJK JP}
\setmonofont{Noto Sans Mono CJK JP}
\begin{document}
\noindent
{日本語test \bfseries 日本語test}\\
{\sffamily 日本語test \bfseries 日本語test}\\
{\ttfamily 日本語test \bfseries 日本語test}
\end{document}
I believe that the same issue should arise with the Chinese and
the Korean texts, but have not verified it.
Noto CJK Super OTC font (everything is included in the single file)
does not increase the memory usage (= 6GB). Debian Noto CJK package
does not employ super OTC.
Best regards,
Ryutaroh Matsumoto
-- Package-specific info:
IMPORTANT INFORMATION: We will only consider bug reports concerning
the packaging of TeX Live as relevant. If you have problems with
combination of packages in a LaTeX document, please consult your
local TeX User Group, the comp.text.tex user group, the author of
the original .sty file, or any other help resource.
In particular, bugs that are related to up-upstream, i.e., neither
Debian nor TeX Live (upstream), but the original package authors,
will be closed immediately.
*** The Debian TeX Team is *not* a LaTeX Help Desk ***
If you report an error when running one of the TeX-related binaries
(latex, pdftex, metafont,...), or if the bug is related to bad or wrong
output, please include a MINIMAL example input file that produces the
error in your report.
Please run your example with
(pdf)latex -recorder ...
(or any other program that supports -recorder) and send us the generated
file with the extension .fls, it lists all the files loaded during
the run and can easily explain problems induced by outdated files in
your home directory.
Don't forget to also include minimal examples of other files that are
needed, e.g. bibtex databases. Often it also helps
to include the logfile. Please, never send included pictures!
If your example file isn't short or produces more than one page of
output (except when multiple pages are needed to show the problem),
you can probably minimize it further. Instructions on how to do that
can be found at
http://www.minimalbeispiel.de/mini-en.html (english)
or
http://www.minimalbeispiel.de/mini.html (german)
##
minimal input file
##
other files
##
List of ls-R files
-rw-r--r-- 1 root root 2887 Jun 10 12:23 /var/lib/texmf/ls-R
lrwxrwxrwx 1 root root 29 Feb 28 23:28 /usr/share/texmf/ls-R ->
/var/lib/texmf/ls-R-TEXMFMAIN
lrwxrwxrwx 1 root root 31 Jun 5 12:18 /usr/share/texlive/texmf-dist/ls-R ->
/var/lib/texmf/ls-R-TEXLIVEDIST
lrwxrwxrwx 1 root root 31 Jun 5 12:18 /usr/share/texlive/texmf-dist/ls-R ->
/var/lib/texmf/ls-R-TEXLIVEDIST
##
Config files
-rw-r--r-- 1 root root 475 Apr 15 12:58 /etc/texmf/web2c/texmf.cnf
lrwxrwxrwx 1 root root 33 Jun 5 12:18 /usr/share/texmf/web2c/fmtutil.cnf ->
/var/lib/texmf/fmtutil.cnf-DEBIAN
lrwxrwxrwx 1 root root 32 Jun 5 12:18 /usr/share/texmf/web2c/updmap.cfg ->
/var/lib/texmf/updmap.cfg-DEBIAN
-rw-r--r-- 1 root root 5089 Jun 10 12:22
/var/lib/texmf/tex/generic/config/language.dat
##
Files in /etc/texmf/web2c/
total 8
-rw-r--r-- 1 root root 283 Jan 17 2017 mktex.cnf
-rw-r--r-- 1 root root 475 Apr 15 12:58 texmf.cnf
##
md5sums of texmf.d
ca40c66f144b4bafc3e59a2dd32ecb9c /etc/texmf/texmf.d/00debian.cnf
-- System Information:
Debian Release: 10.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-5-amd64 (SMP w/4 CPU cores)
Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8),
LANGUAGE=ja_JP.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages texlive-luatex depends on:
ii libjs-jquery 3.3.1~dfsg-3
ii tex-common6.11
ii texlive-base 2019.20190605-1
ii texlive-binaries 2019.20190605.51237-1
texlive-luatex recommends no packages.
texlive-luatex suggests no packages.
Versions of packages tex-common depends on:
ii dpkg 1.19.7
ii ucf 3.0038+nmu1
Versions of packages tex-common suggests:
pn debhel
