Re: Failing tex2lyx test

[Richard Heck]

On 03/13/2018 08:03 AM, Jürgen Spitzmüller wrote:
> 2018-03-13 12:24 GMT+01:00 Kornel Benko  >:
>
> #0  strlen () at ../sysdeps/x86_64/strlen.S:106
> #1  0x7f5abf26043c in std::__cxx11::basic_string std::char_traits, std::allocator >::compare(char
> const*) const ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #2  0x005b403d in std::operator== std::char_traits,
> std::allocator > (__lhs=" ",
>     __rhs=0x34333231302b2d20  0x34333231302b2d20>)
>     at /usr/include/c++/5/bits/basic_string.h:4939
> #3  0x0060c359 in lyx::is_known (str=" ",
>     what=0x72c5a0  namespace)::known_escaped_chars+64>)
>     at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
>
>  
> Oh, I see. Silly me. Should be fixed in master (at least the crash).
> Is the other issue fixed as well?
>
> In any case, should also go to 2.3.2-staging.

OK.

rh

Re: Failing tex2lyx test

[Kornel Benko]

Am Dienstag, 13. März 2018 13:03:04 CET schrieb Jürgen Spitzmüller 
:
> 2018-03-13 12:24 GMT+01:00 Kornel Benko :
> > #0  strlen () at ../sysdeps/x86_64/strlen.S:106
> > #1  0x7f5abf26043c in std::__cxx11::basic_string > std::char_traits, std::allocator >::compare(char const*) const
> > ()
> > from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> > #2  0x005b403d in std::operator== > std::allocator > (__lhs=" ",
> > 
> > __rhs=0x34333231302b2d20  > 
> > 0x34333231302b2d20>)
> > 
> > at /usr/include/c++/5/bits/basic_string.h:4939
> > 
> > #3  0x0060c359 in lyx::is_known (str=" ",
> > 
> > what=0x72c5a0 )
> > at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
> 
> Oh, I see. Silly me. Should be fixed in master (at least the crash). Is the
> other issue fixed as well?
> 
> In any case, should also go to 2.3.2-staging.
> 
> Jürgen

Crash with qt4 is gone. Test with CJK is OK here.
tex2lyx test with qt5 passes now too.

Thanks,
Kornel


signature.asc
Description: This is a digitally signed message part.

Re: Failing tex2lyx test

[Jürgen Spitzmüller]

2018-03-13 12:24 GMT+01:00 Kornel Benko :

> #0  strlen () at ../sysdeps/x86_64/strlen.S:106
> #1  0x7f5abf26043c in std::__cxx11::basic_string std::char_traits, std::allocator >::compare(char const*) const
> ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #2  0x005b403d in std::operator== std::allocator > (__lhs=" ",
> __rhs=0x34333231302b2d20  0x34333231302b2d20>)
> at /usr/include/c++/5/bits/basic_string.h:4939
> #3  0x0060c359 in lyx::is_known (str=" ",
> what=0x72c5a0 )
> at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
>

Oh, I see. Silly me. Should be fixed in master (at least the crash). Is the
other issue fixed as well?

In any case, should also go to 2.3.2-staging.

Jürgen

Re: Failing tex2lyx test

[Kornel Benko]

Am Dienstag, 13. März 2018 11:31:19 CET schrieb Jürgen Spitzmüller 
:
> 2018-03-13 10:15 GMT+01:00 Kornel Benko :
> > I see. The CJK.lyx is different here.
> 
> An the CJK.tex is identical to minde?
> 
> > Compiling now with a clean source tree.
> > 
> > NOW, with qt4.8.7, I see a crash from tex2lyx.
> > 
> > Changing to qt5.6.1. Compiling.
> > 
> > No crash. Same problem as before.
> > 
> > But how can tex2lyx be dependent on QT-version?
> 
> No idea. Can you do a backtrace?

Yes.
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x7f5abf26043c in std::__cxx11::basic_string::compare(char const*) const () 
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x005b403d in std::operator== (__lhs=" ", 
__rhs=0x34333231302b2d20 )
at /usr/include/c++/5/bits/basic_string.h:4939
#3  0x0060c359 in lyx::is_known (str=" ", 
what=0x72c5a0 )
at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
#4  0x00629a50 in lyx::(anonymous namespace)::parse_environment 
(p=..., os=..., outer=true, 
last_env="CJK", parent_context=...) at /usr2/src/lyx/lyx-git/src/tex2lyx/
text.cpp:1819
#5  0x0063432f in lyx::parse_text (p=..., os=..., flags=8, outer=true, 
context=..., rdelim="")
at /usr2/src/lyx/lyx-git/src/tex2lyx/text.cpp:3066

Debugging:
$ cd src/tex2lyx/test
$ ./runtests.py /BUILD/BUILDMint18/BuildLyxGitQt4.8.7local-gcc5.4.0/bin/
tex2lyx2.4 ../../../lib/scripts/ /tmp CJK.tex
$ gdb /BUILD/BUILDMint18/BuildLyxGitQt4.8.7local-gcc5.4.0/bin/tex2lyx2.4 core
...
(gdb) bt
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x7f5abf26043c in std::__cxx11::basic_string::compare(char const*) const () 
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x005b403d in std::operator== (__lhs=" ", 
__rhs=0x34333231302b2d20 )
at /usr/include/c++/5/bits/basic_string.h:4939
#3  0x0060c359 in lyx::is_known (str=" ", what=0x72c5a0)
at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
#4  0x00629a50 in lyx::(anonymous namespace)::parse_environment 
(p=..., os=..., outer=true, 
last_env="CJK", parent_context=...) at /usr2/src/lyx/lyx-git/src/tex2lyx/
text.cpp:1819
#5  0x0063432f in lyx::parse_text (p=..., os=..., flags=8, outer=true, 
context=..., rdelim="")
at /usr2/src/lyx/lyx-git/src/tex2lyx/text.cpp:3066
#6  0x00610ada in lyx::(anonymous namespace)::tex2lyx (is=..., os=..., 
encoding="UTF-8", 
outfiledir="/tmp/") at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:876
...
(gdb) b /usr2/src/lyx/lyx-git/src/tex2lyx/text.cpp:1819
Haltepunkt 1 at 0x629a24: file /usr2/src/lyx/lyx-git/src/tex2lyx/text.cpp, line 
1819.
(gdb) R -roundtrip -copyfiles -f /usr2/src/lyx/lyx-git/src/tex2lyx/test/CJK.tex 
/tmp/CJK.lyx
Starting program: /BUILD/BUILDMint18/BuildLyxGitQt4.8.7local-gcc5.4.0/bin/
tex2lyx2.4 -roundtrip -copyfiles -f /usr2/src/lyx/lyx-git/src/tex2lyx/test/
CJK.tex /tmp/CJK.lyx
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Overwriting existing file /tmp/CJK.lyx

Breakpoint 1, lyx::(anonymous namespace)::parse_environment (p=..., os=..., 
outer=true, last_env="CJK", 
parent_context=...) at /usr2/src/lyx/lyx-git/src/tex2lyx/text.cpp:1819
1819if (snip == "\\" || is_known(snip, 
known_escaped_chars))
(gdb) p known_escaped_chars
$1 = {0x72c550 "&", 0x72c552 "_", 0x72c554 "$", 0x72c556 "%", 0x72c558 "#", 
0x72be53 "^", 0x72c55a "{", 
  0x72c55c "}"}

(gdb) b /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
Haltepunkt 3 at 0x60c343: file /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp, 
line 84.
(gdb) c
Continuing.

Breakpoint 3, lyx::is_known (str=" ", what=0x72c560 )
at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
84  if (str == *what)
(gdb) p *what
$2 = 0x72c550 "&"
(gdb) c
Continuing.

Breakpoint 3, lyx::is_known (str=" ", what=0x72c590 )
at /usr2/src/lyx/lyx-git/src/tex2lyx/tex2lyx.cpp:84
84  if (str == *what)
...
(gdb) p *what
$9 = 0x72c55c "}"
(gdb) c
Continuing.

Breakpoint 3, lyx::is_known (str=" ", what=0x72c5a0) at /usr2/src/lyx/lyx-git/
src/tex2lyx/tex2lyx.cpp:84
84  if (str == *what)
(gdb) p *what
$10 = 0x34333231302b2d20 

Kornel



signature.asc
Description: This is a digitally signed message part.

Re: Failing tex2lyx test

[Jürgen Spitzmüller]

2018-03-13 10:15 GMT+01:00 Kornel Benko :

> I see. The CJK.lyx is different here.
>

An the CJK.tex is identical to minde?


> Compiling now with a clean source tree.
> NOW, with qt4.8.7, I see a crash from tex2lyx.
> Changing to qt5.6.1. Compiling.
> No crash. Same problem as before.
>
> But how can tex2lyx be dependent on QT-version?
>

No idea. Can you do a backtrace?

Jürgen


>
> Kornel
>
>

Re: Failing tex2lyx test

[Kornel Benko]

Am Dienstag, 13. März 2018 08:57:42 CET schrieb Jürgen Spitzmüller 
:
> 2018-03-12 20:14 GMT+01:00 Kornel Benko :
> > Apparently you are testing lyx2.3.x.
> 
> No, master, actually. The bugfix of the mentioned commit shows up, but not
> the problem you face. See attached file.
> 
> Jürgen

I see. The CJK.lyx is different here.
Compiling now with a clean source tree.
NOW, with qt4.8.7, I see a crash from tex2lyx.
Changing to qt5.6.1. Compiling.
No crash. Same problem as before.

But how can tex2lyx be dependent on QT-version?

Kornel



signature.asc
Description: This is a digitally signed message part.

Re: Failing tex2lyx test

[Jürgen Spitzmüller]

2018-03-12 20:14 GMT+01:00 Kornel Benko :

> Apparently you are testing lyx2.3.x.


No, master, actually. The bugfix of the mentioned commit shows up, but not
the problem you face. See attached file.

Jürgen


CJK.lyx
Description: application/lyx
%% LyX 2.4.0dev created this file.  For more info, see https://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass[english,english]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{CJKutf8}

\makeatletter
%% User specified LaTeX commands.
\usepackage{babel}


% It is impossible to get the document language because the document could start with a command,
% whitespace, and English word or whatever and the rest is in Japanese. Checking for the first CJK
% environment is no solution because the document could be English and contain only some Japanese.



\makeatother

\usepackage{babel}
\begin{document}
\begin{CJK}{EUC-JP}{}Japanese 

Chinese traditional

 Japanese 

hello 

Chinese simplified 

\begin{CJK}{EUC-JP}{hei} Japanese (this CJK environment will be put
in ERT because LyX supports only one CJK font mapping per document,
this environment uses the \texttt{hei} mapping, and the first mapping
is empty.\end{CJK}

Chinese simplified 

% The following Bg5 encoded text cannot be processed as is by latex.
% It needs to be preprocessed by bg5conv or bg5latex needs to be used.
% See the documentation of CJK.sty for details.

 Big5  
\end{CJK}

\inputencoding{latin9}English

\begin{CJK}{EUC-JP}{}% The following SJIS encoded text cannot be processed as is by latex.
% It needs to be preprocessed by sjisconv or sjislatex needs to be used.
% See the documentation of CJK.sty for details.

 Shift\_JIS  

hello

\begin{CJK}{JIS}{} JIS-code $BF|K\8l$NJ8>O(B\end{CJK}

Korean 
\end{CJK}

\end{document}

Re: Failing tex2lyx test

[Kornel Benko]

Am Montag, 12. März 2018 19:48:39 CET schrieb Jürgen Spitzmüller 
:
> Am Montag, den 12.03.2018, 18:26 +0100 schrieb Kornel Benko:
> > Should do as in case of test-structure.tex
> > 
> > cd src/tex2lyx/test
> > ./runtests.py path-to-tex2lyx ../../../lib/scripts /tmp CJK.tex
> 
> Did that, the result generated in /tmp looks good (does not show the
> problem you describe).
> 
> So:
> > > I need a reproducible case in order to fix this.

Apparently you are testing lyx2.3.x. My mistake.
The error shows with master-tree and master executables.

> Jürgen
> 
Kornel



signature.asc
Description: This is a digitally signed message part.

Re: Failing tex2lyx test

[Jürgen Spitzmüller]

Am Montag, den 12.03.2018, 18:26 +0100 schrieb Kornel Benko:
> Should do as in case of test-structure.tex
>   cd src/tex2lyx/test
>   ./runtests.py path-to-tex2lyx ../../../lib/scripts /tmp CJK.tex

Did that, the result generated in /tmp looks good (does not show the
problem you describe).

So:

> > I need a reproducible case in order to fix this.

Jürgen

> > 
> > Jürgen
> > 
> 
>  Kornel
> 

signature.asc
Description: This is a digitally signed message part

Re: Failing tex2lyx test

[Kornel Benko]

Am Montag, 12. März 2018 18:10:16 CET schrieb Jürgen Spitzmüller 
:
> 2018-03-12 16:05 GMT+01:00 Kornel Benko :
> > The problematic diff looks like:
> > 
> > --- /usr2/src/lyx/lyx-git/src/tex2lyx/test/CJK.lyx.lyx  Sun Mar 11
> > 19:35:25
> > 2018
> > +++ /BUILD/BUILDMint18/BuildLyxGitQt5.6.1local-gcc5.4.0/src/tex2lyx/test/
> > CJK.lyxMon Mar 12 16:03:37 2018
> > @@ -130,7 +130,27 @@
> > 
> >  \end_inset
> >  
> >   Japanese
> > 
> > - (this CJK environment will be put in ERT because LyX supports only one
> > CJK
> > + (this
> > +\begin_inset ERT
> > +status collapsed
> > +
> > +\begin_layout Plain Layout
> > +C
> > +\end_layout
> > +
> > +\end_inset
> > +
> > +JK environment will be put in ERT because LyX supports only one
> > +\begin_inset ERT
> > +status collapsed
> > +
> > +\begin_layout Plain Layout
> > +C
> > +\end_layout
> > +
> > +\end_inset
> > +
> > +JK
> 
> I cannot reproduce. Note that I cannot run the tests themselves (since
> cmake keeps on segfaulting), but when I do a roundtrip on the file
> manually, I do not see the problem.

cmake keeps on segfaulting?
Or is it tex2lyx? Or python?
Should do as in case of test-structure.tex
cd src/tex2lyx/test
./runtests.py path-to-tex2lyx ../../../lib/scripts /tmp CJK.tex

> I need a reproducible case in order to fix this.
> 
> Jürgen
> 
 Kornel



signature.asc
Description: This is a digitally signed message part.

Re: Failing tex2lyx test

[Jürgen Spitzmüller]

2018-03-12 16:05 GMT+01:00 Kornel Benko :

> The problematic diff looks like:
> 
> --- /usr2/src/lyx/lyx-git/src/tex2lyx/test/CJK.lyx.lyx  Sun Mar 11
> 19:35:25
> 2018
> +++ /BUILD/BUILDMint18/BuildLyxGitQt5.6.1local-gcc5.4.0/src/tex2lyx/test/
> CJK.lyxMon Mar 12 16:03:37 2018
> @@ -130,7 +130,27 @@
>  \end_inset
>
>   Japanese
> - (this CJK environment will be put in ERT because LyX supports only one
> CJK
> + (this
> +\begin_inset ERT
> +status collapsed
> +
> +\begin_layout Plain Layout
> +C
> +\end_layout
> +
> +\end_inset
> +
> +JK environment will be put in ERT because LyX supports only one
> +\begin_inset ERT
> +status collapsed
> +
> +\begin_layout Plain Layout
> +C
> +\end_layout
> +
> +\end_inset
> +
> +JK
>


I cannot reproduce. Note that I cannot run the tests themselves (since
cmake keeps on segfaulting), but when I do a roundtrip on the file
manually, I do not see the problem.

I need a reproducible case in order to fix this.

Jürgen



>
> Kornel

Failing tex2lyx test

[Kornel Benko]

Lately, the test "tex2lyx/roundtrip/CJK.tex" fails.
Bisecting and searching for the wrong looking diff led to
***
0f4c9027056a6f4f771382e9ebfc7940274bf5c0 is the first bad commit
commit 0f4c9027056a6f4f771382e9ebfc7940274bf5c0
Author: Juergen Spitzmueller 
Date:   Sat Mar 10 14:58:55 2018 +0100

tex2lyx: make nested CJK parsing slightly less dumb.

Fixes: #9562

:04 04 4cc422bc2f78eddf376ee4d52a4fab54c2b2a397 
a3e56ae7deca45e8959bc1df11530937878816d2 M  src
***

The problematic diff looks like:

--- /usr2/src/lyx/lyx-git/src/tex2lyx/test/CJK.lyx.lyx  Sun Mar 11 19:35:25 
2018
+++ /BUILD/BUILDMint18/BuildLyxGitQt5.6.1local-gcc5.4.0/src/tex2lyx/test/
CJK.lyxMon Mar 12 16:03:37 2018
@@ -130,7 +130,27 @@
 \end_inset

  Japanese
- (this CJK environment will be put in ERT because LyX supports only one CJK
+ (this
+\begin_inset ERT
+status collapsed
+
+\begin_layout Plain Layout
+C
+\end_layout
+
+\end_inset
+
+JK environment will be put in ERT because LyX supports only one
+\begin_inset ERT
+status collapsed
+
+\begin_layout Plain Layout
+C
+\end_layout
+
+\end_inset
+
+JK
  font mapping per document, this environment uses the
 \begin_inset ERT
 status collapsed
***

Kornel

signature.asc
Description: This is a digitally signed message part.