Re: inputencoding default again
Am Dienstag, 16. Januar 2007 10:50 schrieb Dov Feldstern:
I just spent two hours playing around with this, and it turns out that
the problem *is* with the way LyX is writing the output. I don't exactly
understand what's going on, but I was able to fix it with a really
simple patch (attached) --- although since I don't exactly understand
the problem, I can't vouch for the patch's correctness. It does work for
me, however.
If you want to see the example file I worked with, I can give it to you,
but I'd rather not post it publicly (it's an old version of my CV). Is
there any way for me to get it to you privately?
Dov sent me the file, and I found the problem. When I added the
switchEncoding call after the paragraph I assumed that outputting
additional newlines would not harm, since we are at a paragraph end
anyway. This is not true in general, if we have a size change LyX will
create code like this:
\par\end{center}{\LARGE \par}
\begin{spacing}{1.20}
If the paragraph needs an encoding change at the end this will become:
\par\end{center}{\LARGE \par}
\inputencoding{latin9}
\begin{spacing}{1.20}
This results in an additional empty paragraph. Therefore this patch avoids
the additional newline with the help of a new flag pending_newline, and
the output will look now like this:
\par\end{center}{\LARGE \par}\inputencoding{latin9}
\begin{spacing}{1.20}
The patch is going in now, since it fixes a regression to 1.4.x (Dov's
document now has identical output with inputenc == default and
with auto). The call of switchEncoding() before the paragraph start does
not need to be changed, similar code is also in 1.4.x, and it does not
harm.
Georg
Index: src/output_latex.C
===
--- src/output_latex.C (Revision 16739)
+++ src/output_latex.C (Arbeitskopie)
@@ -393,14 +393,13 @@ TeXOnePar(Buffer const buf,
} else if (is_command)
os '}';
+ bool pending_newline = false;
switch (style-latextype) {
case LATEX_ITEM_ENVIRONMENT:
case LATEX_LIST_ENVIRONMENT:
if (boost::next(pit) != paragraphs.end()
- (pit-params().depth() boost::next(pit)-params().depth())) {
- os '\n';
- texrow.newline();
- }
+ (pit-params().depth() boost::next(pit)-params().depth()))
+ pending_newline = true;
break;
case LATEX_ENVIRONMENT: {
// if its the last paragraph of the current environment
@@ -416,10 +415,8 @@ TeXOnePar(Buffer const buf,
// fall through possible
default:
// we don't need it for the last paragraph!!!
- if (boost::next(pit) != paragraphs.end()) {
- os '\n';
- texrow.newline();
- }
+ if (boost::next(pit) != paragraphs.end())
+ pending_newline = true;
}
if (!pit-forceDefaultParagraphs()) {
@@ -427,9 +424,12 @@ TeXOnePar(Buffer const buf,
(boost::next(pit) == paragraphs.end()
|| !boost::next(pit)-hasSameLayout(*pit)))
{
- os from_ascii(pit-params().spacing().writeEnvirEnd())
- '\n';
- texrow.newline();
+ if (pending_newline) {
+os '\n';
+texrow.newline();
+ }
+ os from_ascii(pit-params().spacing().writeEnvirEnd());
+ pending_newline = true;
}
}
@@ -439,19 +439,21 @@ TeXOnePar(Buffer const buf,
// we need to reset the language at the end of footnote or
// float.
+ if (pending_newline) {
+ os '\n';
+ texrow.newline();
+ }
if (lyxrc.language_command_end.empty())
os from_ascii(subst(
lyxrc.language_command_begin,
$$lang,
-doc_language-babel()))
- '\n';
+doc_language-babel()));
else
os from_ascii(subst(
lyxrc.language_command_end,
$$lang,
-language-babel()))
- '\n';
- texrow.newline();
+language-babel()));
+ pending_newline = true;
}
// FIXME we switch from the encoding of this paragraph to the
@@ -459,8 +461,9 @@ TeXOnePar(Buffer const buf,
// to take the encoding of the next paragraph into account.
// This may result in some unneeded encoding changes.
basefont = pit-getLayoutFont(bparams, outerfont);
- if (switchEncoding(os, bparams, *(basefont.language()-encoding()),
- outer_encoding)) {
+ switchEncoding(os, bparams, *(basefont.language()-encoding()),
+ outer_encoding);
+ if (pending_newline) {
os '\n';
texrow.newline();
}
Re: inputencoding "default" again
Am Dienstag, 16. Januar 2007 10:50 schrieb Dov Feldstern:
> I just spent two hours playing around with this, and it turns out that
> the problem *is* with the way LyX is writing the output. I don't exactly
> understand what's going on, but I was able to fix it with a really
> simple patch (attached) --- although since I don't exactly understand
> the problem, I can't vouch for the patch's correctness. It does work for
> me, however.
>
> If you want to see the example file I worked with, I can give it to you,
> but I'd rather not post it publicly (it's an old version of my CV). Is
> there any way for me to get it to you privately?
Dov sent me the file, and I found the problem. When I added the
switchEncoding call after the paragraph I assumed that outputting
additional newlines would not harm, since we are at a paragraph end
anyway. This is not true in general, if we have a size change LyX will
create code like this:
\par\end{center}{\LARGE \par}
\begin{spacing}{1.20}
If the paragraph needs an encoding change at the end this will become:
\par\end{center}{\LARGE \par}
\inputencoding{latin9}
\begin{spacing}{1.20}
This results in an additional empty paragraph. Therefore this patch avoids
the additional newline with the help of a new flag pending_newline, and
the output will look now like this:
\par\end{center}{\LARGE \par}\inputencoding{latin9}
\begin{spacing}{1.20}
The patch is going in now, since it fixes a regression to 1.4.x (Dov's
document now has identical output with inputenc == "default" and
with "auto"). The call of switchEncoding() before the paragraph start does
not need to be changed, similar code is also in 1.4.x, and it does not
harm.
Georg
Index: src/output_latex.C
===
--- src/output_latex.C (Revision 16739)
+++ src/output_latex.C (Arbeitskopie)
@@ -393,14 +393,13 @@ TeXOnePar(Buffer const & buf,
} else if (is_command)
os << '}';
+ bool pending_newline = false;
switch (style->latextype) {
case LATEX_ITEM_ENVIRONMENT:
case LATEX_LIST_ENVIRONMENT:
if (boost::next(pit) != paragraphs.end()
- && (pit->params().depth() < boost::next(pit)->params().depth())) {
- os << '\n';
- texrow.newline();
- }
+ && (pit->params().depth() < boost::next(pit)->params().depth()))
+ pending_newline = true;
break;
case LATEX_ENVIRONMENT: {
// if its the last paragraph of the current environment
@@ -416,10 +415,8 @@ TeXOnePar(Buffer const & buf,
// fall through possible
default:
// we don't need it for the last paragraph!!!
- if (boost::next(pit) != paragraphs.end()) {
- os << '\n';
- texrow.newline();
- }
+ if (boost::next(pit) != paragraphs.end())
+ pending_newline = true;
}
if (!pit->forceDefaultParagraphs()) {
@@ -427,9 +424,12 @@ TeXOnePar(Buffer const & buf,
&& (boost::next(pit) == paragraphs.end()
|| !boost::next(pit)->hasSameLayout(*pit)))
{
- os << from_ascii(pit->params().spacing().writeEnvirEnd())
- << '\n';
- texrow.newline();
+ if (pending_newline) {
+os << '\n';
+texrow.newline();
+ }
+ os << from_ascii(pit->params().spacing().writeEnvirEnd());
+ pending_newline = true;
}
}
@@ -439,19 +439,21 @@ TeXOnePar(Buffer const & buf,
// we need to reset the language at the end of footnote or
// float.
+ if (pending_newline) {
+ os << '\n';
+ texrow.newline();
+ }
if (lyxrc.language_command_end.empty())
os << from_ascii(subst(
lyxrc.language_command_begin,
"$$lang",
-doc_language->babel()))
- << '\n';
+doc_language->babel()));
else
os << from_ascii(subst(
lyxrc.language_command_end,
"$$lang",
-language->babel()))
- << '\n';
- texrow.newline();
+language->babel()));
+ pending_newline = true;
}
// FIXME we switch from the encoding of this paragraph to the
@@ -459,8 +461,9 @@ TeXOnePar(Buffer const & buf,
// to take the encoding of the next paragraph into account.
// This may result in some unneeded encoding changes.
basefont = pit->getLayoutFont(bparams, outerfont);
- if (switchEncoding(os, bparams, *(basefont.language()->encoding()),
- outer_encoding)) {
+ switchEncoding(os, bparams, *(basefont.language()->encoding()),
+ outer_encoding);
+ if (pending_newline) {
os << '\n';
texrow.newline();
}
Re: inputencoding default again
Georg Baum wrote:
Am Sonntag, 14. Januar 2007 13:18 schrieb Dov Feldstern:
One strange thing that I'm seeing, though, is that the dvi output when
using auto and default is not identical: I tried this out on a file
which fits really tightly on the page --- when using default it all
fits on one page, but when using auto it doesn't. I don't know if this
matters or not, but it's a little strange. If you're interested in
following this up, I can give you the files, but I don't think that this
is something which should hold anything up.
There are two possible reasons I can think of:
a) the \inputencoding command causes additional whitespace
b) LyX outputs some additional whitespace before or after \inputencoding
commands.
It would be nice if you could check whether b) is the case by manually
removing all whitespace before and/or after \inputencoding commands
leaving paragraph breaks intact of course). Does it then fit on one page?
If yes, we should fix the output of LyX, and I would be interested in the
example file. If not, we cannot do anything.
Georg
Hi!
I just spent two hours playing around with this, and it turns out that
the problem *is* with the way LyX is writing the output. I don't exactly
understand what's going on, but I was able to fix it with a really
simple patch (attached) --- although since I don't exactly understand
the problem, I can't vouch for the patch's correctness. It does work for
me, however.
If you want to see the example file I worked with, I can give it to you,
but I'd rather not post it publicly (it's an old version of my CV). Is
there any way for me to get it to you privately?
Hope this helps!
Dov
Index: src/output_latex.C
===
--- src/output_latex.C (revision 16673)
+++ src/output_latex.C (working copy)
@@ -307,7 +307,7 @@
// account. This may result in some unneeded encoding changes.
if (switchEncoding(os, bparams, outer_encoding,
*(basefont.language()-encoding( {
- os '\n';
+ os %\n;
texrow.newline();
}
@@ -474,7 +474,7 @@
basefont = pit-getLayoutFont(bparams, outerfont);
if (switchEncoding(os, bparams, *(basefont.language()-encoding()),
outer_encoding)) {
- os '\n';
+ os %\n;
texrow.newline();
}
Re: inputencoding default again
Am Dienstag, 16. Januar 2007 10:50 schrieb Dov Feldstern: I just spent two hours playing around with this, and it turns out that the problem *is* with the way LyX is writing the output. That is good, because it means that we can fix it. I don't exactly understand what's going on, but I was able to fix it with a really simple patch (attached) --- although since I don't exactly understand the problem, I can't vouch for the patch's correctness. It does work for me, however. When I did something similar in one of the first versions of the patch it had the effect that a paragraph break vanished, but since I have a test case for that it should be possible to revise the patch so that it works in both cases. If you want to see the example file I worked with, I can give it to you, but I'd rather not post it publicly (it's an old version of my CV). Is there any way for me to get it to you privately? Just send it to my email address. Georg
Re: inputencoding "default" again
Georg Baum wrote:
Am Sonntag, 14. Januar 2007 13:18 schrieb Dov Feldstern:
One strange thing that I'm seeing, though, is that the dvi output when
using "auto" and "default" is not identical: I tried this out on a file
which fits really tightly on the page --- when using "default" it all
fits on one page, but when using "auto" it doesn't. I don't know if this
matters or not, but it's a little strange. If you're interested in
following this up, I can give you the files, but I don't think that this
is something which should hold anything up.
There are two possible reasons I can think of:
a) the \inputencoding command causes additional whitespace
b) LyX outputs some additional whitespace before or after \inputencoding
commands.
It would be nice if you could check whether b) is the case by manually
removing all whitespace before and/or after \inputencoding commands
leaving paragraph breaks intact of course). Does it then fit on one page?
If yes, we should fix the output of LyX, and I would be interested in the
example file. If not, we cannot do anything.
Georg
Hi!
I just spent two hours playing around with this, and it turns out that
the problem *is* with the way LyX is writing the output. I don't exactly
understand what's going on, but I was able to fix it with a really
simple patch (attached) --- although since I don't exactly understand
the problem, I can't vouch for the patch's correctness. It does work for
me, however.
If you want to see the example file I worked with, I can give it to you,
but I'd rather not post it publicly (it's an old version of my CV). Is
there any way for me to get it to you privately?
Hope this helps!
Dov
Index: src/output_latex.C
===
--- src/output_latex.C (revision 16673)
+++ src/output_latex.C (working copy)
@@ -307,7 +307,7 @@
// account. This may result in some unneeded encoding changes.
if (switchEncoding(os, bparams, outer_encoding,
*(basefont.language()->encoding( {
- os << '\n';
+ os << "%\n";
texrow.newline();
}
@@ -474,7 +474,7 @@
basefont = pit->getLayoutFont(bparams, outerfont);
if (switchEncoding(os, bparams, *(basefont.language()->encoding()),
outer_encoding)) {
- os << '\n';
+ os << "%\n";
texrow.newline();
}
Re: inputencoding "default" again
Am Dienstag, 16. Januar 2007 10:50 schrieb Dov Feldstern: > I just spent two hours playing around with this, and it turns out that > the problem *is* with the way LyX is writing the output. That is good, because it means that we can fix it. > I don't exactly > understand what's going on, but I was able to fix it with a really > simple patch (attached) --- although since I don't exactly understand > the problem, I can't vouch for the patch's correctness. It does work for > me, however. When I did something similar in one of the first versions of the patch it had the effect that a paragraph break vanished, but since I have a test case for that it should be possible to revise the patch so that it works in both cases. > If you want to see the example file I worked with, I can give it to you, > but I'd rather not post it publicly (it's an old version of my CV). Is > there any way for me to get it to you privately? Just send it to my email address. Georg
Re: inputencoding default again
Georg Baum wrote: Dov Feldstern wrote: Georg Baum wrote: I propose to change the conversion of files with default inputencoding to be the same as with auto, together with the corresponding changes in LyX itself. The only difference between auto and default in LyX 1.5 would then be that documents with auto would use the inputenc package, but documents with default would not. This is exactly the right behavior as far as Hebrew is concerned. This is in now. Dov, could you please check whether it works for you? Yes, it appears to work. And not only does it work, but the latex file generated using default is significantly smaller than when using auto, because of all the extra encoding instructions that auto has; and the latex processing time is also significantly shorter --- about 4 times as fast for the example I tested! One strange thing that I'm seeing, though, is that the dvi output when using auto and default is not identical: I tried this out on a file which fits really tightly on the page --- when using default it all fits on one page, but when using auto it doesn't. I don't know if this matters or not, but it's a little strange. If you're interested in following this up, I can give you the files, but I don't think that this is something which should hold anything up. So, all in all, default seems to work perfectly, and at least for the situations in which it works --- it appears to even be preferable to auto. Thanks Georg! Dov
Re: inputencoding default again
Am Sonntag, 14. Januar 2007 13:18 schrieb Dov Feldstern: One strange thing that I'm seeing, though, is that the dvi output when using auto and default is not identical: I tried this out on a file which fits really tightly on the page --- when using default it all fits on one page, but when using auto it doesn't. I don't know if this matters or not, but it's a little strange. If you're interested in following this up, I can give you the files, but I don't think that this is something which should hold anything up. There are two possible reasons I can think of: a) the \inputencoding command causes additional whitespace b) LyX outputs some additional whitespace before or after \inputencoding commands. It would be nice if you could check whether b) is the case by manually removing all whitespace before and/or after \inputencoding commands leaving paragraph breaks intact of course). Does it then fit on one page? If yes, we should fix the output of LyX, and I would be interested in the example file. If not, we cannot do anything. Georg
Re: inputencoding "default" again
Georg Baum wrote: Dov Feldstern wrote: Georg Baum wrote: I propose to change the conversion of files with "default" inputencoding to be the same as with "auto", together with the corresponding changes in LyX itself. The only difference between "auto" and "default" in LyX 1.5 would then be that documents with "auto" would use the inputenc package, but documents with "default" would not. This is exactly the right behavior as far as Hebrew is concerned. This is in now. Dov, could you please check whether it works for you? Yes, it appears to work. And not only does it work, but the latex file generated using "default" is significantly smaller than when using "auto", because of all the extra encoding instructions that "auto" has; and the latex processing time is also significantly shorter --- about 4 times as fast for the example I tested! One strange thing that I'm seeing, though, is that the dvi output when using "auto" and "default" is not identical: I tried this out on a file which fits really tightly on the page --- when using "default" it all fits on one page, but when using "auto" it doesn't. I don't know if this matters or not, but it's a little strange. If you're interested in following this up, I can give you the files, but I don't think that this is something which should hold anything up. So, all in all, "default" seems to work perfectly, and at least for the situations in which it works --- it appears to even be preferable to "auto". Thanks Georg! Dov
Re: inputencoding "default" again
Am Sonntag, 14. Januar 2007 13:18 schrieb Dov Feldstern: > One strange thing that I'm seeing, though, is that the dvi output when > using "auto" and "default" is not identical: I tried this out on a file > which fits really tightly on the page --- when using "default" it all > fits on one page, but when using "auto" it doesn't. I don't know if this > matters or not, but it's a little strange. If you're interested in > following this up, I can give you the files, but I don't think that this > is something which should hold anything up. There are two possible reasons I can think of: a) the \inputencoding command causes additional whitespace b) LyX outputs some additional whitespace before or after \inputencoding commands. It would be nice if you could check whether b) is the case by manually removing all whitespace before and/or after \inputencoding commands leaving paragraph breaks intact of course). Does it then fit on one page? If yes, we should fix the output of LyX, and I would be interested in the example file. If not, we cannot do anything. Georg
Re: inputencoding default again
Dov Feldstern wrote: Georg Baum wrote: I propose to change the conversion of files with default inputencoding to be the same as with auto, together with the corresponding changes in LyX itself. The only difference between auto and default in LyX 1.5 would then be that documents with auto would use the inputenc package, but documents with default would not. This is exactly the right behavior as far as Hebrew is concerned. This is in now. Dov, could you please check whether it works for you? Georg
Re: inputencoding "default" again
Dov Feldstern wrote: > Georg Baum wrote: > >> I propose to change the conversion of files with "default" inputencoding >> to be the same as with "auto", together with the corresponding changes in >> LyX itself. The only difference between "auto" and "default" in LyX 1.5 >> would then be that documents with "auto" would use the inputenc package, >> but documents with "default" would not. > > This is exactly the right behavior as far as Hebrew is concerned. This is in now. Dov, could you please check whether it works for you? Georg
inputencoding default again
After the unicode merge to trunk in summer we discussed the question what \inputencoding default in a LyX file was supposed to do. We came to the conclusion then that this simply meant to not use the inputenc package, and simply output the document in 8bit characters as is. Then the question arose how to convert those documents to utf8. LyX 1.4 can ignore the encoding very well, but this is not possible anymore in 1.5, since we can't pass 8bit characters as is anymore. We decided that the best lyx2lyx can do is to assume latin1 encoding for those documents. Meanwhile we found out that the original assumption of lyx2lyx that the whole file is encoded using only one encoding was wrong. I implemented support for multiple encodings that is used for files with \inputencoding default I propose to change the conversion of files with default inputencoding to be the same as with auto, together with the corresponding changes in LyX itself. The only difference between auto and default in LyX 1.5 would then be that documents with auto would use the inputenc package, but documents with default would not. Reasons: - This will make LyX 1.-5 display the same stuff on screen as 1.4, because the encoding for this is derived from the language in 1.4 - It will avoid nasty conversion errors like that decsribed in http://bugzilla.lyx.org/show_bug.cgi?id=3043#c6 Of course that means that all default documents that are already converted to 1.5 would not be valid anymore. IMO this is better than to add a new file format with an intermediate conversion step that might fail as in http://bugzilla.lyx.org/show_bug.cgi?id=3043#c6. Opinions? If this is OK I'll develop a patch (will be easy). Georg
Re: inputencoding default again
Georg Baum wrote: I propose to change the conversion of files with default inputencoding to be the same as with auto, together with the corresponding changes in LyX itself. The only difference between auto and default in LyX 1.5 would then be that documents with auto would use the inputenc package, but documents with default would not. This is exactly the right behavior as far as Hebrew is concerned. Thanks!
Re: inputencoding default again
On Tuesday 09 January 2007 8:02 pm, Georg Baum wrote: Opinions? If this is OK I'll develop a patch (will be easy). Yes, it is OK. That is the price to pay for using the bleeding edge, sometimes bleeds happen. ;-) Georg -- José Abílio
inputencoding "default" again
After the unicode merge to trunk in summer we discussed the question what \inputencoding default in a LyX file was supposed to do. We came to the conclusion then that this simply meant to not use the inputenc package, and simply output the document in 8bit characters as is. Then the question arose how to convert those documents to utf8. LyX 1.4 can ignore the encoding very well, but this is not possible anymore in 1.5, since we can't "pass 8bit characters as is" anymore. We decided that the best lyx2lyx can do is to assume latin1 encoding for those documents. Meanwhile we found out that the original assumption of lyx2lyx that the whole file is encoded using only one encoding was wrong. I implemented support for multiple encodings that is used for files with \inputencoding default I propose to change the conversion of files with "default" inputencoding to be the same as with "auto", together with the corresponding changes in LyX itself. The only difference between "auto" and "default" in LyX 1.5 would then be that documents with "auto" would use the inputenc package, but documents with "default" would not. Reasons: - This will make LyX 1.-5 display the same stuff on screen as 1.4, because the encoding for this is derived from the language in 1.4 - It will avoid nasty conversion errors like that decsribed in http://bugzilla.lyx.org/show_bug.cgi?id=3043#c6 Of course that means that all "default" documents that are already converted to 1.5 would not be valid anymore. IMO this is better than to add a new file format with an intermediate conversion step that might fail as in http://bugzilla.lyx.org/show_bug.cgi?id=3043#c6. Opinions? If this is OK I'll develop a patch (will be easy). Georg
Re: inputencoding "default" again
Georg Baum wrote: I propose to change the conversion of files with "default" inputencoding to be the same as with "auto", together with the corresponding changes in LyX itself. The only difference between "auto" and "default" in LyX 1.5 would then be that documents with "auto" would use the inputenc package, but documents with "default" would not. This is exactly the right behavior as far as Hebrew is concerned. Thanks!
Re: inputencoding "default" again
On Tuesday 09 January 2007 8:02 pm, Georg Baum wrote: > Opinions? If this is OK I'll develop a patch (will be easy). Yes, it is OK. That is the price to pay for using the bleeding edge, sometimes bleeds happen. ;-) > > Georg -- José Abílio
