Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Stephan Witt]

Am 02.11.2015 um 16:44 schrieb Enrico Forestieri :

> On Sun, Nov 01, 2015 at 11:47:42PM -0500, PhilipPirrip wrote:
> 
>> Hi Enrico, thank you  for your response. I hope the others will help us
>> resolve this too.
> 
> Given the overwhelming amount of replies, I doubt anyone is interested
> in this matter.

I'm really interested and I worked on that topic already.
(It was the thread "Yosemite + LyX" on lyx-devel.)
Unfortunately I'm too short on time, ATM.

Stephan

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Enrico Forestieri]

On Tue, Nov 03, 2015 at 02:29:32AM -0500, PhilipPirrip wrote:
> On 11/02/2015 03:16 PM, Enrico Forestieri wrote:
> >>>This last observation suggests what is the minimal change to be performed,
> >>>i.e., let the latex run work as if the document dir was the current
> >>>directory. This means setting the relevant environment variables such
> >>>that the document dir is also scanned, without replicating the setting
> >>>for TEXINPUTS. I'll prepare a patch along these lines for review.
> >>
> >>
> >>I thank you for that! I'd be glad to be the first one to test it in both
> >>Windows and Linux.
> >
> >Please, find attached the patch I come up with. It seems to work for me.
> 
> 
> No surprises here, everything worked as intended:
> Linux, WinXP, Qt 4.8, Qt 5.5, pdf, eps, pdf with eps figures, classicthesis
> template (including biblatex and biber/bibtex8), a few manuals...
> 
> 
> 
> 
> > You mean like Yoda? I don't know whether to feel flattered or offended ;)
> 
> May the force be with you... while into 2.2.0 you're pushing this.
> PLEASE ;)

Scott?

-- 
Enrico

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Scott Kostyshak]

On Tue, Nov 03, 2015 at 08:24:49PM +0100, Enrico Forestieri wrote:
> On Tue, Nov 03, 2015 at 02:29:32AM -0500, PhilipPirrip wrote:
> > On 11/02/2015 03:16 PM, Enrico Forestieri wrote:
> > >>>This last observation suggests what is the minimal change to be 
> > >>>performed,
> > >>>i.e., let the latex run work as if the document dir was the current
> > >>>directory. This means setting the relevant environment variables such
> > >>>that the document dir is also scanned, without replicating the setting
> > >>>for TEXINPUTS. I'll prepare a patch along these lines for review.
> > >>
> > >>
> > >>I thank you for that! I'd be glad to be the first one to test it in both
> > >>Windows and Linux.
> > >
> > >Please, find attached the patch I come up with. It seems to work for me.
> > 
> > 
> > No surprises here, everything worked as intended:
> > Linux, WinXP, Qt 4.8, Qt 5.5, pdf, eps, pdf with eps figures, classicthesis
> > template (including biblatex and biber/bibtex8), a few manuals...
> > 
> > 
> > 
> > 
> > > You mean like Yoda? I don't know whether to feel flattered or offended ;)
> > 
> > May the force be with you... while into 2.2.0 you're pushing this.
> > PLEASE ;)
> 
> Scott?

Please put it in. Thanks for the patch, Enrico.

Philip, thanks for your detailed testing on both Linux and Windows. That
is very important.

Scott

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Richard Heck]

On 11/03/2015 06:38 PM, PhilipPirrip wrote:
Just curious: what and when executes the code in os_win32.cpp 
(os_.cpp in general). Whatever I did to test the patch, these 
functions were not called.


The one Enrico modified is called when you try to view or edit a file 
that is referenced from an external inset. In fact, it does nothing 
except on OSX. A compiler might even notice this and omit the function call.


Richard

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Enrico Forestieri]

On Tue, Nov 03, 2015 at 08:30:53PM -0500, Richard Heck wrote:
> On 11/03/2015 06:38 PM, PhilipPirrip wrote:
> >Just curious: what and when executes the code in os_win32.cpp (os_.cpp
> >in general). Whatever I did to test the patch, these functions were not
> >called.
> 
> The one Enrico modified is called when you try to view or edit a file that
> is referenced from an external inset. In fact, it does nothing except on
> OSX. A compiler might even notice this and omit the function call.

More precisely, that code is used only on Windows and OSX for launching
an editor or viewer registered in the OS for that file type. One can
tell what these are by the fact that the corresponding entries are
denoted by "auto" in the preferences, instead of the name of a program.

-- 
Enrico

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

Just curious: what and when executes the code in os_win32.cpp 
(os_.cpp in general). Whatever I did to test the patch, these 
functions were not called.

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Enrico Forestieri]

On Sun, Nov 01, 2015 at 11:47:42PM -0500, PhilipPirrip wrote:

> Hi Enrico, thank you  for your response. I hope the others will help us
> resolve this too.

Given the overwhelming amount of replies, I doubt anyone is interested
in this matter.

> I'd rather call this a bug in the script than in setProcessEnvironment and
> QProcess. Why would they need to guess what shell they have to execute.

The default shell. As is done when setProcessEnvironment is not called.

> Isn't it polite, at least, for each shell script to have a shebang line?
> http://unix.stackexchange.com/questions/87560/does-the-shebang-determine-the-shell-which-runs-the-script

It is perfectly legal to not have a shebang line and sometimes this is even
necessary. The shebang line must contain an absolute path to the shell that
has to interpret the script. These paths may vary among platforms, so one
can construct a first line that simply reexecutes the script with the
wanted shell located anywhere in the PATH variable. This is what the
"myprog" example script does: it will execute the script using the first
perl interpreter found in PATH. This is not something theoretical and it
seems it is the way texlive scripts are executed, as discussed here:
http://www.mail-archive.com/lyx-devel%40lists.lyx.org/msg169646.html

> Does the same 'bug' happen on Windows, or is this only for the shell 'bugs'
> on *nix?

A similar bug affects Windows, because batch files are not executed by
QProcess.

> Fail: "Exec format error"  in your test example only points to the fact that
> QProcess does not know it's supposed to spawn a shell and execute the script
> in it, nothing more.

No, it means that QProcess doesn't take into account the way a script
can be executed on a given platform. See the link below.

> If instead of myprog script you had a real binary executable (like
> proc->start("/usr/bin/ls") or proc->start("/usr/bin/env")) no "Exec format
> error" would appear in either case. Why?

Because QProcess is buggy:
http://www.mail-archive.com/lyx-devel%40lists.lyx.org/msg169634.html

> >>It maybe does not execute cmd_ if cmd_="some.bat", but will execute
> >>cmd_="cmd.exe /c some.bat". The shell will (hopefully) have the environment
> >>variables set by setProcessEnvironment().
> >>(bat files are not executables by themselves, are they? windows, when you
> >>click on them, starts them by opening cmd.exe first)
> >>This just means that some.bat is not a good cmd, and should be changed to
> >>"cmd.exe /c some.bat"
> >>(now... who and where uses .bat's?)
> >
> >Now I hope you are joking. What would you think if you had to call a
> >converter of yours as "bash myscript" instead of simply "myscript"?
> >Do you expect to have to do that?
> 
> Well, isn't that what Windows does?

No, it will happily execute any file whose extension is listed in PATHEXT:

C:\> echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

Here, the extension plays the same role of a shebang line.

> Isn't that why you had to go back to
> using QProcess when calling a viewer on Windows instead of the script...

No.

> You didn't want that black window to pop out, right?

This can be avoided by other means than using QProcess, which didn't
solve any real problem but only introduced new ones.

> bat files are not
> executables, they can be executed... by an executable, which is in this case
> cmd.exe.

Reasoning along these lines, it also holds true for *nix scripts.

> How many faulty converters are there in LyX? Why do they have to be run the
> same way LaTeX are, by using latexEnvCmdPrefix? Does evince (pdf
> viewer) really need TEXINPUTS variable that gets set by using
> latexEnvCmdPrefix?

There are converters that actually need to be run with the same
environment as the TeX engine. We cannot say which ones they are,
so we should assure that they run with the proper environment set.

> >Batch files are executable and you don't have to type "cmd /c some.bat"
> >to execute them but simply "some". The "cmd /c" prefix is necessary to
> >overcome the silly limitations imposed by QProcess.
> 
> 
> You type some, but Windows has to spawn a shell, cmd.com, to execute it.
> some.bat has no binary code in it, and it is not executable in that sense.

And linux has to do exactly the same in order to run a script.

> Anway... some of the developers here assumed TEXINPUTS covers even bib
> files. (see the posts on \input@path) I wouldn't mind if BIBINPUTS were not
> even configurable, make them . and the current document directory.
> I don't see why this is such a big deal. It's even standard latex behavior,
> it always searches in the current dir. The trouble with LyX is that it
> compiles in a temp dir, but does not copy any files it does not know of.

This last observation suggests what is the minimal change to be performed,
i.e., let the latex run work as if the document dir was the current
directory. This means setting the relevant environment variables such
that the document 

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

On 11/02/2015 03:16 PM, Enrico Forestieri wrote:

This last observation suggests what is the minimal change to be performed,
i.e., let the latex run work as if the document dir was the current
directory. This means setting the relevant environment variables such
that the document dir is also scanned, without replicating the setting
for TEXINPUTS. I'll prepare a patch along these lines for review.



I thank you for that! I'd be glad to be the first one to test it in both
Windows and Linux.


Please, find attached the patch I come up with. It seems to work for me.



No surprises here, everything worked as intended:
Linux, WinXP, Qt 4.8, Qt 5.5, pdf, eps, pdf with eps figures, 
classicthesis template (including biblatex and biber/bibtex8), a few 
manuals...





> You mean like Yoda? I don't know whether to feel flattered or offended ;)

May the force be with you... while into 2.2.0 you're pushing this.
PLEASE ;)

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Enrico Forestieri]

On Mon, Nov 02, 2015 at 12:26:36PM -0500, PhilipPirrip wrote:

> On 11/02/2015 10:44 AM, Enrico Forestieri wrote:
> >On Sun, Nov 01, 2015 at 11:47:42PM -0500, PhilipPirrip wrote:
> >
> >>Hi Enrico, thank you  for your response. I hope the others will help us
> >>resolve this too.
> >
> >Given the overwhelming amount of replies, I doubt anyone is interested
> >in this matter.
> 
> Oh no, no, you're wrong. This is what I received from one of the well known
> developers: "Enrico is the Jedi Master of such things. I'd suggest you
> re-send this just to him."

You mean like Yoda? I don't know whether to feel flattered or offended ;)

> >>Anway... some of the developers here assumed TEXINPUTS covers even bib
> >>files. (see the posts on \input@path) I wouldn't mind if BIBINPUTS were not
> >>even configurable, make them . and the current document directory.
> >>I don't see why this is such a big deal. It's even standard latex behavior,
> >>it always searches in the current dir. The trouble with LyX is that it
> >>compiles in a temp dir, but does not copy any files it does not know of.
> >
> >This last observation suggests what is the minimal change to be performed,
> >i.e., let the latex run work as if the document dir was the current
> >directory. This means setting the relevant environment variables such
> >that the document dir is also scanned, without replicating the setting
> >for TEXINPUTS. I'll prepare a patch along these lines for review.
> 
> 
> I thank you for that! I'd be glad to be the first one to test it in both
> Windows and Linux.

Please, find attached the patch I come up with. It seems to work for me.

> Philip Pirrip wrote on 2015-09-02:
> > Since the way \input@path is being used in the biblatex hack from
> > http://wiki.lyx.org/BibTeX/Biblatex is barely legal, maybe LyX should
> > define its own command, say \LyX@basepath that'd contain the absolute
> > path to the master .lyx document (no qoutes, no curly braces - only for
> > the brave).
> > Would you be willing to do that? (being that biblatex support is nowhere
> > on the horizon yet)
> 
> Georg: After all these discussions it is clear that something like this is
> needed.
> However, I do not want to clutter all documents with that (since 95% of the
> users don't need it). I would prefer a solution where we extend the layout
> definition language so that you could use a placeholder that will be
> replaced by LyX with the master document path. Then everybody who needs this
> path could write a module that pulls in the path into a LaTeX macro in the
> most simple case, or in more advanced cases it could directly be used in the
> preamble code that needs it.

I still think this is not necessary. Instead of fiddling with \input@path,
if you are willing to add the --enable-write18 option to your call for
latex, the following lines in the preamble should do the trick:

\usepackage{catchfile,ifplatform}
\immediate\write18{kpsewhich --var-value=TEXINPUTS > \jobname.tmp}
\CatchFileDef{\temp}{\jobname.tmp}{\endlinechar=-1}
\ifwindows
  \def\getdocdir#1;#2;#3;#4\next{\gdef\docdir{#3}}
\else
  \def\getdocdir#1:#2:#3\next{\gdef\docdir{#2}}
\fi
\expandafter\getdocdir\temp\next

After that, you have the path to the document dir in the macro \docdir.
The only catch is that TEXINPUTS might be customized, but then one
could use BIBINPUTS instead, if the patch is accepted.

-- 
Enrico
diff --git a/src/support/filetools.cpp b/src/support/filetools.cpp
index 204c842..310e621 100644
--- a/src/support/filetools.cpp
+++ b/src/support/filetools.cpp
@@ -710,8 +710,12 @@ string latexEnvCmdPrefix(string const & path, string const 
& lpath)
string texinputs_prefix = lyxrc.texinputs_prefix.empty() ? string()
: os::latex_path_list(
replaceCurdirPath(path, lyxrc.texinputs_prefix));
+   string const allother_prefix = os::latex_path_list(path);
string const sep = string(1, os::path_separator(os::TEXENGINE));
string const texinputs = getEnv("TEXINPUTS");
+   string const bibinputs = getEnv("BIBINPUTS");
+   string const bstinputs = getEnv("BSTINPUTS");
+   string const texfonts = getEnv("TEXFONTS");
 
if (use_lpath) {
string const abslpath = FileName::isAbsolute(lpath)
@@ -727,13 +731,28 @@ string latexEnvCmdPrefix(string const & path, string 
const & lpath)
 
if (os::shell() == os::UNIX)
return "env TEXINPUTS=\"." + sep + texinputs_prefix
- + sep + texinputs + "\" ";
+  + sep + texinputs + "\" "
++ "BIBINPUTS=\"." + sep + allother_prefix
+  + sep + bibinputs + "\" "
++ "BSTINPUTS=\"." + sep + allother_prefix
+  + sep + bstinputs + "\" "
++ "TEXFONTS=\"."  + sep + allother_prefix
+   

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Scott Kostyshak]

On Mon, Nov 02, 2015 at 09:16:41PM +0100, Enrico Forestieri wrote:
> On Mon, Nov 02, 2015 at 12:26:36PM -0500, PhilipPirrip wrote:
> 
> > On 11/02/2015 10:44 AM, Enrico Forestieri wrote:
> > >On Sun, Nov 01, 2015 at 11:47:42PM -0500, PhilipPirrip wrote:
> > >
> > >>Hi Enrico, thank you  for your response. I hope the others will help us
> > >>resolve this too.
> > >
> > >Given the overwhelming amount of replies, I doubt anyone is interested
> > >in this matter.
> > 
> > Oh no, no, you're wrong. This is what I received from one of the well known
> > developers: "Enrico is the Jedi Master of such things. I'd suggest you
> > re-send this just to him."
> 
> You mean like Yoda? I don't know whether to feel flattered or offended ;)

It is indeed ambiguous. Darth Vader was also a jedi I believe... :)

Scott

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

On 11/02/2015 10:44 AM, Enrico Forestieri wrote:

On Sun, Nov 01, 2015 at 11:47:42PM -0500, PhilipPirrip wrote:


Hi Enrico, thank you  for your response. I hope the others will help us
resolve this too.


Given the overwhelming amount of replies, I doubt anyone is interested
in this matter.


Oh no, no, you're wrong. This is what I received from one of the well 
known developers: "Enrico is the Jedi Master of such things. I'd suggest 
you re-send this just to him."

I'd say they all trust your judgment.



You type some, but Windows has to spawn a shell, cmd.com, to execute it.
some.bat has no binary code in it, and it is not executable in that sense.


And linux has to do exactly the same in order to run a script.


With one difference: windows has one single shell, and linux - how 
many?, half a million?





Anway... some of the developers here assumed TEXINPUTS covers even bib
files. (see the posts on \input@path) I wouldn't mind if BIBINPUTS were not
even configurable, make them . and the current document directory.
I don't see why this is such a big deal. It's even standard latex behavior,
it always searches in the current dir. The trouble with LyX is that it
compiles in a temp dir, but does not copy any files it does not know of.


This last observation suggests what is the minimal change to be performed,
i.e., let the latex run work as if the document dir was the current
directory. This means setting the relevant environment variables such
that the document dir is also scanned, without replicating the setting
for TEXINPUTS. I'll prepare a patch along these lines for review.



I thank you for that! I'd be glad to be the first one to test it in both 
 Windows and Linux.







BTW; I once before suggested there should be a command similar to 
\input@path defined, that will contain the path to the current document 
folder with absolutely no latex formatting in it.
That could be used in modules for LaTeX packages that have no support in 
LyX. The fact that LyX compiles its documents in a temp folder makes the 
implementation of such (underground, but still) enhancements very hard. 
Georg, are you reading this?



Philip Pirrip wrote on 2015-09-02:
> Since the way \input@path is being used in the biblatex hack from
> http://wiki.lyx.org/BibTeX/Biblatex is barely legal, maybe LyX should
> define its own command, say \LyX@basepath that'd contain the absolute
> path to the master .lyx document (no qoutes, no curly braces - only for
> the brave).
> Would you be willing to do that? (being that biblatex support is nowhere
> on the horizon yet)

Georg: After all these discussions it is clear that something like this 
is needed.

However, I do not want to clutter all documents with that (since 95% of the
users don't need it). I would prefer a solution where we extend the layout
definition language so that you could use a placeholder that will be
replaced by LyX with the master document path. Then everybody who needs 
this

path could write a module that pulls in the path into a LaTeX macro in the
most simple case, or in more advanced cases it could directly be used in 
the

preamble code that needs it.

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

Hi Enrico, thank you  for your response. I hope the others will help us 
resolve this too.




I propose another simple test, it will show you that env variables do get
set by setProcessEnvironment(). Attaching the modified files.
Hope you'll trust me now ;)


There is nothing to trust but something to understand. A script without a
shebang line is executed by the current shell and QProcess doesn't take
that into account. Your example works because you add a shebang to your
script. This has a real impact and can cause hard to understand bugs.
Please, try reading the thread here:
http://www.mail-archive.com/lyx-devel%40lists.lyx.org/msg169545.html
to see that this causes real problems.




I'd rather call this a bug in the script than in setProcessEnvironment 
and QProcess. Why would they need to guess what shell they have to 
execute. Isn't it polite, at least, for each shell script to have a 
shebang line?

http://unix.stackexchange.com/questions/87560/does-the-shebang-determine-the-shell-which-runs-the-script



Does the same 'bug' happen on Windows, or is this only for the shell 
'bugs' on *nix?



Something tells me we're never gonna agree, so I'll rest my case. 
Nothing has to be changed in SystemcallPrivate::startProcess part of the 
code for the BIBINPUTS to work. It would be nice, I think, for many 
other reasons, but that's totally up to you.




Fail: "Exec format error"  in your test example only points to the fact 
that QProcess does not know it's supposed to spawn a shell and execute 
the script in it, nothing more.
If instead of myprog script you had a real binary executable (like 
proc->start("/usr/bin/ls") or proc->start("/usr/bin/env")) no "Exec 
format error" would appear in either case. Why?












It maybe does not execute cmd_ if cmd_="some.bat", but will execute
cmd_="cmd.exe /c some.bat". The shell will (hopefully) have the environment
variables set by setProcessEnvironment().
(bat files are not executables by themselves, are they? windows, when you
click on them, starts them by opening cmd.exe first)
This just means that some.bat is not a good cmd, and should be changed to
"cmd.exe /c some.bat"
(now... who and where uses .bat's?)


Now I hope you are joking. What would you think if you had to call a
converter of yours as "bash myscript" instead of simply "myscript"?
Do you expect to have to do that?



Well, isn't that what Windows does? Isn't that why you had to go back to 
using QProcess when calling a viewer on Windows instead of the script... 
You didn't want that black window to pop out, right? bat files are not 
executables, they can be executed... by an executable, which is in this 
case cmd.exe.  I'm not pretending I know everything about this, I'm 
telling all this from my limited experience.


How many faulty converters are there in LyX? Why do they have to be run 
the same way LaTeX are, by using latexEnvCmdPrefix? Does evince 
(pdf viewer) really need TEXINPUTS variable that gets set by using 
latexEnvCmdPrefix?






Batch files are executable and you don't have to type "cmd /c some.bat"
to execute them but simply "some". The "cmd /c" prefix is necessary to
overcome the silly limitations imposed by QProcess.



You type some, but Windows has to spawn a shell, cmd.com, to execute it. 
some.bat has no binary code in it, and it is not executable in that sense.




In a more conservative approach, BIBINPUTS support could be added by the
patch I proposed on http://www.lyx.org/trac/ticket/2645


This can be done, if now the limit for a command line on Windows is 8191
characters as documented at https://support.microsoft.com/en-us/kb/830473
but you are forgetting BSTINPUTS and TEXFONTS.


Did anyone complain about the lack of these two? I don't think many people
would need them.  The reason I need them are all these ugly hacks with
\input@path here http://wiki.lyx.org/BibTeX/Biblatex


The fact that these two are not useful for you doesn't mean that they
are not useful. And even if people doesn't ask for them means nothing.
It may simply be that they are not aware of them. As an example, see
http://www.lyx.org/trac/ticket/6634 where TEXFONTS was not explicitly
requested but would have solved that problem.



I know, but by the same argument nothing's worth fixing before we can 
fix EVERYTHING. biblatex support won't be possible until it's total, no 
one wants partial solutions - hope it's not that what you're saying.
Anway... some of the developers here assumed TEXINPUTS covers even bib 
files. (see the posts on \input@path) I wouldn't mind if BIBINPUTS were 
not even configurable, make them . and the current document directory.
I don't see why this is such a big deal. It's even standard latex 
behavior, it always searches in the current dir. The trouble with LyX is 
that it compiles in a temp dir, but does not copy any files it does not 
know of.





Points to be discussed are:

- Should those variables simply replicate the setting for TEXINPUTS?
   · Consider that 

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

Hi Enrico, thank you  for your response. I hope the others will help us 
resolve this too.




I propose another simple test, it will show you that env variables do get
set by setProcessEnvironment(). Attaching the modified files.
Hope you'll trust me now ;)


There is nothing to trust but something to understand. A script without a
shebang line is executed by the current shell and QProcess doesn't take
that into account. Your example works because you add a shebang to your
script. This has a real impact and can cause hard to understand bugs.
Please, try reading the thread here:
http://www.mail-archive.com/lyx-devel%40lists.lyx.org/msg169545.html
to see that this causes real problems.




I'd rather call this a bug in the script than in setProcessEnvironment 
and QProcess. Why would they need to guess what shell they have to 
execute. Isn't it polite, at least, for each shell script to have a 
shebang line?

http://unix.stackexchange.com/questions/87560/does-the-shebang-determine-the-shell-which-runs-the-script



Does the same 'bug' happen on Windows, or is this only for the shell 
'bugs' on *nix?



Something tells me we're never gonna agree, so I'll rest my case. 
Nothing has to be changed in SystemcallPrivate::startProcess part of the 
code for the BIBINPUTS to work. It would be nice, I think, for many 
other reasons, but that's totally up to you.




Fail: "Exec format error"  in your test example only points to the fact 
that QProcess does not know it's supposed to spawn a shell and execute 
the script in it, nothing more.
If instead of myprog script you had a real binary executable (like 
proc->start("/usr/bin/ls") or proc->start("/usr/bin/env")) no "Exec 
format error" would appear in either case. Why?












It maybe does not execute cmd_ if cmd_="some.bat", but will execute
cmd_="cmd.exe /c some.bat". The shell will (hopefully) have the environment
variables set by setProcessEnvironment().
(bat files are not executables by themselves, are they? windows, when you
click on them, starts them by opening cmd.exe first)
This just means that some.bat is not a good cmd, and should be changed to
"cmd.exe /c some.bat"
(now... who and where uses .bat's?)


Now I hope you are joking. What would you think if you had to call a
converter of yours as "bash myscript" instead of simply "myscript"?
Do you expect to have to do that?



Well, isn't that what Windows does? Isn't that why you had to go back to 
using QProcess when calling a viewer on Windows instead of the script... 
You didn't want that black window to pop out, right? bat files are not 
executables, they can be executed... by an executable, which is in this 
case cmd.exe.  I'm not pretending I know everything about this, I'm 
telling all this from my limited experience.


How many faulty converters are there in LyX? Why do they have to be run 
the same way LaTeX are, by using latexEnvCmdPrefix? Does evince 
(pdf viewer) really need TEXINPUTS variable that gets set by using 
latexEnvCmdPrefix?






Batch files are executable and you don't have to type "cmd /c some.bat"
to execute them but simply "some". The "cmd /c" prefix is necessary to
overcome the silly limitations imposed by QProcess.



You type some, but Windows has to spawn a shell, cmd.com, to execute it. 
some.bat has no binary code in it, and it is not executable in that sense.




In a more conservative approach, BIBINPUTS support could be added by the
patch I proposed on http://www.lyx.org/trac/ticket/2645


This can be done, if now the limit for a command line on Windows is 8191
characters as documented at https://support.microsoft.com/en-us/kb/830473
but you are forgetting BSTINPUTS and TEXFONTS.


Did anyone complain about the lack of these two? I don't think many people
would need them.  The reason I need them are all these ugly hacks with
\input@path here http://wiki.lyx.org/BibTeX/Biblatex


The fact that these two are not useful for you doesn't mean that they
are not useful. And even if people doesn't ask for them means nothing.
It may simply be that they are not aware of them. As an example, see
http://www.lyx.org/trac/ticket/6634 where TEXFONTS was not explicitly
requested but would have solved that problem.



I know, but by the same argument nothing's worth fixing before we can 
fix EVERYTHING. biblatex support won't be possible until it's total, no 
one wants partial solutions - hope it's not that what you're saying.
Anway... some of the developers here assumed TEXINPUTS covers even bib 
files. (see the posts on \input@path) I wouldn't mind if BIBINPUTS were 
not even configurable, make them . and the current document directory.
I don't see why this is such a big deal. It's even standard latex 
behavior, it always searches in the current dir. The trouble with LyX is 
that it compiles in a temp dir, but does not copy any files it does not 
know of.





Points to be discussed are:

- Should those variables simply replicate the setting for TEXINPUTS?
   · Consider that 

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Scott Kostyshak]

On Sun, Nov 01, 2015 at 11:47:42PM -0500, PhilipPirrip wrote:
> 
> I hope you all (developers) understand we're grateful users who are paying
> it forward by helping you (or trying ourselves to) make LyX better.

I don't think there's any doubt about this. Thank you for your time and
effort, Philip!

> I personally don't need any of these enhancements. I've been working on
> another popular project that just switched to using biblatex. It will be
> disappointing if it would take LyX another 9 years
> (http://www.lyx.org/trac/ticket/2645) to support it, even by just defining
> BININPUTS properly.

I agree that it would be great for LyX to have proper biblatex support.
See also http://www.lyx.org/trac/ticket/4065.

Scott

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Enrico Forestieri]

On Thu, Oct 29, 2015 at 09:36:24PM -0400, PhilipPirrip wrote:

> Thanks for responding, Enrico. I appreciate it!
> 
> 
> >I am sorry, but it still doesn't work.
> >As you can see, now it doesn't work because setProcessEnvironment() is used.
> 
> I don't understand what that script (myprog) is exactly supposed to do, but
> I believe the problem is in it rather than setProcessEnvironment().
> As I said, the patch (be it good or bad), did work for me on Linux,
> BIBINPUTS env. variable did have effect as my .bib files were found only
> when that variable was defined.
> I propose another simple test, it will show you that env variables do get
> set by setProcessEnvironment(). Attaching the modified files.
> 
> Hope you'll trust me now ;)

There is nothing to trust but something to understand. A script without a
shebang line is executed by the current shell and QProcess doesn't take
that into account. Your example works because you add a shebang to your
script. This has a real impact and can cause hard to understand bugs.
Please, try reading the thread here:
http://www.mail-archive.com/lyx-devel%40lists.lyx.org/msg169545.html
to see that this causes real problems.

> >However, even when https://bugreports.qt.io/browse/QTBUG-19885 is solved
> >(if ever), we would still be stuck with the current way of setting the
> >environment because QProcess does not execute batch files on Windows and
> >doing as you suggest would introduce a regression.
> 
> It maybe does not execute cmd_ if cmd_="some.bat", but will execute
> cmd_="cmd.exe /c some.bat". The shell will (hopefully) have the environment
> variables set by setProcessEnvironment().
> (bat files are not executables by themselves, are they? windows, when you
> click on them, starts them by opening cmd.exe first)
> This just means that some.bat is not a good cmd, and should be changed to
> "cmd.exe /c some.bat"
> (now... who and where uses .bat's?)

Now I hope you are joking. What would you think if you had to call a
converter of yours as "bash myscript" instead of simply "myscript"?
Do you expect to have to do that?

Batch files are executable and you don't have to type "cmd /c some.bat"
to execute them but simply "some". The "cmd /c" prefix is necessary to
overcome the silly limitations imposed by QProcess.

> >>In a more conservative approach, BIBINPUTS support could be added by the
> >>patch I proposed on http://www.lyx.org/trac/ticket/2645
> >
> >This can be done, if now the limit for a command line on Windows is 8191
> >characters as documented at https://support.microsoft.com/en-us/kb/830473
> >but you are forgetting BSTINPUTS and TEXFONTS.
> 
> Did anyone complain about the lack of these two? I don't think many people
> would need them.  The reason I need them are all these ugly hacks with
> \input@path here http://wiki.lyx.org/BibTeX/Biblatex

The fact that these two are not useful for you doesn't mean that they
are not useful. And even if people doesn't ask for them means nothing.
It may simply be that they are not aware of them. As an example, see
http://www.lyx.org/trac/ticket/6634 where TEXFONTS was not explicitly
requested but would have solved that problem.

> >Unfortunately, there was
> >no discussion about the way it should be implemented. I mean, while
> >always setting TEXINPUTS seems reasonable, I don't think it is the same
> >for all other variables, and I am actually not really sure that carrying
> >the settings of TEXINPUTS is a good idea. Maybe we can use the following
> >scheme:
> 
> TEXINPUTS is now by default set to ".:/home/user/current_folder/"
> (. stands for the temporary LyX folder, I believe)
> BIBINPUTS is now set to nothing; it would be nice to have it set, at least,
> to the same two paths as TEXINPUTS
> .bib files will still be searched in 'standard bib database directories'
> (that in 2015, I bet, no one is using)
> 
> Both of these are for the people (majority, I dare to say) that keep their
> lyx and tex and bib and png and ... files in the same folder (for the sake
> of portability and easier backups)
> 
> 
> I therefore do not see the need for this:
> 
> >- set BIBINPUTS if the directory /bib exists
> >- set BSTINPUTS if the directory /bst exists
> >- set TEXFONTS if the directory /fonts exists
> >so that one can put the relevant files in an appropriate subdir in the
> >document dir.

Ok, this is your opinion, but I have not heard any other one. I will not
introduce any change if there is no explicit consent.

Points to be discussed are:

- Should those variables simply replicate the setting for TEXINPUTS?
  · Consider that TEXINPUTS is customizable, so it may be unexpected
that the other variables are set to the same paths and this could
cause problems. Think for example if someone customizes TEXINPUTS
for a recursive search in a directory and suddenly some unrelated
files are now used instead of the system ones.

- Should they be customizable, instead?

- If they are not to be customizable, should we 

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[Enrico Forestieri]

On Wed, Oct 28, 2015 at 08:12:44PM -0400, PhilipPirrip wrote:
> I've been playing with SystemcallPrivate::startProcess (in Systemcall.cpp)
> and latexEnvCmdPrefix (in filetools.cpp), in an attempt to resolve the issue
> with missing BIBINPUTS, a 9 year old request at
> http://www.lyx.org/trac/ticket/2645
> 
> Enrico mentioned there that setEnvironment couldn't be used to set the
> environments due to a bug, and that BIBINPUTS and TEXINPUTS can only be
> defined in shell escape commands of latexEnvCmdPrefix.
> Now that setEnvironment is deprecated, setProcessEnvironment can be used,
> and it turned out that it works... at least for me, on Linux.

I am sorry, but it still doesn't work. You can verify this by yourself as
follows. Put the 3 attached files in the same directory and make executable
the script myprog (chmod 755 myprog). Then execute the following commands:
1) qmake test-env.pro
2) make
3) ./test-env

As you can see, it works because setProcessEnvironment() is not used.

Now do:
1) make clean
2) qmake QMAKE_CXXFLAGS+=-DSETENV test-env.pro
3) make
4) ./test-env

As you can see, now it doesn't work because setProcessEnvironment() is used.

However, even when https://bugreports.qt.io/browse/QTBUG-19885 is solved
(if ever), we would still be stuck with the current way of setting the
environment because QProcess does not execute batch files on Windows and
doing as you suggest would introduce a regression.

> In a more conservative approach, BIBINPUTS support could be added by the
> patch I proposed on http://www.lyx.org/trac/ticket/2645

This can be done, if now the limit for a command line on Windows is 8191
characters as documented at https://support.microsoft.com/en-us/kb/830473
but you are forgetting BSTINPUTS and TEXFONTS. Unfortunately, there was
no discussion about the way it should be implemented. I mean, while
always setting TEXINPUTS seems reasonable, I don't think it is the same
for all other variables, and I am actually not really sure that carrying
the settings of TEXINPUTS is a good idea. Maybe we can use the following
scheme:
- set BIBINPUTS if the directory /bib exists
- set BSTINPUTS if the directory /bst exists
- set TEXFONTS if the directory /fonts exists
so that one can put the relevant files in an appropriate subdir in the
document dir.

> Anyhow, please consider having BIBINPUTS defined in LyX 2.2. PLEASE

;)

I have not so much time ATM, but I will see what I can do.

-- 
Enrico
eval '(exit $?0)' && eval 'exec perl -S $0 ${1+"$@"}' && eval 'exec perl -S $0 
$argv:q'
  if 0;

print "it works";
exit(0);
#include 
#include 
#include 

int main()
{
QProcess *proc = new QProcess();
#ifdef SETENV
proc->setProcessEnvironment(QProcessEnvironment::systemEnvironment());
#endif
proc->setProcessChannelMode(QProcess::MergedChannels);
proc->start("./myprog");
if (proc->waitForFinished())
qDebug() << "Output:" << proc->readAll();
else
qDebug() << "Fail:" << proc->errorString();
proc->close();
delete proc;
return 0;
}
HEADERS =
SOURCES = test-env.cpp

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

Thanks for responding, Enrico. I appreciate it!



I am sorry, but it still doesn't work.
As you can see, now it doesn't work because setProcessEnvironment() is used.


I don't understand what that script (myprog) is exactly supposed to do, 
but I believe the problem is in it rather than setProcessEnvironment().
As I said, the patch (be it good or bad), did work for me on Linux, 
BIBINPUTS env. variable did have effect as my .bib files were found only 
when that variable was defined.
I propose another simple test, it will show you that env variables do 
get set by setProcessEnvironment(). Attaching the modified files.


Hope you'll trust me now ;)



However, even when https://bugreports.qt.io/browse/QTBUG-19885 is solved
(if ever), we would still be stuck with the current way of setting the
environment because QProcess does not execute batch files on Windows and
doing as you suggest would introduce a regression.


It maybe does not execute cmd_ if cmd_="some.bat", but will execute 
cmd_="cmd.exe /c some.bat". The shell will (hopefully) have the 
environment variables set by setProcessEnvironment().
(bat files are not executables by themselves, are they? windows, when 
you click on them, starts them by opening cmd.exe first)
This just means that some.bat is not a good cmd, and should be changed 
to "cmd.exe /c some.bat"

(now... who and where uses .bat's?)




In a more conservative approach, BIBINPUTS support could be added by the
patch I proposed on http://www.lyx.org/trac/ticket/2645


This can be done, if now the limit for a command line on Windows is 8191
characters as documented at https://support.microsoft.com/en-us/kb/830473
but you are forgetting BSTINPUTS and TEXFONTS.


Did anyone complain about the lack of these two? I don't think many 
people would need them.  The reason I need them are all these ugly hacks 
with \input@path here http://wiki.lyx.org/BibTeX/Biblatex




 Unfortunately, there was

no discussion about the way it should be implemented. I mean, while
always setting TEXINPUTS seems reasonable, I don't think it is the same
for all other variables, and I am actually not really sure that carrying
the settings of TEXINPUTS is a good idea. Maybe we can use the following
scheme:


TEXINPUTS is now by default set to ".:/home/user/current_folder/"
(. stands for the temporary LyX folder, I believe)
BIBINPUTS is now set to nothing; it would be nice to have it set, at 
least, to the same two paths as TEXINPUTS
.bib files will still be searched in 'standard bib database directories' 
(that in 2015, I bet, no one is using)


Both of these are for the people (majority, I dare to say) that keep 
their lyx and tex and bib and png and ... files in the same folder (for 
the sake of portability and easier backups)



I therefore do not see the need for this:


- set BIBINPUTS if the directory /bib exists
- set BSTINPUTS if the directory /bst exists
- set TEXFONTS if the directory /fonts exists
so that one can put the relevant files in an appropriate subdir in the
document dir.










Anyhow, please consider having BIBINPUTS defined in LyX 2.2. PLEASE



;)
I have not so much time ATM, but I will see what I can do.



That's why I'm helping you ;) PLEASE

(wasn't the whole LyX built by the people who didn't have much time AthatM)

#!/bin/bash
echo ""
echo "SOME_ENV_VARIABLE"
printenv SOME_ENV_VARIABLE
echo "SOME_OTHER_ENV_VARIABLE" #will not be defined
printenv SOME_OTHER_ENV_VARIABLE

#include 
#include 
#include 

int main()
{
QProcess* proc = new QProcess();;
#ifdef SETENV
QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
env.insert("SOME_ENV_VARIABLE", "see, it works!");
proc->setProcessEnvironment(env);
#endif
proc->setProcessChannelMode(QProcess::MergedChannels);
proc->start("./myprog");
if (proc->waitForFinished())
	qDebug() << "Output:" << proc->readAll();
else
	qDebug() << "Fail:" << proc->errorString();
proc->close();
delete proc;
return 0;
}

Re: [patch] use setProcessEnvironment, not shell escape (cmd / env)

[PhilipPirrip]

Thanks for responding, Enrico. I appreciate it!



I am sorry, but it still doesn't work.
As you can see, now it doesn't work because setProcessEnvironment() is used.


I don't understand what that script (myprog) is exactly supposed to do, 
but I believe the problem is in it rather than setProcessEnvironment().
As I said, the patch (be it good or bad), did work for me on Linux, 
BIBINPUTS env. variable did have effect as my .bib files were found only 
when that variable was defined.
I propose another simple test, it will show you that env variables do 
get set by setProcessEnvironment(). Attaching the modified files.


Hope you'll trust me now ;)



However, even when https://bugreports.qt.io/browse/QTBUG-19885 is solved
(if ever), we would still be stuck with the current way of setting the
environment because QProcess does not execute batch files on Windows and
doing as you suggest would introduce a regression.


It maybe does not execute cmd_ if cmd_="some.bat", but will execute 
cmd_="cmd.exe /c some.bat". The shell will (hopefully) have the 
environment variables set by setProcessEnvironment().
(bat files are not executables by themselves, are they? windows, when 
you click on them, starts them by opening cmd.exe first)
This just means that some.bat is not a good cmd, and should be changed 
to "cmd.exe /c some.bat"

(now... who and where uses .bat's?)




In a more conservative approach, BIBINPUTS support could be added by the
patch I proposed on http://www.lyx.org/trac/ticket/2645


This can be done, if now the limit for a command line on Windows is 8191
characters as documented at https://support.microsoft.com/en-us/kb/830473
but you are forgetting BSTINPUTS and TEXFONTS.


Did anyone complain about the lack of these two? I don't think many 
people would need them.  The reason I need them are all these ugly hacks 
with \input@path here http://wiki.lyx.org/BibTeX/Biblatex




 Unfortunately, there was

no discussion about the way it should be implemented. I mean, while
always setting TEXINPUTS seems reasonable, I don't think it is the same
for all other variables, and I am actually not really sure that carrying
the settings of TEXINPUTS is a good idea. Maybe we can use the following
scheme:


TEXINPUTS is now by default set to ".:/home/user/current_folder/"
(. stands for the temporary LyX folder, I believe)
BIBINPUTS is now set to nothing; it would be nice to have it set, at 
least, to the same two paths as TEXINPUTS
.bib files will still be searched in 'standard bib database directories' 
(that in 2015, I bet, no one is using)


Both of these are for the people (majority, I dare to say) that keep 
their lyx and tex and bib and png and ... files in the same folder (for 
the sake of portability and easier backups)



I therefore do not see the need for this:


- set BIBINPUTS if the directory /bib exists
- set BSTINPUTS if the directory /bst exists
- set TEXFONTS if the directory /fonts exists
so that one can put the relevant files in an appropriate subdir in the
document dir.










Anyhow, please consider having BIBINPUTS defined in LyX 2.2. PLEASE



;)
I have not so much time ATM, but I will see what I can do.



That's why I'm helping you ;) PLEASE

(wasn't the whole LyX built by the people who didn't have much time AthatM)

#!/bin/bash
echo ""
echo "SOME_ENV_VARIABLE"
printenv SOME_ENV_VARIABLE
echo "SOME_OTHER_ENV_VARIABLE" #will not be defined
printenv SOME_OTHER_ENV_VARIABLE

#include 
#include 
#include 

int main()
{
QProcess* proc = new QProcess();;
#ifdef SETENV
QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
env.insert("SOME_ENV_VARIABLE", "see, it works!");
proc->setProcessEnvironment(env);
#endif
proc->setProcessChannelMode(QProcess::MergedChannels);
proc->start("./myprog");
if (proc->waitForFinished())
	qDebug() << "Output:" << proc->readAll();
else
	qDebug() << "Fail:" << proc->errorString();
proc->close();
delete proc;
return 0;
}