Re: Gnuplot format & converter script
On Tue, Jun 06, 2017 at 12:51:33AM +0200, Tommaso Cucinotta wrote: > from the help message this sounds interesting, I also thought > we could have had various terminal types, and get .gp to .pdf, > to .eps, etc, albeit it's sufficient to convert to one > vectorial format, and LyX knows how to get to the others. > Perhaps among the most useful things, is that 1) you can use this > script to render on screen through the .gp interactive capabilities, > using the same unmodified .gp script that is included in .lyx; > 2) the terminal options, but theburden is that each document > (even each use of .gp scripts) might need slightly different options... > > not really much time in these days, but this script might easily > replace the gnuplot2pdf.py one, with proper invokation in configure.py. > (-t pdf). However, I can't get at a glance the why of all of this > lines' parsing & mangling. guess it's a try to be robust w.r.t. various > different contents of the .gp script. Thanks for taking a look, Tommaso. I CC the author of the script, Koji. He might be able to answer any specific questions whenever you have time. Scott
Re: Gnuplot format & converter script
from the help message this sounds interesting, I also thought we could have had various terminal types, and get .gp to .pdf, to .eps, etc, albeit it's sufficient to convert to one vectorial format, and LyX knows how to get to the others. Perhaps among the most useful things, is that 1) you can use this script to render on screen through the .gp interactive capabilities, using the same unmodified .gp script that is included in .lyx; 2) the terminal options, but theburden is that each document (even each use of .gp scripts) might need slightly different options... not really much time in these days, but this script might easily replace the gnuplot2pdf.py one, with proper invokation in configure.py. (-t pdf). However, I can't get at a glance the why of all of this lines' parsing & mangling. guess it's a try to be robust w.r.t. various different contents of the .gp script. T. On 25/05/2017 19:36, Scott Kostyshak wrote: On Fri, May 05, 2017 at 08:03:02PM -0400, Scott Kostyshak wrote: Tommaso, I found an old script from Koji for gnuplot. Is it useful now? Tommaso, No worries if you don't have time, but I just wanted to make sure you saw this. Scott
Re: Gnuplot format & converter script
On Fri, May 05, 2017 at 08:03:02PM -0400, Scott Kostyshak wrote: > Tommaso, > > I found an old script from Koji for gnuplot. Is it useful now? Tommaso, No worries if you don't have time, but I just wanted to make sure you saw this. Scott signature.asc Description: PGP signature
Re: Gnuplot format & converter script
On Fri, May 05, 2017 at 08:35:48AM +0200, Tommaso Cucinotta wrote: > On 04/05/2017 11:53, Scott Kostyshak wrote: > > > as a consequence, I came up with the attached patch that solves the > > pushed that anyway -- even if copying with images and their conversion is > desired, we shouldn't need dryrun then, should we ? From the definition of dryrun it looks like we shouldn't generate images: /** whether or not to do actual file copying and image conversion * This mode will be used to preview the source code */ So if file copying and image conversion is the only thing that prepareHTMLFile does, then I think your patch is correct. I did not check. signature.asc Description: PGP signature
Re: Gnuplot format & converter script
On Thu, May 04, 2017 at 12:54:47AM +0200, Tommaso Cucinotta wrote: > On 03/05/2017 15:00, Scott Kostyshak wrote: > > Tommaso, I'm still curious if you are planning to implement the gnuplot > > patch for 2.3.0? Or perhaps you already did and I missed it? > > > > The reason I ask is that time is running out to get features into 2.3.0 > > before we should focus completely on bug fixes. > > I had it in my own tommaso/master branch, rebased up to last summer > (LinuxDay@Pisa). > > I thought it was late, but as you're asking, I took the chance to rebase the > patch (trivial), check it's still working as expected, and push [1]. > > I wish someone else could try it out, in case you're short of gnuplot > scripts, you can use the attached sample. Just run LyX, and do > "Insert->Graphics...", then choose the .gp file, then enjoy the plot preview > on screen, as well as in the PDF formatted output. Tommaso, I found an old script from Koji for gnuplot. Is it useful now? Scott #!/usr/bin/env python # -*- coding: utf-8 -*- # file gnuplot.py # This file is an extention to LyX, the document processor. # Licence details can be found in the file COPYING. # \author Koji Yokota # Full author contact details are available in file CREDITS. # This script creates a graphic file from a Gnuplot source file. # It is aimed to be called from external templates. # For it to work correctly, gnuplot must be found in path. # # Gnuplot: http://www.gnuplot.info/ # # IMPORTANT! # gnuplot is known to have a security hole which enables execution of # ANY system commands from the gnuplot source file. Because of this, # usage of this script exposes the system to a potentially severe # security risk if the gnuplot source file contains a malicious code. # PLEASE USE THIS SCRIPT AT YOUR OWN RISK! # # Usage: python gnuplot.py [-t type [-o option]] [-f] \ # [] # # is omitted either when "-s" switch #is specified or when "-t" switch passes "wxt", "xterm", #"windows" or "x11" as an argument. # #-t typeterminal type #-o option comma-separated options for a given terminal type #-s prints out gnuplot source code to be executed to # standard output without actually executing gnuplot #-f allows execution of shell or load/call command # from gnuplot. Note that this exposes the system to # the risk thatmalicious code is executed from the # gnuplot source. #Default values are: #type: "postscript" #option: "eps enhanced color" #(It will be reset to null once type is specified) import re, sys, os, getopt, tempfile class Error: pass ## comment_out() function rearranges logical lines into physical lines, ## comments out unwanted lines and prints other lines as is. def comment_out(line_buffer): # line_buffer can contain multiple commands, so we separate them # into different lines. We want to comment out only related commands. # If multiple commands are in the same line, commenting out one command # also comments out the following commands in the same line. We want to # avoid this. # ";" is used to separate commands, but we must make sure that it # appears outside of quatations. split_buffer = find_quotations.split(line_buffer) # In split_buffer, (4n+3)th item is always strings in quotations, # (2n)th item is always quotation marks, and (4n+1)th item is strings # out of quotations. So we only care about ";" in (4n+1)th item. for i in range(len(split_buffer)): if i % 4 == 0 and find_separator.search(split_buffer[i]): # Replace ';' with '\n' in strings outside of quotations split_buffer[i] = \ find_extra_space.sub( r'\1', find_separator.sub(r'\n', split_buffer[i])) # Reset line_buffer to restructure it from the list split_buffer line_buffer = '' for i in range(0,len(split_buffer)-1): line_buffer = line_buffer + split_buffer[i] line_buffer = line_buffer + split_buffer[len(split_buffer)-1] # As line_buffer is restructured so that a logical line corresponds # to a physical line, we can safely comment out unwanted commands. if find_terminal.search(line_buffer): line_buffer = find_terminal.sub(r'# \1', line_buffer) count_terminal = 1 if find_output.search(line_buffer): line_buffer = find_output.sub(r'# \1', line_buffer) count_output = 1 if not allow_shell_execution and find_shell_execution.search(line_buffer): line_buffer = find_shell_execution.sub(r'# \1', line_buffer) print line_buffer def print_help(script_name): print '\n' + script_name print '\nThis script outputs a graphics file from a Gnuplot source file.\n' print 'Usage:\npython ' + script_name + ' [-t type [-o option]] \\' print '
Re: Gnuplot format & converter script
On 04/05/2017 11:53, Scott Kostyshak wrote: as a consequence, I came up with the attached patch that solves the pushed that anyway -- even if copying with images and their conversion is desired, we shouldn't need dryrun then, should we ? T.
Re: further add-ons for 2.3.0 ? (was: Re: Gnuplot format & converter script)
Am Donnerstag, 4. Mai 2017 um 01:06:44, schrieb Tommaso Cucinotta > As we're on this, a few other things I had in my tommaso/master [1], out of > which: > I'd really love to have 3)... > about 2) I'm stuck with recurring to Emacs-editing the first line of a .lyx > file every time I face that issue :-)... > just pushed 1), needs to be tried by someone > guess 4) might be debatable already for a 3.0 :-). > > ... if there's interest in any of these, it may not be impossible to rebase > over the w.e. :-) ... > 2) commit cb7a69b1 > Author: Tommaso Cucinotta > Date: Wed Oct 19 11:18:10 2016 +0200 > > Tolerate formats that are not supported by lyx2lyx.> Yes. > 3) commit bf3cda7b > Author: Tommaso Cucinotta > Date: Sat Oct 15 01:14:02 2016 +0200 > > Create new graphics from within LyX choosing a sample file to copy > from Feels interesting. > 4) commit 410ce81e > Author: Tommaso Cucinotta > Date: Wed Oct 16 22:55:40 2013 +0100 > > LyX XMPP Chat > > This patch enables XMPP-based chatting within LyX. > > With contributions from Kornel Benko > > Thanks, > > T. Would be nice to get this one into the master tree. Kornel signature.asc Description: This is a digitally signed message part.
Re: Gnuplot format & converter script
On 04/05/2017 11:53, Scott Kostyshak wrote: Assuming we do want to embed the picture on the clipboard, the next but I don't think any of this is actually working, copying from an inset containing a real image file (.jpg) into LibreOffice, it just inserts an XML header line, along with a seemingly useless link to a file:///path/to/image-source.jpg, which is not displayed on screen nor rendered in pdf (from within LibreOffice). Perhaps M$Word users on Win are having a different experience ? T.
Re: Gnuplot format & converter script
On Thu, May 04, 2017 at 07:52:00AM +0200, Tommaso Cucinotta wrote: > On 04/05/2017 01:36, Scott Kostyshak wrote: > > Works well! I found what I think is unexpected behavior: if I select the > > graphics inset (e.g. after I have followed your instructions and clicked > > on "run" and everything looks good), and copy it, I am presented with > > the authorization dialog. [...] Can you reproduce? > > yes, and it is unexpected that the converter question comes on the Copy > operation, not the Paste one! > > In my stack trace I'm seeing > > convert - Converter.cpp:462 > prepareHTMLFile - InsetGraphics.cpp:940 > ... > writeLyXHTMLSource - Buffer.cpp:2239 > putClipboard - CutAndPaste.cpp:574 > copySelection - CutAndPaste.cpp:1042 > > interestingly, just above the copySelection() line, we can see: > > // We do not need to produce images, etc. > runparams.dryrun = true; > > as a consequence, I came up with the attached patch that solves the > annoyance for me, but I'm not expert of this area of the code, so > can please someone review? > (I just checked that I can still copy and paste the graphics as before) Thanks for looking into it. It seems we are generating the image for storing the text/html MIME on the clipboard. That actually doesn't seem strange to me if we want to embed the image in the clipboard. Does that actually work? I tried copy/pasting from LyX with an image to LibreOffice and I don't get the picture. I get a placeholder. If I try from the browser, I get the embedded picture. Assuming we do want to embed the picture on the clipboard, the next question is should we run the converter again? Or if the format we already have stored for the preview is lossless (e.g. vector) then won't it be more efficient to convert from that than to rerun the converter? Perhaps it would just be difficult to do that. Scott signature.asc Description: PGP signature
Re: Gnuplot format & converter script
On 04/05/2017 01:36, Scott Kostyshak wrote: Works well! I found what I think is unexpected behavior: if I select the graphics inset (e.g. after I have followed your instructions and clicked on "run" and everything looks good), and copy it, I am presented with the authorization dialog. [...] Can you reproduce? yes, and it is unexpected that the converter question comes on the Copy operation, not the Paste one! In my stack trace I'm seeing convert - Converter.cpp:462 prepareHTMLFile - InsetGraphics.cpp:940 ... writeLyXHTMLSource - Buffer.cpp:2239 putClipboard - CutAndPaste.cpp:574 copySelection - CutAndPaste.cpp:1042 interestingly, just above the copySelection() line, we can see: // We do not need to produce images, etc. runparams.dryrun = true; as a consequence, I came up with the attached patch that solves the annoyance for me, but I'm not expert of this area of the code, so can please someone review? (I just checked that I can still copy and paste the graphics as before) Thanks, T. >From f3ec463a3a422dbdf98f61add03f68580ccda653 Mon Sep 17 00:00:00 2001 From: Tommaso Cucinotta Date: Thu, 4 May 2017 07:49:07 +0200 Subject: [PATCH] skip graphics conversion when runparams.dryrun is try ... so we avoid the "run converter ?" confirmation dialog for needauth converters simply when copying (Ctrl+C) an InsetGraphics inset. --- src/insets/InsetGraphics.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/insets/InsetGraphics.cpp b/src/insets/InsetGraphics.cpp index 04ed735f..16a482f1 100644 --- a/src/insets/InsetGraphics.cpp +++ b/src/insets/InsetGraphics.cpp @@ -947,7 +947,9 @@ string InsetGraphics::prepareHTMLFile(OutputParams const & runparams) const docstring InsetGraphics::xhtml(XHTMLStream & xs, OutputParams const & op) const { - string const output_file = prepareHTMLFile(op); + string const output_file; + if (!op.dryrun) + prepareHTMLFile(op); if (output_file.empty()) { LYXERR0("InsetGraphics::xhtml: Unable to prepare file `" -- 2.9.3
Re: further add-ons for 2.3.0 ? (was: Re: Gnuplot format & converter script)
On Thu, May 04, 2017 at 01:06:44AM +0200, Tommaso Cucinotta wrote: > As we're on this, a few other things I had in my tommaso/master [1], out of > which: > I'd really love to have 3)... Is it polished? Is there a trac ticket for this one? Or an archived discussion? > about 2) I'm stuck with recurring to Emacs-editing the first line of a .lyx > file every time I face that issue :-)... If I remember correctly, this is the idea that LyX might as well try to open files from future LyX versions, if the author accepts the consequences. Is that right? If so, I think I will be hesitant to accept this one at this point. I think it will need discussion and will need to consider several possible situations. But if you feel strongly about it, continue the discussion on lyx-devel. Who knows---maybe I'm wrong and we could arrive at a consensus soon enough to put it in before the freeze. > just pushed 1), needs to be tried by someone I tried it, just found a minor unexpected behavior that I reported in the thread. > guess 4) might be debatable already for a 3.0 :-). Yes probably we should hold off on that one. Scott signature.asc Description: PGP signature
Re: Gnuplot format & converter script
On Thu, May 04, 2017 at 12:54:47AM +0200, Tommaso Cucinotta wrote: > On 03/05/2017 15:00, Scott Kostyshak wrote: > > Tommaso, I'm still curious if you are planning to implement the gnuplot > > patch for 2.3.0? Or perhaps you already did and I missed it? > > > > The reason I ask is that time is running out to get features into 2.3.0 > > before we should focus completely on bug fixes. > > I had it in my own tommaso/master branch, rebased up to last summer > (LinuxDay@Pisa). > > I thought it was late, but as you're asking, I took the chance to rebase the > patch (trivial), check it's still working as expected, and push [1]. > > I wish someone else could try it out, in case you're short of gnuplot > scripts, you can use the attached sample. Just run LyX, and do > "Insert->Graphics...", then choose the .gp file, then enjoy the plot preview > on screen, as well as in the PDF formatted output. Works well! I found what I think is unexpected behavior: if I select the graphics inset (e.g. after I have followed your instructions and clicked on "run" and everything looks good), and copy it, I am presented with the authorization dialog. I assume this is because of a temporary buffer that LyX creates for the copy or something like that? In any case, the dialog was unexpected to me. Can you reproduce? I don't use gnuplot so I'm glad that you gave the simple example. I tested both gnuplot and knitr together (these both require "dangerous" converters) and everything worked smoothly. Scott signature.asc Description: PGP signature
further add-ons for 2.3.0 ? (was: Re: Gnuplot format & converter script)
As we're on this, a few other things I had in my tommaso/master [1], out of which: I'd really love to have 3)... about 2) I'm stuck with recurring to Emacs-editing the first line of a .lyx file every time I face that issue :-)... just pushed 1), needs to be tried by someone guess 4) might be debatable already for a 3.0 :-). ... if there's interest in any of these, it may not be impossible to rebase over the w.e. :-) [1] 1) commit be98c5e1 Author: Tommaso Cucinotta Date: Mon Oct 17 08:44:16 2016 +0200 Enable graphics generation from external gnuplot scripts. 2) commit cb7a69b1 Author: Tommaso Cucinotta Date: Wed Oct 19 11:18:10 2016 +0200 Tolerate formats that are not supported by lyx2lyx. 3) commit bf3cda7b Author: Tommaso Cucinotta Date: Sat Oct 15 01:14:02 2016 +0200 Create new graphics from within LyX choosing a sample file to copy from. 4) commit 410ce81e Author: Tommaso Cucinotta Date: Wed Oct 16 22:55:40 2013 +0100 LyX XMPP Chat This patch enables XMPP-based chatting within LyX. With contributions from Kornel Benko Thanks, T. On 04/05/2017 00:54, Tommaso Cucinotta wrote: On 03/05/2017 15:00, Scott Kostyshak wrote: Tommaso, I'm still curious if you are planning to implement the gnuplot patch for 2.3.0? Or perhaps you already did and I missed it? The reason I ask is that time is running out to get features into 2.3.0 before we should focus completely on bug fixes. I had it in my own tommaso/master branch, rebased up to last summer (LinuxDay@Pisa). I thought it was late, but as you're asking, I took the chance to rebase the patch (trivial), check it's still working as expected, and push [1]. I wish someone else could try it out, in case you're short of gnuplot scripts, you can use the attached sample. Just run LyX, and do "Insert->Graphics...", then choose the .gp file, then enjoy the plot preview on screen, as well as in the PDF formatted output. Thx, T. [1] commit b474aa5d Author: Tommaso Cucinotta Date: Thu May 4 00:49:18 2017 +0200 Add needauth option to gnuplot->PDF converter introduced in [066edd3c/lyxgit]. commit 066edd3c Author: Tommaso Cucinotta Date: Mon Oct 17 08:44:16 2016 +0200 Enable graphics generation from external gnuplot scripts.
Re: Gnuplot format & converter script
On 03/05/2017 15:00, Scott Kostyshak wrote: Tommaso, I'm still curious if you are planning to implement the gnuplot patch for 2.3.0? Or perhaps you already did and I missed it? The reason I ask is that time is running out to get features into 2.3.0 before we should focus completely on bug fixes. I had it in my own tommaso/master branch, rebased up to last summer (LinuxDay@Pisa). I thought it was late, but as you're asking, I took the chance to rebase the patch (trivial), check it's still working as expected, and push [1]. I wish someone else could try it out, in case you're short of gnuplot scripts, you can use the attached sample. Just run LyX, and do "Insert->Graphics...", then choose the .gp file, then enjoy the plot preview on screen, as well as in the PDF formatted output. Thx, T. [1] commit b474aa5d Author: Tommaso Cucinotta Date: Thu May 4 00:49:18 2017 +0200 Add needauth option to gnuplot->PDF converter introduced in [066edd3c/lyxgit]. commit 066edd3c Author: Tommaso Cucinotta Date: Mon Oct 17 08:44:16 2016 +0200 Enable graphics generation from external gnuplot scripts. sample.gp Description: application/gnuplot
Re: Gnuplot format & converter script
On Sun, Mar 12, 2017 at 11:28:27PM -0400, Scott Kostyshak wrote: > On Fri, Nov 04, 2016 at 11:51:55AM +0100, Tommaso Cucinotta wrote: > > On 04/11/2016 10:31, Jean-Marc Lasgouttes wrote: > > > > enable the feature. Also, I guess that integration of external materials > > > > suffers of similar security risks. > > > Another big hole is Sweave/knitr. > > > > so, guess we deserve a TT: https://www.lyx.org/trac/ticket/10481 > > Hi Tommaso, > > Now that the ticket has been mostly addressed, is there any other > barrier for your gnuplot patch? Tommaso, I'm still curious if you are planning to implement the gnuplot patch for 2.3.0? Or perhaps you already did and I missed it? The reason I ask is that time is running out to get features into 2.3.0 before we should focus completely on bug fixes. Scott signature.asc Description: PGP signature
Re: Gnuplot format & converter script
On Fri, Nov 04, 2016 at 11:51:55AM +0100, Tommaso Cucinotta wrote: > On 04/11/2016 10:31, Jean-Marc Lasgouttes wrote: > > > enable the feature. Also, I guess that integration of external materials > > > suffers of similar security risks. > > Another big hole is Sweave/knitr. > > so, guess we deserve a TT: https://www.lyx.org/trac/ticket/10481 Hi Tommaso, Now that the ticket has been mostly addressed, is there any other barrier for your gnuplot patch? Scott signature.asc Description: PGP signature
Re: Gnuplot format & converter script
Jean-Marc Lasgouttes wrote: >> Are we sure a bomb-virus .lyx file is not already possible as of the >> nowadays LyX features :-) ? > > The solution is probably to mark some converters or templates as > "dangerous" and show warnings to the user the first time they are used > (maybe on a per document basis). Perhaps introduce checkbox in preferences, which allows running third party material like gnuplot/knitr/opening URL in citations, whatver. This will be disabled by default and anyone who enables it is on his own. Pavel
Re: Gnuplot format & converter script
On 04/11/2016 10:31, Jean-Marc Lasgouttes wrote: enable the feature. Also, I guess that integration of external materials suffers of similar security risks. Another big hole is Sweave/knitr. so, guess we deserve a TT: https://www.lyx.org/trac/ticket/10481 T.
Re: Gnuplot format & converter script
Le 04/11/2016 à 10:22, Tommaso Cucinotta a écrit : Guess one could try to filter out execution of external commands before sending them to gnuplot, within the gnuplot2pdf.py, but that might limit functionality (should catch the !cmd syntax, but also the plot "< cmd" syntax, with the latter one being one of my common use-cases :-) ). I am sure that such a filter could be worked around. On a related note, you can already execute external scripts from LaTeX through \write18{...} or \input{|command...}, albeit that seems to need an explicit -shell-escape on the command-line to LaTeX, in order to enable the feature. Also, I guess that integration of external materials suffers of similar security risks. Another big hole is Sweave/knitr. Are we sure a bomb-virus .lyx file is not already possible as of the nowadays LyX features :-) ? The solution is probably to mark some converters or templates as "dangerous" and show warnings to the user the first time they are used (maybe on a per document basis). JMarc
Re: Gnuplot format & converter script
On 03/11/2016 23:02, Pavel Sanda wrote: How your patch deals with scripts like "! rm -rf *" ? At the moment, I'm not sure what the current directory would be, for such a command, if it's the temporary directory where LyX is doing the temp conversions, or the current working directory for the .lyx document being edited, and anyway it could have been a "rm -rf $HOME" or similar :-) Guess one could try to filter out execution of external commands before sending them to gnuplot, within the gnuplot2pdf.py, but that might limit functionality (should catch the !cmd syntax, but also the plot "< cmd" syntax, with the latter one being one of my common use-cases :-) ). On a related note, you can already execute external scripts from LaTeX through \write18{...} or \input{|command...}, albeit that seems to need an explicit -shell-escape on the command-line to LaTeX, in order to enable the feature. Also, I guess that integration of external materials suffers of similar security risks. Are we sure a bomb-virus .lyx file is not already possible as of the nowadays LyX features :-) ? Thanks, T.
Re: Gnuplot format & converter script
Tommaso Cucinotta wrote: > Comments welcome, thanks! How your patch deals with scripts like "! rm -rf *" ? I hope I don't need to write you shouldn't try the snippet above ;) Pavel