from:"Tommaso Cucinotta"


On 18/08/2017 02:09, Tommaso Cucinotta wrote:

no more time for today... but perhaps the cache wipe-out work-around is already 
good enough.


pushed, seems acceptable (except for 0_tmp_tmp_*, 1_tmp_tmp_*, etc...)

T.

Re: [LyX/master] oops, git is playing games with me.


On 17/08/2017 21:49, Pavel Sanda wrote:

To recap the current problems from my POV:
1. en/ should have it's own directory as a normal language
because xhtml output target produces lot of images which clutter
root dir (please use xhtml, not html).


sure, easy.


2. exported icons still contain original path in the filename.
ideally we want fixed filenames for images so when you regenerate
documentation only real changes need to be committed.


tracked down somewhat, 2 problems:
1) conversion cache: if the image (footnote.png) is found in the cache, then 
its associated absolute path is pulled in;
   workaround: wipe out ~/.lyx-trunk/cache, or equivalently run with a custom 
temporary userdir when converting manuals

still, the image is converted with the full pathname of the temporary folder where 
the conversion is being done (no conversion in this case, as it's a .png -> 
.png, but still a lot of code is executed, the file is copied, etc.), being smth. 
like /tmp/tmp.X/orig_filename.png, resulting in an exported image filename 
like: 0_tmp_tmp_X_orig_filename.png which seems still sub-optimal; this comes 
from:

2) when loading a graphics inset, LyX stores into the inset params().filename 
the absolute file path of the references image file, not the relative path; 
this can be avoided with [1], but it's not sufficient, because still LyX 
references the 0_tmp_tmp_* filename instead.

no more time for today... but perhaps the cache wipe-out work-around is already 
good enough.

T.

[1]

diff --git a/src/insets/InsetGraphics.cpp b/src/insets/InsetGraphics.cpp
index 3cc550b5..2aa1b8ff 100644
--- a/src/insets/InsetGraphics.cpp
+++ b/src/insets/InsetGraphics.cpp
@@ -299,7 +299,7 @@ void InsetGraphics::read(Lexer & lex)
 {
lex.setContext("InsetGraphics::read");
//lex >> "Graphics";
-   readInsetGraphics(lex, buffer(), true, params_);
+   readInsetGraphics(lex, buffer(), false, params_);
graphic_->update(params().as_grfxParams());
 }
 
@@ -568,7 +568,7 @@ string InsetGraphics::prepareFile(OutputParams const & runparams) const

if (params().filename.empty())
return string();
 
-   string const orig_file = params().filename.absFileName();

+   string const orig_file = 
params().filename.relFileName(buffer().filePath());
// this is for dryrun and display purposes, do not use latexFilename
string const rel_file = 
params().filename.relFileName(buffer().filePath());
 
@@ -902,7 +902,7 @@ string InsetGraphics::prepareHTMLFile(OutputParams const & runparams) const
 
string const to   = findTargetFormat(from, runparams);

string const ext  = theFormats().extension(to);
-   string const orig_file = params().filename.absFileName();
+   string const orig_file = 
params().filename.relFileName(buffer().filePath());
string output_file = onlyFileName(temp_file.absFileName());
LYXERR(Debug::GRAPHICS, "\t we have: from " << from << " to " << to);
LYXERR(Debug::GRAPHICS, "\tthe orig file is: " << orig_file);

Re: LyX version 2.3.0 (beta 1)


On 17/08/2017 20:25, Scott Kostyshak wrote:

An overview of the new features can be found here:

   http://wiki.lyx.org/LyX/NewInLyX23


AFAICR, the new shell-escape [+minted] support made it to the 2.3, but it's not 
mentioned (yet).

T.

Re: [LyX/master] oops, git is playing games with me.


Thx Pavel for pushing this. I'd like to add to the script the capability to 
generate the index in PmWiki format, so that its output can be copy'n'pasted 
straight to

  http://wiki.lyx.org/LyX/Manuals2

What'd'u think?

T.

On 17/08/2017 01:25, Pavel Sanda wrote:

commit e8b324a984e6cbbeb4ea02ba553bd5da520eb3f8
Author: Pavel Sanda 
Date:   Thu Aug 17 01:24:44 2017 +0200

 oops, git is playing games with me.
---
  development/tools/generate_manuals_for_web.sh |3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/development/tools/generate_manuals_for_web.sh 
b/development/tools/generate_manuals_for_web.sh
index 2ae337c..b28a638 100755
--- a/development/tools/generate_manuals_for_web.sh
+++ b/development/tools/generate_manuals_for_web.sh
@@ -18,8 +18,7 @@
  #   $TOC - final index.html that links all converted manuals
  #   $TMP - temporary folder where all the conversion is done
  
-#MAIN_DOCS=${MAIN_DOCS:-"Intro Tutorial UserGuide Math Additional Customization Shortcuts LFUNs"}

-MAIN_DOCS=${MAIN_DOCS:-"Math"}
+MAIN_DOCS=${MAIN_DOCS:-"Intro Tutorial UserGuide Math Additional Customization 
Shortcuts LFUNs"}
  LYX=${LYX:-${PWD}/src/lyx}
  OUT=${OUT:-$HOME/web/lyxdoc}
  TOC=${TOC:-lyxdoc/index.html}

wiki - Category link pages

2017-08-15 Thread Tommaso Cucinotta


Hi,

I took some time to clean-up the empty template that was annoyingly showing up 
when clicking on the bottom category link of each wiki page, namely:

"" ...

which I replaced with a simple

!!! Pages in Category: {*$Name}

So, now those links should look like as having some more meaningful and usable 
contents:

  http://wiki.lyx.org/Category/Development
  http://wiki.lyx.org/Category/Documentation
  http://wiki.lyx.org/Category/Glossary
  http://wiki.lyx.org/Category/Keyboard
  http://wiki.lyx.org/Category/Layouts
  http://wiki.lyx.org/Category/Manuals

[just in case u notice anything mistakenly broken on the wiki category links...]

T.

Re: 404 on manuals on lyx.org

2017-08-15 Thread Tommaso Cucinotta


On 15/08/2017 22:11, Uwe Stöhr wrote:

So fine with me if you change the order.


done. Also, I gave a try to pmwiki tables, to get to a more compact form:

  http://wiki.lyx.org/LyX/Manuals2
  (might be visually improved with some .lyx/.pdf icons, plus country-code 
flags, but don't know how to attach images to the lyx.org wiki)

this might be a start for providing URLs for more languages (some of the) 
manuals are available into.

Feedback welcome, thanks.

T.

Re: 404 on manuals on lyx.org

2017-08-15 Thread Tommaso Cucinotta


On 15/08/2017 02:53, Uwe Stöhr wrote:

I set it now again and updated the manuals to LyX 2.2.3:

http://wiki.lyx.org/LyX/Manuals


Great! One minor comment: probably the order of manuals might be exactly 
reversed, to have easier stuff first, and more advanced stuff last.
Also, PDFs are a great help for people looking for info on the web, but HTML 
exports would probably be even better on the web...

Out of curiosity: this page links only manuals for a few languages, what about 
the plethora of others ? Is that because these languages are the only ones for 
which we have a complete set of manuals ?

my2c,

T.

Re: 404 on manuals on lyx.org


On 13/08/2017 16:58, Tommaso Cucinotta wrote:

On my system, despite having installed texlive-lang-all, not all of them 
succeed in compilation.


I was missing xetex for the ja (and presumably he) languages.

T.

Re: 404 on manuals on lyx.org


On 13/08/2017 17:33, Pavel Sanda wrote:

Maybe we are seeing different phenomena. Here EmbeddedObjects compiles

[...]

Apart from that it looks sort of OK. TeXLive 2012 here :)


yes, the difference seems that you're exporting xhtml, whilst I was exporting 
.html.

Just checked that the .xhtml out seems ok, instead the .html export is 
completely broken, I get on terminal a number of Empty picture! errors, and I 
just get a number of random PICT in the output, without anything / any text 
visible (see attached).

T.

  

  
 


  

  
  

  

  

  

LY X’s detailed Figure, Table,
Floats, Notes, Boxes and External
Material manual
   by the LY X Team*
  
   
   
  
   
 Version 2.2.x 

August 13, 2017
*If you have comments or error corrections, please send them to the LY X Documentation
mailing list: lyx-d...@lists.lyx.org
  

  

  

  
  

  

  

  


Contents

Re: 404 on manuals on lyx.org


On 13/08/2017 12:48, Pavel Sanda wrote:

You need latin,french&german to build the main (english) manuals?


yes, latin for the "ipse lorum blah blah", french I suspect accidentally tags a 
few English paragraphs in one of the manuals, whilst german is part of the description of 
some of the inter-language features of LyX (umlauts, accents, ...) can't remember more 
now.


-) Intro, UserGuide, Tutorial, Development, Math, Customization,
Additional, LFUNs compile fine (pdf and html);
-) major: I'm able to pdf-compile EmbeddedObjects, but HTML compilation has
problems;


I'd say it's 'solvable' if the .lyx file which is dumped via listings
is changed to one which is much much shorter. It won't be fixed, but
it wouldn't dump 30 pages of nonsense either...


for now, EmbeddedObjects has been excluded from the auto-build in the script I 
just sent.
I'm not understanding your explanation above of the problem.


that has easy workaround along those lines (not supposed to be working out of 
the box):
SVN_OUT=~/web/stabledoc/
W=/tmp/stabledoc
B=~/lyx/stable/src/lyx


got it, took into account into the script just sent.

Thanks,

T.

Re: 404 on manuals on lyx.org


Here you go, script attached, it compiles the main manuals for all available 
languages, and auto-builds an index.html linking them all. Feel free to add it 
to the LyX git (couldn't identify a suitable location in the git tree for such 
a tool).

On my system, despite having installed texlive-lang-all, not all of them 
succeed in compilation.

Result as can be seen at (file lyxdoc.tar.gz, once it's done uploading):

  https://drive.google.com/open?id=0B6A-8XYYe1nYUlg3U2YyQm0xQVU

T.

On 13/08/2017 12:48, Pavel Sanda wrote:

Tommaso Cucinotta wrote:

On 11/08/2017 16:51, Pavel Sanda wrote:

I can prepare web corner for it on www.lyx.org if you decide to do it.



I quickly check the link you posted and the results seems good enough to
me to push it on the web if we can somewhat automatize it.


I gave a try to manual compilation of manuals :-), and here's the
experience:
-) minors along the way: I was missing texlive-lang-german, french, latin,


You need latin,french&german to build the main (english) manuals?


and texlive-humanities;


That's to be expected.


-) Intro, UserGuide, Tutorial, Development, Math, Customization,
Additional, LFUNs compile fine (pdf and html);
-) major: I'm able to pdf-compile EmbeddedObjects, but HTML compilation has
problems;


I'd say it's 'solvable' if the .lyx file which is dumped via listings
is changed to one which is much much shorter. It won't be fixed, but
it wouldn't dump 30 pages of nonsense either...


-) major: MergedManuals compilation complains about not being able to find
SpecialParagraphShape.tex or smth like that; that comes from
Additional.lyx, but Additional standalone compiles without problems, and
the .tex file is there, so why does it fail when embedded within
MergedManuals ?


We don't need the merged manuals at this stage, just the main manuals would be 
good enough.


-) minor: converted HTML files include images with export names alike
55_home_tommaso_lyx-trunk-ws_lyx_*, which includes the absolute path I was
exporting from -- ideally, we shouldn't include any (mangled) absolute path
when exporting (but only (mangled) relative paths), is there a TT/# for
this ?


that has easy workaround along those lines (not supposed to be working out of 
the box):
SVN_OUT=~/web/stabledoc/
W=/tmp/stabledoc
B=~/lyx/stable/src/lyx

#prepare
cd lyx; mkdir $W; cp -r lib/doc $W ; cd ${W}/doc

#export
for manual in tutorial.lyx userguide.lyx ... ; do
  $B -E pdf $manual
  $B -E xhtml $manual
done

#sum results html; similarly for pdf
for outman in tutorialdir userguidedir ... ; do
  rm -r ${SVN_OUT}/$outman
  cp -r $outman ${SVN_OUT}/$outman
done


It just needs someone to tune the above script for proper names,
switches and so on...
The rest is deploying to web via svn (I can take care of myself if I get
the above camera-ready).

Pavel





manuals_for_web.sh
Description: application/shellscript

Re: 404 on manuals on lyx.org

2017-08-12 Thread Tommaso Cucinotta


On 12/08/2017 15:52, Tommaso Cucinotta wrote:

I gave a try to manual compilation of manuals :-), and here's the experience:
-) minors along the way: I was missing texlive-lang-german, french, latin, and 
texlive-humanities;
-) Intro, UserGuide, Tutorial, Development, Math, Customization, Additional, 
LFUNs compile fine (pdf and html);
-) major: I'm able to pdf-compile EmbeddedObjects, but HTML compilation has 
problems;
-) major: MergedManuals compilation complains about not being able to find 
SpecialParagraphShape.tex or smth like that; that comes from Additional.lyx, 
but Additional standalone compiles without problems, and the .tex file is 
there, so why does it fail when embedded within MergedManuals ?


forgot:

-) minor: converted HTML files include images with export names alike 
55_home_tommaso_lyx-trunk-ws_lyx_*, which includes the absolute path I was 
exporting from -- ideally, we shouldn't include any (mangled) absolute path 
when exporting (but only (mangled) relative paths), is there a TT/# for this ?

Thx,

T.

Re: Can postscript code be embedded in a LyX document?

2017-08-12 Thread Tommaso Cucinotta


On 12/08/2017 10:21, Guenter Milde wrote:

This may change if we develop an "all inclusive" LyX document format.


I used to remember a discussion about that, to the point to be mistakenly 
believing we did support an inclusive/embedded .lyx format, but I'm likely not 
remembering well, or confusing with the document modules thing.

T.

Re: 404 on manuals on lyx.org

2017-08-12 Thread Tommaso Cucinotta


On 11/08/2017 16:51, Pavel Sanda wrote:

I can prepare web corner for it on www.lyx.org if you decide to do it.


There's an autotest exporting manuals in a few formats that used to be 
troublesome (xhtml and lyx16x) in the past, in

  development/autotests/export-in.sh

I gave a try to manual compilation of manuals :-), and here's the experience:
-) minors along the way: I was missing texlive-lang-german, french, latin, and 
texlive-humanities;
-) Intro, UserGuide, Tutorial, Development, Math, Customization, Additional, 
LFUNs compile fine (pdf and html);
-) major: I'm able to pdf-compile EmbeddedObjects, but HTML compilation has 
problems;
-) major: MergedManuals compilation complains about not being able to find 
SpecialParagraphShape.tex or smth like that; that comes from Additional.lyx, 
but Additional standalone compiles without problems, and the .tex file is 
there, so why does it fail when embedded within MergedManuals ?

I dropped the result here:

  
https://drive.google.com/drive/folders/0B6A-8XYYe1nYUlg3U2YyQm0xQVU?usp=sharing

T.

Re: 404 on manuals on lyx.org

2017-08-11 Thread Tommaso Cucinotta


On 11/08/2017 15:55, Christian Ridderström wrote:

simply to ensure that LyX doesn't fail when reading and exporting the manuals 
as PDFs.


from my viewpoint, it would be good to have the manuals just available and 
searchable on-line, i.e., when I'm looking for something on a search engine, 
being redirected to either related topics throughout the manuals, or to related 
discussions throughout our mailing list archives, or related pages on the LyX 
wiki, would all be good things to have.
Albeit, someone pointed out previously that we purposely stop bots to avoid our 
web server to be overloaded, which is understandable but not nice for 
visibility of LyX.

Thanks,

T.

Re: What/how are converters triggered?

2017-08-11 Thread Tommaso Cucinotta


On 11/08/2017 11:44, Christian Ridderström wrote:

Do we have a description somewhere, perhaps in the source, as to what and how 
the different converters are triggered?

How are files found for which converters are to be triggered?
How is the type of the file determined?


Format.cpp: Formats::getFormatFromFile(filename)

if (HAVE_MAGIC && file exists) {
  magic_open() et al. => ~~ what `file filename' does on Linux
  ~~> detect MIME type from magic library, then look-up a file format matching 
it
  return format if found, otherwise fall back to what follows:
}
getExtension(filename)
if (no extension)
guessFormatFromContents(filename) {
  this is our own version of libmagic, looking up a prefix in the file 
contents,
  recognizes most popular image formats, plus ZIP, GZIP, PKZIP and COMPRESS 
archives
}
plus, quite a number of heuristics for correct detection of compressed formats,
postscript formats, latex formats and I'm not sure what else :-).

my2c,

T.

Re: LyX manuals online?


On 10/08/2017 22:27, Christian Ridderström wrote:

Do we have the LyX manuals online, either as PDF or web pages?


I just asked this as well, which I guess provides a partial reply to your Q.

T.
--- Begin Message ---

Hi,

I just noticed these broken links

  http://wiki.lyx.org/LyX/Manuals#download
  http://ftp.lyx.de/Documentation/en/Customization.lyx
  http://ftp.lyx.de/Documentation/en/Customization.pdf

guess they're all broken.

There's also a seemingly outdated (from y 2009) merged manual available at:

  http://wiki.lyx.org/LyX/Documentation#toc2
  
https://wiki.lyx.org/uploads//LyX/Manuals/1.4.2/merged_manuals/merged_manuals-1.4.2.pdf

However, besides fixing the links, considering the powerful conversion 
capabilities of LyX, I'd propose to keep at least some manuals accessible as 
HTML pages on the web, rather than, or in addition to, .lyx/.pdf downloads.

Thanks,

T.
--- End Message ---

shell-escape UI (and needauth)


Hi,

I just had some time to finally try out the new shell-escape feature and UI, so 
I thought to drop a few comments:

- the UI visible red icon is fantastic :-)! guess we can re-use it when the 
user authorized needauth converters for the currently open doc;
- the shell-escape would resemble needauth (and perhaps share some -- or even 
all -- of the security prefs) if:
  - the document contained a settings requiring the use of shell-escape for its 
correct formatting/conversion;
  - we had a preference option that forbids shell-escape altogether
  - [optional] we had a preference option that allows it always

This way:
-)  a document requiring a 'needauth' converter would be equivalent 
(security-wise) to one requiring shell-escape;
-) if a doc ever contained both ERT insets with \write18{...}, and scripts 
requiring a 'needauth' converter, the user would be prompted once about the 
risk;
-) LyX would share the same memory in its session, namely that a given .lyx 
file is trusted to run external programs
-) the defaults on a system would be to have these features disabled by default;
-) even when enabling these unsafe features, the user would normally be 
prompted for each new doc;
-) [optional] the unsafe user might turn off the security prompts altogether 
(albeit being warned about the risks of doing so and being prompted to confirm 
his/her awareness)

My2c, thanks,

T.

Re: Can postscript code be embedded in a LyX document?


On 10/08/2017 11:02, Tommaso Cucinotta wrote:

On 09/08/2017 09:05, Guenter Milde wrote:

For EPS to PDF this means we could preferably use "repstopdf" instead of
"epstopdf".

   "repstopdf" is the version "whitelisted" in texmf.cnf for use with the
   restricted shell access.
   On my Debian system, it is just a symlink to epstopdf:


Ubuntu 17.04:

$ epstopdf -h | grep restricted
   --restricted   use restricted mode(default: false)

$ repstopdf -h | grep restricted
   --restricted   use restricted mode(default: true)

Guess this is the standard. So, would we need the attached patch ?


However, on my Ubuntu, even the regular epstopdf fails to write on the file-system, 
unless one explicitly adds the "--nosafer" option.

The --nosafer option allows for writing files while parsing .[e]ps contents, namely 
allowing for the PS "(w) file def" construct to not fail. On the other hand, 
the --restricted vs non-restricted option doesn't have effect on that specific capability.

T.

Re: Can postscript code be embedded in a LyX document?


On 09/08/2017 09:05, Guenter Milde wrote:

For EPS to PDF this means we could preferably use "repstopdf" instead of
"epstopdf".

   "repstopdf" is the version "whitelisted" in texmf.cnf for use with the
   restricted shell access.
   
   On my Debian system, it is just a symlink to epstopdf:


Ubuntu 17.04:

$ epstopdf -h | grep restricted
  --restricted   use restricted mode(default: false)

$ repstopdf -h | grep restricted
  --restricted   use restricted mode(default: true)

Guess this is the standard. So, would we need the attached patch ?

    T.
commit b5c5a5a8
Author: Tommaso Cucinotta 
Date:   Thu Aug 10 10:59:47 2017 +0200

Switch to restricted repstopdf instead of regular epstopdf, for security reasons.

diff --git a/lib/configure.py b/lib/configure.py
index 8e7fedbb..46d797e6 100644
--- a/lib/configure.py
+++ b/lib/configure.py
@@ -996,15 +996,15 @@ def checkConverterEntries():
 checkProg('an EMF -> PDF converter', [inkscape_name + ' --file=$$i --export-area-drawing --without-gui --export-pdf=$$o'],
 rc_entry = [ r'\converter emfpdf6"%%"	""'])
 # Only define a converter to pdf6 for graphics
-checkProg('an EPS -> PDF converter', ['epstopdf'],
-rc_entry = [ r'\converter epspdf6   "epstopdf --outfile=$$o $$i"	""'])
+checkProg('an EPS -> PDF converter', ['repstopdf'],
+rc_entry = [ r'\converter epspdf6   "repstopdf --outfile=$$o $$i"	""'])
 #
 checkProg('an EPS -> PNG converter', ['magick $$i $$o', 'convert $$i $$o'],
 rc_entry = [ r'\converter epspng"%%"	""'])
 #
 # no agr -> pdf6 converter, since the pdf library used by gracebat is not
 # free software and therefore not compiled in in many installations.
-# Fortunately, this is not a big problem, because we will use epstopdf to
+# Fortunately, this is not a big problem, because we will use repstopdf to
 # convert from agr to pdf6 via eps without loss of quality.
 checkProg('a Grace -> Image converter', ['gracebat'],
 rc_entry = [

404 on manuals on lyx.org

2017-08-08 Thread Tommaso Cucinotta


Hi,

I just noticed these broken links

  http://wiki.lyx.org/LyX/Manuals#download
  http://ftp.lyx.de/Documentation/en/Customization.lyx
  http://ftp.lyx.de/Documentation/en/Customization.pdf

guess they're all broken.

There's also a seemingly outdated (from y 2009) merged manual available at:

  http://wiki.lyx.org/LyX/Documentation#toc2
  
https://wiki.lyx.org/uploads//LyX/Manuals/1.4.2/merged_manuals/merged_manuals-1.4.2.pdf

However, besides fixing the links, considering the powerful conversion 
capabilities of LyX, I'd propose to keep at least some manuals accessible as 
HTML pages on the web, rather than, or in addition to, .lyx/.pdf downloads.

Thanks,

T.

Re: [LyX/master] prefs/needauth: added warning if user tries to disable authorization for needauth converters.

2017-08-04 Thread Tommaso Cucinotta

As per the HTML, if this is the standard for LyX then let's go for it.

As per the rephrase, I'm ok with removing the "unless you know...", feel free
to commit.

Thanks,

On 31/07/2017 23:10, Christian Ridderström wrote:

On 27 July 2017 at 00:06, Tommaso Cucinotta mailto:tomm...@lyx.org>> wrote:

commit 8a4fcd3d95ca4aeed1c46152cecadf29ed21e774
+ _("SECURITY WARNING!"), _("Unchecking this option has the effect
that potentially harmful converters would be run without asking your permission first. This is
UNSAFE and NOT recommended, unless you know what you are doing. Are you sure you would like to
proceed ? The recommended and safe answer is NO!"),

The warning as per the patch:

Unchecking this option has the effect that potentially harmful converters would
be run without asking your permission first. This is UNSAFE and NOT
recommended, unless you know what you are doing. Are you sure you would like to
proceed ? The recommended and safe answer is NO!

There's an extra space before the question mark (?). However, I would suggest removing
"unless you know what you are doing" from the message to make it stronger, i.e.

Unchecking this option has the effect that potentially harmful converters would
be run without asking your permission first. This is UNSAFE and NOT
recommended. Are you sure you would like to proceed? The recommended and safe
answer is NO!

And actually, I wouldn't mind making the warning even stronger:

Unchecking this option is UNSAFE and NOT RECOMMENDED. With this option unchecked LyX
will NOT EVEN ASK you for permission before executing converters that might be
potentially harmful to your computer system. Please read
for more information.
Are you sure you would like to proceed? The recommended and safe answer is NO!

The above would of course mean that we also need to add a section to the user
guide or something.
Hmm... could we suggest asking the users' list for advice?

/Christian

Re: Discussing minted/safety/etc (Was: Options for resolving the minted + shell-escape issue)

2017-08-04 Thread Tommaso Cucinotta


On 02/08/2017 01:00, Christian Ridderström wrote:

The long threads likely by now do contain a lot of separate pieces of 
description, but it's unfortunately not so easy (at least for me) to find this 
information among all the posts. [...] Or perhaps to copy them to into one 
place, adding minor markup as to what's still valid.


I think this is already a great improvement

  http://wiki.lyx.org/Devel/SafetyAndSecurity

we just need more eyes to go through it, so to add possible important missing 
bits.

T.

Re: Options for resolving the minted + shell-escape issue

2017-08-04 Thread Tommaso Cucinotta


On 02/08/2017 17:45, Richard Heck wrote:

do even better. [...] People who have the expertise to design a new solution, 
and then to implement it, whether via AppArmor or whatever, should feel free to 
make it a priority for 2.4 and create a feature branch for it.


+1, I'll take some time to rebase my old patch and double-check it has a switch 
to turn it off on other OSes and Linux distros w/out AA.

T.

Re: Options for resolving the minted + shell-escape issue

2017-07-28 Thread Tommaso Cucinotta


On 17/07/2017 15:18, Enrico Forestieri wrote:

I don't understand if the 'needauth' is new in LyX 2.3.0 or already existed.


This is new to 2.3.0 and is instead deemed a security improvement. Yes,
security is improved by allowing running dangerous converters.


I have to step in for a clarification:

1) without the needauth mechanism, LyX would run arbitrary R code snippets 
embedded within a .lyx code whenever you tried to compile your doc, without 
asking anything, without telling the user, etc, such R code could even grab 
your ~/.ssh/ folder and upload it to a remote site, or just rm -rf $HOME/ 
(delete your entire home folder)

2) needauth was added as a safety measure that tags those potentially dangerous 
converters (the one executing R scripts) as 'needauth' (i.e., needs user's 
authorization before being run), so that, whenever LyX tries to run that 
converter, it will ask the user first

Therefore, it should be quite clear that needauth is a security improvement 
(compared with LyX supporting R/sweave/knitr without needauth).

3) I liked to propose addition of support for running Gnuplot scripts as well, 
with the corresponding converter tagged as a 'needauth' one
[ this last feature cannot be tagged as a security enhancement, but it's not 
more dangerous than what we already had with R scripting ]


The text indicates to me that it's possible for a document to store some
kind of setting that allows a converter (here external program) to be run
in an automated manner without my manual intervention or consent.


it is possible to use preferences settings in THREE possible combinations:

A) forbid any use of any 'needauth' converters altogether (will be the default 
for all users), that means users trying to compile a doc embedding a R script 
would see a dialog telling them that it was not possible to convert that part 
of the document because it is unsafe and the preferences options forbid it

  \use_converter_needauth_forbidden true

B) allow for using 'needauth' converters in the normal/designed mode, i.e., only after 
gaining user's authorization through a confirmation dialog that pops up on-demand 
per-converter invocation and per-document (but it's not asked again once the converted 
image is cached, so a second compilation of the same doc would NOT ask anything, because 
LyX would use images from its cache in ~/.lyx/cache/*): users can check a "don't ask 
me again for this document" if they trust the document, and there's quite a few 
converters therein, so any further 'needauth' converter in the doc is run without further 
auth question

  \use_converter_needauth_forbidden false
  \use_converter_needauth true

C) allow for using 'needauth' converters in the old unsafe way, as per status 
quo before 'needauth' was merged in: any converter is just run, including 
'needauth' ones, without asking anything; after commit [8a4fcd3d/lyxgit], when 
a user tries to get to this configuration from the preferences, he/she gets an 
additional SECURITY WARNING dialog explaining how unsafe/dangerous an action 
they're doing -- at this point, if users go on and approve the config options 
change, then they're assumed to know what they're doing

  \use_converter_needauth_forbidden false
  \use_converter_needauth false

There's an additional patch worked out in the TT below, which I like to think 
of as the final solution:

D) execute any converter within a protected/safe sandbox, this can easily be 
accomplished on Ubuntu/Linux using AppArmor, but it's not easy to merge such a 
change unless there's a portable corss-OS way -- albeit, for Linux users, this 
would be THE right way to go (with an initial transient until the AppArmor 
rules are correctly tunes, which I cannot claim is a trivial task). This would 
require no needauth, because the converter could be restricted to write only 
within a temporary folder, and be forced into having no network access, then 
only the created image file would be copied back from there, and anything else 
trashed away.

I hope the above clarifies.


but
consider the following example:
I create a document with some embedded code to be run by converters. It's
my document, I trust it. Then I e-mail it to a colleague or perhaps my
customer for review. Time goes by and eventually I get the document back
from review, but the review took longer than expected and my next deadline
was yesterday, so I'm in a hurry and build "my" document.


this is an excellent scenario to add to the list of additional enhancements, 
thanks; just TT-ified it:

  http://www.lyx.org/trac/ticket/10735

The enhancement that was already discussed back long ago, was an expiry 
timestamp
of the user's authorization, so that, when user says "don't ask me again", this
will be remembered only for X days, weeks, months, then it will expire, and if
the same doc is opened back 1y later, one would be warned again.

  http://www.lyx.org/trac/ticket/10730

Another discussed one was about hashing the content

Re: Living with shell-escape: Using two LyX instances - critique invited

2017-07-28 Thread Tommaso Cucinotta


On 28/07/2017 23:17, Tommaso Cucinotta wrote:

Actually, I have to check how the above "-x lyxrc-apply " behaves w.r.t. 
options not mentioned, e.g., whether reset to the default, or it does what one might be 
tempted to imagine, i.e., just override the mentioned options, leaving others as from 
prefs file.


just verified that the behavior seems the one you'd imagine, i.e.:
1) "Format..." is just one of the many prefs recognized tags, it can be omitted 
without problems
2) without recurring to weirdnesses of the Linux shell that allows to send a command 
options including "\n", one can simply use the -x option multiple times and 
specify the pref options to modify; I've modified the section accordingly, in 
http://wiki.lyx.org/Devel/SafetyAndSecurity#sDevel.SafetyAndSecurity_11

It's a bit long to write -x "lyxrc_appy \\option value", but the option is 
there nonetheless, so there's no need for another -Doption=value syntax.

T.

Re: Living with shell-escape: Using two LyX instances - critique invited

2017-07-28 Thread Tommaso Cucinotta


On 27/07/2017 23:08, Christian Ridderström wrote:

./src/lyx -x 'lyxrc-apply Format 22
\use_converter_needauth_forbidden true'

This is lyx-unsafe:

./src/lyx -x 'lyxrc-apply Format 22
\use_converter_needauth_forbidden false
\use_converter_needauth false'


I haven't added this to the wiki page as I don't understand the 'Format 22' 
part.

Perhaps you could explain it, or add it to the page?

Would this be applied after LyX preferences/config has been read, but before a 
document is opened?


couldn't study this for long, I just noticed that, when you click the Apply button from 
the Prefs dialog, we trigger the LYXRC_APPLY LFUN, and we pass as argument a string being 
an exact copy of a whole preferences file, including the first line "Format 
22", carriage returns, backslashes, and even comments #... :-). I assume the Format 
line is useful to allow for format upgrade(s) when starting a new version of LyX with a 
~/.lyx/preferences file written by an older version.
Actually, I have to check how the above "-x lyxrc-apply " behaves w.r.t. 
options not mentioned, e.g., whether reset to the default, or it does what one might be 
tempted to imagine, i.e., just override the mentioned options, leaving others as from 
prefs file.

T.

Re: Living with shell-escape: Using two LyX instances - critique invited

2017-07-27 Thread Tommaso Cucinotta


[ adding chr@, gm@ explicitly ]

On 27/07/2017 17:31, Tommaso Cucinotta wrote:

I think what we might do in a relatively easy and generic way, is to allow
for setting individual preferences settings from the command-line, e.g.:

   alias lyx-safe='/usr/bin/lyx -Duse_converter_needauth_forbidden=true'
   alias lyx-unsafe='/usr/bin/lyx -Duse_converter_needauth_forbidden=false 
-Duse_converter_needauth=false'

that would override whatever one has in the ~/.lyx/preferences config.


we have an already existing way through the LYXRC_APPLY LFUN, with a relatively 
heavy syntax:

This is lyx-safe:

./src/lyx -x 'lyxrc-apply Format 22
\use_converter_needauth_forbidden true'

This is lyx-unsafe:

./src/lyx -x 'lyxrc-apply Format 22
\use_converter_needauth_forbidden false
\use_converter_needauth false'

T.

Re: Living with shell-escape: Using two LyX instances - critique invited

2017-07-27 Thread Tommaso Cucinotta


On 27/07/2017 17:31, Tommaso Cucinotta wrote:

I think what we might do in a relatively easy and generic way, is to allow
for setting individual preferences settings from the command-line, e.g.:

   alias lyx-safe='/usr/bin/lyx -Duse_converter_needauth_forbidden=true'
   alias lyx-unsafe='/usr/bin/lyx -Duse_converter_needauth_forbidden=false 
-Duse_converter_needauth=false'

that would override whatever one has in the ~/.lyx/preferences config.


we have an already existing way through the LYXRC_APPLY LFUN, with a relatively 
heavy syntax:

This is lyx-safe:

./src/lyx -x 'lyxrc-apply Format 22
\use_converter_needauth_forbidden true'

This is lyx-unsafe:

./src/lyx -x 'lyxrc-apply Format 22
\use_converter_needauth_forbidden false
\use_converter_needauth false'

T.

Re: Living with shell-escape: Using two LyX instances - critique invited

2017-07-27 Thread Tommaso Cucinotta


On 27/07/2017 16:10, Guillaume MM wrote:

It makes sense that an option in the command line could disable needauth
entirely. It is becoming hard to track all the suggestions so if you are
interested in this I suggest filing enhancement reports / the wiki page.


I think what we might do in a relatively easy and generic way, is to allow
for setting individual preferences settings from the command-line, e.g.:

  alias lyx-safe='/usr/bin/lyx -Duse_converter_needauth_forbidden=true'
  alias lyx-unsafe='/usr/bin/lyx -Duse_converter_needauth_forbidden=false 
-Duse_converter_needauth=false'

that would override whatever one has in the ~/.lyx/preferences config.

This might be useful beyond needauth et al.

Bye,

T.

Re: Can shell-escape take advantage of needauth framework?

2017-07-26 Thread Tommaso Cucinotta


On 26/07/2017 22:55, Pavel Sanda wrote:

Tommaso Cucinotta wrote:

On 25/07/2017 11:10, Pavel Sanda wrote:

1. No "needauth" preferences (do not allow needauth from being disabled).


instead of this, why don't we put some more stress on the action of
disabling the "Use needauth option", e.g., the attached patch ?


I am fine with this patch.


[lyxgit/8a4fcd3d]
(included a little fix so that the GUI knows when the dialog actually changed() 
)

T.

Re: Can shell-escape take advantage of needauth framework?

2017-07-25 Thread Tommaso Cucinotta


On 25/07/2017 11:10, Pavel Sanda wrote:

1. No "needauth" preferences (do not allow needauth from being disabled).


instead of this, why don't we put some more stress on the action of disabling the 
"Use needauth option", e.g., the attached patch ?

Some nice text would have to be properly formulated of course...

Also, the "Use needauth option" checkbox label we have now could be reworded to 
smth like
"needauth converters need user permission", or
"ask user permission before running needauth converters", or
"don't run needauth converters without user's consent"
...

The forbid vs enable needauth of the left checkbox sounds ambiguous/confusing 
w.r.t. the use vs don't use needauth we have on the right checkbox.

T.


2. The dialog has a checkbox "I have read the above and I understand the
consequences", unchecked by default, which one has to check before
clicking "allow" or "always allow for the document". This checkbox is
remembered per-user (this replaces the "forbid use of needauth" option).
3. For command-line only (without GUI), have a command-line options
--needauth=[never(default)|always|ask].


I am not sure I understood this proposal, but if by 1&2 you mean
to replace "Forbid.." checkbox by unchecked "I have read...",
then I have no problem with it. (Though if we wanted to make it scary
it should contain words like "Warning" or "Dangerous".)

Ask Christian about the string, he seemed to be the main critic of UI...

Pavel



diff --git a/src/frontends/qt4/GuiPrefs.cpp b/src/frontends/qt4/GuiPrefs.cpp
index 5f2f4740..d33cc0f8 100644
--- a/src/frontends/qt4/GuiPrefs.cpp
+++ b/src/frontends/qt4/GuiPrefs.cpp
@@ -1874,6 +1874,21 @@ void PrefConverters::on_needauthForbiddenCB_toggled(bool checked)
 }
 
 
+void PrefConverters::on_needauthCB_toggled(bool checked)
+{
+	if (checked)
+		return;
+
+	int ret = frontend::Alert::prompt(
+		_("SECURITY WARNING!"), _("Unchecking this option has the effect that potentially harmful converters would be run without asking your permission first. This is UNSAFE and NOT recommended, unless you know what you are doing. Are you sure you would like to proceed ? The recommended and safe answer is NO!"),
+		0, 1, _("&No"), _("&Yes"));
+	if (ret == 0) {
+		needauthCB->setChecked(true);
+		return;
+	}
+}
+
+
 /
 //
 // FormatValidator
diff --git a/src/frontends/qt4/GuiPrefs.h b/src/frontends/qt4/GuiPrefs.h
index 3b6ff604..8f43a37e 100644
--- a/src/frontends/qt4/GuiPrefs.h
+++ b/src/frontends/qt4/GuiPrefs.h
@@ -338,6 +338,7 @@ private Q_SLOTS:
 	void changeConverter();
 	void on_cacheCB_stateChanged(int state);
 	void on_needauthForbiddenCB_toggled(bool);
+	void on_needauthCB_toggled(bool);
 
 private:
 	void updateButtons();

Re: Can shell-escape take advantage of needauth framework?


On 19/07/2017 11:06, Pavel Sanda wrote:

I disagree though that we should ban needauth mechanism right now and
if the argument really is that I can trick someone into unchecking
whatever I want, then I can directly trick him into writing rm -rf /
on the commandline.


+1, albeit quite evidently implicit from my side :-)!

T.

Re: Can shell-escape take advantage of needauth framework?


On 18/07/2017 21:50, Christian Ridderström wrote:

I do not know how many KGB/CIA agents will be willing attend the 'hack LyX' 
classes. How much is it worth on a spy resume ?


haha! something like that must have been said by someone in Redmond while coming out with 
this new brilliant and super-useful business-oriented automation feature called 
"Word Macro", before Melissa came out in 1999!

Personally, I wrote patent applications using LyX during one of my prior jobs in 
industry, and helped my colleagues into getting used to LyX, disregarding recommendations 
of my colleagues in that I should have used the "proper" .docx template... that 
can be conveniently edited using a much more secure program and OS?!?

Btw, +1 on threat analysis, and np with the minefield, even simply talking 
about security risks is being a good thing for LyX, and a few mines could 
already be spotted :-)!

Cheers,

T.

Re: Types of LyX users (Was: Can shell-escape take advantage of needauth framework?)


On 25/07/2017 00:15, Scott Kostyshak wrote:

Then we could easily say "I think this feature
would benefit Lucie, would hurt Raimundo, and would not affect Sara at


Alice was beginning to get very tired of sitting by her sister on the bank, and of having 
nothing to do: once or twice she had peeped into the book her sister was reading, but it had 
no pictures or conversations in it, "and what is the use of a book," thought Alice 
"without pictures or or conversations, or R scripts or automatically generated Gnuplot or 
Python plots?
So, Alice started to write a book using the wildly popular & largely known LyX text editor, allowing her to embed 
Animated contents and even interactive chats within chapters of her new book, which she loves to exchange with Bob the 
Lizzard. Although, Bob is used to write so fast, to get to annoy every one with its resounding theme sounds for each and 
every key press, to the point that Alice likes to send him chapters incorporating little jokingly hazardous scripts that, 
when previewed on the screen, cause Bob's sound volume to definitely shutdown (via removal of the snd-* ALSA drivers from 
the kernel through a root escalation exploit, but that's an unimportant technical detail). Bob would never try to preview 
nor compile those chapters, if it were not for the carefully placed and beautifully decorated "PUSH ME" labels 
that Alice placed on Bob's "View" button in the application Toolbar (if you're curious, other carefully crafted 
icons included "DRINK ME" and "ORANGE MARMELADE" in the same toolbar, all of them calling external 
'needauth' converters), when she helped out with LyX installation on his Linux box, quite a though task for a Lizard. So, 
there he goes, pressing that button over and over again through the day.
However, despite the apparent risks coming from the encrypted arguments of the Caterpillar, Alice 
and Bob didn't realize how much should they have been suspicious about the seemingly harmless and 
well-educated tea sessions they regularly had with Mallory the Mad Hatter, who used to like to 
confuse their little unaware friends with innocent paragraphs like: You might just as well say that 
"I see what I eat" is the same thing as "I eat what I see." Albeit literally 
interesting, Alice would fail to realize the full extent of the so big and remarkable differences 
in the font size between the two seemingly and confusingly self-resembling paragraphs, which, at a 
closer look (i.e., zoom at least 15000x in the preferences/settings pane), would indeed be 
recognized in their very nature of peculiar R and Gnuplot insets embedding scripts able to produce 
original hand-crafted output images with the contents just mentioned before, in addition to little, 
unforgettable changes to Alice and Bob home directories...

... hope you like it as a start ;-P ...

T.

Re: Silent/automatic execution of converter and needauth, concrete questions to clarify my understanding


On 23/07/2017 22:08, Christian Ridderström wrote:

Are the settings that needauth remember done:
a) per document, regardless of converter
b) per document-and-converter pair?
c) Also per snippet of code?


it's only a), but pls keep in mind this is only for those (few) converters 
tagged with the 'needauth' option in configure.py.

The rationale is that trust should be an issue with new docs never 
seen/compiled earlier only. What would it mean to trust Sweave insets in this 
doc, but NOT Gnuplot insets ? If I don't trust the document, then I should keep 
the warning every time a potentially harmful converter is attempted to be run. 
On the other hand, once I'm sure this is the doc I was expecting from my 
colleague, and I trust him/her, then it will be safe to authorize any converter 
in that doc.


E.g., what happens if I'm keeping a document on say a network drive. I
put some code in the document and execute it. When asked by needauth
the first time, I say "always allow for the document".   So the next
time I execute the document I'm not asked again.

What happens now if someone else modifies the code embedded in the
document?  Will the permission(s) still be active, so that the
document executes the new code?  Am I warned in any way?


no further warning happens here: that's to facilitate collaborative editing with 
colleagues: once I said I trust that pathname, then if I check out (git pull) a change 
from my colleagues, I don't want to be bugged again and again about risks. On the other 
hand, if I don't trust the folks I'm co-editing a .lyx doc with (which I assume to be a 
very very unlikely use-case), then I should never check that box saying "Never ask 
me again for the same doc".

Perhaps a variant could be that, even when I don't say "Never ask me again", if 
I authorize the use of a converter on a specific .lyx filename, then any further use of 
the same converter on the same file with the same time-stamp could be allowed without 
further questions to the user ?


If not, perhaps a future improvement could be to be able to approve
specific code snippets to be executed.
The user-dir could e.g. contain a hash of code snippets that's
approved to be run for a certain document. Or perhaps even for all
kinds of documents.


I'd be for keeping track of possible enhancements like this to 'needauth' as 
individual Trac items, to be linked to

  http://www.lyx.org/trac/ticket/10481

T.

Re: Silent/automatic execution of converter and needauth, concrete questions to clarify my understanding


On 18/07/2017 00:49, Guillaume MM wrote:

(Another one is if the path is ~/Download/new1.lyx and you happen to
have given permanent permissions for a file with the same path three
years earlier, deleted and forgotten about since...)


there's been discussion during the needauth development about an expiry time 
for the per-document authorization

  http://www.lyx.org/trac/ticket/10481

perhaps we should recover that add-on as a separate #, and give it a proper 
priority ?

Thanks,

T.

Re: short report from LyX under Linux

On 24/07/2017 23:39, Uwe Stöhr wrote:

One of my daily work is CAD and common programs like Solidworks are not
available under Linux:
https://forum.solidworks.com/thread/90517
The only program I found so far is VariCAD:
http://www.varicad.com/en/home/products/download/

now I see your point, if you get into professional desktop applications needed
in specific areas, then you'll get good/complete/professional ones just for
Windows and perhaps Mac, Linux has very little chances... albeit there's more
and more open-source clones coming out.

I have myself an example of a Windows freeware application I love for
annotating PDFs, which I'm lucky I can use through wine ;-).

Next step was to check for a tax program. There are no German tax programs
available for Linux:
https://wiki.ubuntuusers.de/Steuer-Spar-Erkl%C3%A4rung/

My wife is an accounting consultant ... no way you can see a Linux supported app talking the right
standards that allow official submissions to the IRS website ("Agenzia delle Entrate"),
with periodic upgrades due to new laws/regulations issued every year etc. :-). But, would you like
to hear a sad story ? The software she uses and pays for, officially supported on Windows only, and
on a specific version of Windows only, needs this particular nasty dependency called
"JRE", but there doesn't seem to exist an easy way to run it on Linux (likely one should
crack its licensing scheme, making use of a USB license physical key).

The low market share of Linux is for sure a reason

yes, just for desktops, but don't worry it's already changing, as witnessed by the
"Linux subsystem" embedded within Windows 10 ;-P!

https://msdn.microsoft.com/en-us/commandline/wsl/install_guide
https://msdn.microsoft.com/en-us/commandline/wsl/faq

(don't know much about this, just weird to see all these explanations about
getting Linux stuff working throughout the MSDN docs!)

Re: Can shell-escape take advantage of needauth framework?


[slowly catching up...]

On 18/07/2017 19:46, Christian Ridderström wrote:

I just did a test with gnuplot. In the LyX settings I had unchecked 'Forbid of 
use of needauth converters' and unchecked 'Use needauth option'. Then I opened 
a LyX doc with a gnuplot script. Result: LyX tried to run the script due to the 
preview, without asking or alerting me.


that's the purpose of the "Use needauth option", namely, allow for a workflow 
without burdens for those who know what they're doing, disabling needauth (which you did) 
just gets rid of the security measures, opens up your firewall, turns off security etc., 
compare with these other common software like a firewall or antivirus, if I go the its 
admin panel and turn it off...

What if, when unchecking that option, LyX would have popped up another dialog 
with a HUGE SECURITY warning ? I'm now feeling this would be needed.


In my opinion this demonstrates a case where the security is _not_ good enough, 
as I don't think it'd very difficult to trick someone into unchecking these 
boxes.


aware of the limitations of needauth, the final remedy was ...


- ?


sandboxing, as discussed in http://www.lyx.org/trac/ticket/10481, where there's 
a few notes about how to possibly design a portable mechanism across Win, Mac 
and Lin.

Thanks,

T.

Re: Any descriptions of the security aspects (related to needauth and shell-escape)?


On 23/07/2017 16:56, Christian Ridderström wrote:

Does anyone feel we should _not_ keep such a security document in the general 
LyX repository?

Note: Generally speaking I'm all for being open and transparent, but such a 
document might end up containing descriptions of ways in which LyX could be 
compromised. It sort of goes with the territory of describing threat 
scenarios/mechanisms.


also during the needauth work, I remember we exchanged a few jokingly dangerous 
.lyx documents off-list, via private e-mails, if memory serves, exactly just to 
avoid providing ourselves a bad starting point :-)... albeit these can be 
crafted in 5 mins.

T.

Re: Any descriptions of the security aspects (related to needauth and shell-escape)?


On 21/07/2017 22:28, Scott Kostyshak wrote:

I support the suggestion to create such a document and suppose to make it
a section in "Development.lyx":

+ bundled with other project policies and developer documentation
+ write access for all developers
+ we can use LyX's version control for to-be-reviewed parts and diverging
   opinions/comments


+1

Development.lyx does not have a rigorous structure. If anyone is
interested in writing something more formal, when we can at reference
that file from within Development.lyx.


I support the idea as well, and I'm interested in contributing to it.

As a starting point for the needauth stuff, I had put a recap of the problem 
and motivations within this TT:

  http://www.lyx.org/trac/ticket/10481

and, just as a reminder, the shell-escape was one of the mentioned use-cases, 
albeit not the very one under my hands at that moment.

I'd suggest not to create a .lyx document, but rather start from a wiki page 
under

  https://wiki.lyx.org/Devel/Devel

with a recap of the current status and exchanged ideas, and pointers to the 
Trac TTs tracking TODOs which we agreed upon.

My2c,

T.

Re: Going into dangerous mode (Was: Can shell-escape take advantage of needauth framework?)


On 23/07/2017 20:55, Christian Ridderström wrote:

Regarding setting something in the preference file manually: The only
thing I mind is that it adds a global state to LyX, as opposed to
starting LyX with some parameters. The global state would likely
affect e.g. testing.


the good thing is that we already have the -userdir command-line optionthat 
allows testing from a predictable initial state, doesn't it :-) ?
(AFAICR, extensively used in autotests for advanced F&R).

T.

Re: Going into dangerous mode (Was: Can shell-escape take advantage of needauth framework?)


On 22/07/2017 00:47, Guenter Milde wrote:

Enrico's patch did not touch "needauth" but has some nice features for
"shell-escape": it addressed the "set and forget" issue by

a) adding a red icon to the status bar if a document has the "allow
shell-escape" flag.
   
b) revoking the permission, if the document is moved/copied to another

location.
   
I like the approach


+1, I like the idea of a visual feedback on the current security/trust status, 
e.g., it resembles the lock icon used in web browsers for https://.


From a user perspective, a common interface to "needauth" and "allow

shell escape" seems the best. "needauth" could actually take advantage of
Enrico's patch.


once I'll gain some spare time this summer, I'll try a merge :-)...


* Add "unsafe pdflatex" (== pdflatex --shell-escape) and "unsafe xelatex"
   as new converters requiring "needauth".


this sounds like something easy but already discussed and unliked/discarded.


* Allow per-converter permission settings (instead of one generic: "I
   trust/don't trust all unsafe converters").


the current system-wide setting is for all converters (disable any needauth,
allow them but warn me, allow them without constraints), whilst the memory
about trusted documents is per-document -- this makes sense because the
main source of untrust seems the document when coming from who knows where;
once the user acks that the doc is trusted, then we go without bugging the
user for each conversion. However, how would a per-converter settings
work, and how could I trust unconditionally, let's say, a R kneave/sweave
inset in a LyX doc coming from unknown sources, while at the same time
trust that an embedded gnuplot script or shell-escape command would not
delete my home folder ?


* Give users the possibility to check scripts before allowing to run them
   with shell-escape or at least list all parts of the document that will be
   allowed to run in unsafe mode
   (e.g. all gnuplot scripts for "gnuplot allowed", all ERT, preamble,
   document classes and packages for latex with shell escape).


that sounds like a feature enhancement deserving an entry on Trac ?

T.

Re: short report from LyX under Linux


On 24/07/2017 16:24, Jean-Marc Lasgouttes wrote:

I am not sure what the difference between recommend and require is in practice. 
Could you expand on that?


Ubuntu way, AFAIK:
-) Depends means LyX would be totally broken without them, e.g., you wouldn't be able to 
launch LyX, thus there's no way to install LyX without its dependencies, unless you're an 
advanced user, you know what you're doing, you like to use the command-line and the 
"--force" flags, ignoring any warning you get about things that may get broken 
afterwards
-) Recommends means the average user probably needs this as well, so it's also 
pulled in by default, unless the user explicitly decides to skip them 
(command-line with --no-install-recommends, probably there are similar options 
in some GUI tool)
-) Suggests means if you like to have a complete installation with all possible 
features, then install these as well (command-line with --install-suggests, or 
likely using some GUI tool)

T.

Re: short report from LyX under Linux


On 24/07/2017 10:30, "Uwe Stöhr" wrote:

I think that explains why software companies don't offer Linux versions.


Just a couple of well-known names...

  https://www.redhat.com/en/store/linux-platforms
  
https://buy.ubuntu.com/?_ga=2.96387035.693003821.1500908991-595300039.1443300335
  
https://platforms.linaro.org/documentation/Reference-Platform/Platforms/Enterprise/README.md/
  https://android.googlesource.com/

Linux is used on billion machines nowadays :-), albeit big numbers are 
especially for servers and mobile devices, not so much for desktops.

T.

Re: short report from LyX under Linux


On 24/07/2017 08:16, Jürgen Spitzmüller wrote:

One of the first steps was of course to try out LyX. Unfortunately I
could not compile the UserGuide because of the missing package
enumitem.
I wanted to install it but cannot find it. there is also no texlive
program to start to install the missing package there - or better to
install missing packages automatically.
How can I do this?


on a Debian-based system (e.g., Ubuntu, which I'm using), you'd get some
hint with apt-cache search :

:~$ apt-cache search enumitem
texlive-lang-german - TeX Live: German
texlive-latex-extra - TeX Live: LaTeX additional packages

Package management in Linux distributions is quite flexible, so packages
usually come with a set of dependencies, i.e., other software packages
that have to be installed in order to let your new package work, and a
set of additional recommended and/or suggested packages, which are not
mandatory, but allow you to do more with your new package.

This is for example this info in the LyX package on Ubuntu:

:~$ apt-cache show lyx
...

Depends: libboost-signals1.62.0, libc6 (>= 2.15), libenchant1c2a (>= 1.6.0), libgcc1 (>= 1:3.0), 
libmagic1 (>= 5.12), libmythes-1.2-0, libqt5core5a (>= 5.6.0~beta), libqt5gui5 (>= 5.2.0) | 
libqt5gui5-gles (>= 5.2.0), libqt5svg5 (>= 5.6.0~beta), libqt5widgets5 (>= 5.6.0~beta), libstdc++6 
(>= 5.2), zlib1g (>= 1:1.1.4), lyx-common (= 2.2.2-1), xdg-utils

Recommends: texlive-latex-recommended, texlive-latex-extra, texlive-science, 
texlive-generic-recommended, texlive-generic-extra, texlive-fonts-recommended, 
preview-latex-style, dvipng, imagemagick, psutils, ghostscript, poppler-utils, 
fonts-lyx, evince-gtk | pdf-viewer, elyxer | tex4ht | hevea | tth | latex2html

Suggests: rcs, latex-xcolor, groff, libtiff-tools, gnuhtml2latex, wv, chktex, 
noweb, sgmltools-lite, linuxdoc-tools, writer2latex, latex2rtf, librsvg2-bin | 
inkscape, texlive-xetex, lyx-dbg (= 2.2.2-1)


Description-en: document processor
 LyX is an almost WYSIWYG-frontend for LaTeX. It makes the power and
...
 You can extend the functionality of LyX by installing these packages:
  * chktex: check for typographical errors
  * gnuhtml2latex: import HTML documents
  * groff: improved table formatting in plain text exports
  * latex-xcolor: for coloured change tracking
  * librsvg2-bin, inkscape: use the SVG image format in LyX documents
  * linuxdoc-tools: export SGML LinuxDoc documents
  * mythes-*: use the OpenOffice.org/LibreOffice Thesaurus
  * noweb: import noweb files
  * rcs: integrated version control
  * sgmltools-lite: export SGML DocBook documents
  * texlive-xetex: use the XeTeX typesetting system
  * wv: import MS Word documents

Now, back to the UserGuide, perhaps what we could actually do is, to simplify 
the UserGuide so thatit can compile without the need for installing additional 
packages besides a minimal latex installation ?

Anyway, the problem is that, technically, LyX can be used without even latex on 
your system :-)... you're perfectly able to *edit* a LyX document without 
having to compile it, right :-) ? Guess that's the rationale behind texlive-* 
packages being among the Recommended ones, not the Depends ones, in the pkg 
metadata.
However, Ubuntu for example seems to install Recommended packages by default (unless 
you do an explicit: apt-get install --no-install-recommends ), while 
Suggested ones are up to the user to install. Your distributions might come with a 
more minimalistic policy.

Another thing LyX might do, is to provide useful hints to the (newbie) user, as 
to why the compilation is failing, and how to fix that on a few common Linux 
distributions, e.g., pointing to a community wiki page where common issues of 
this kind are addressed.

My2c,

T.

Re: short report from LyX under Linux


On 24/07/2017 04:59, Paul A. Rubin wrote:

BTW, you can download a bootable version of Mint to a USB memory stick and run 
it from the stick, to see if everything works (in particular, all peripheral 
devices). If you like it, you can install from the stick. I'm not sure if the 
equivalent is available with Ubuntu, but it might be.


Yes, Ubuntu can be flashed to a USB stick, then you can boot and run the OS 
from the stick and/or start installation to your HD whenever you like.

T.

Re: Can shell-escape take advantage of needauth framework?

2017-06-27 Thread Tommaso Cucinotta


On 28/06/2017 00:02, Enrico Forestieri wrote:

...and those converters can execute
arbitrary commands,


just to be sure, I just double-checked that on current trunk, without any settings in 
one's ~/.lyx/, the default behavior will be "Forbid use of needauth 
converters", so any of those dangerous ones would be disabled by default.

As for shell-escape, I couldn't go through the whole thread yet, but it seems 
very related, so it makes sense to be added as well. Whether in this release or 
next one, it's all up to the release master, though!

AFAICS, a reasonable (needauth-alike) behavior seems:
- a document-specific setting tagging the document as one needing to run latex 
with -shell-escape
- only when trying to run latex (or pdflatex, if it supports -shell-escape, or 
others), at the first attempt, trigger similar security questions as for 
needauth:
  a) the document needs to be compiled with this potentially harmful option, 
are you sure you want to do that ? (y)es, (a)lways for this doc, (n)o [(r)un 
without shell-escape ?]
  b) have another set of settings similar to needauth ones (or re-use them ?) 
that disable the question by default, so the user cannot choose (y)es unless 
changes explicitly the settings
- if one just opens the .lyx, makes edits, but never previews, nor needs to run 
latex, then no question pops up.

Just quick thoughts, though.

Good night.

T.

searching lyx stuff on Google

2017-06-27 Thread Tommaso Cucinotta


Hi,

I was sure we had a wiki page on lyx.org about security issues and the need for 
hardening LyX against potential misuse, due to the risks in running external 
tools, write18{}, shell-escape, gnuplot / R / knitr et al., but perhaps I can't 
really remember.

However, I struggled in finding back

  http://www.lyx.org/trac/ticket/10481

even searching explicitly for "  Hardening LyX against potential misuse" on 
Google, I can only find mail-archive.com e-mail threads, restricting with site:lyx.org 
leads to no results in the search.

1) did we change anything in the website, so that LyX wiki pages are not 
indexed anymore by G. ?
2) did we actually have that wiki page on hardening LyX, or just the trac 
#10481 ?

Thanks,

T.

Re: Can shell-escape take advantage of needauth framework?

2017-06-27 Thread Tommaso Cucinotta


On 20/06/2017 02:43, Guillaume MM wrote:

One must look at the
big picture and see that adding an authorization mechanism for arbitrary
execution of commands is absurd when its sole purpose is to call an
external tool from within LaTeX.


needauth was a urgently needed mitigation of the security issues behind running
arbitrary external tools when compiling LyX documents; a more engineered remedy
AFAICR was actually the use of sandboxing machineries, which was prototyped on
Ubuntu/Linux using AppArmor.


Lastly, I find interesting the idea of a "secure" icon providing visual
feedback and the ability to revoke the permissions, and I believe that
it could be used to improve the current needauth mechanism.


+1 for completing the current needauth with a revocation means; this can include
also a security settings pane where we list the pathnames of all documents in
the user's approved permission list, where one can revoke the permission 
selectively,
or even perhaps edit the list and use wildcards (e.g., always grant when I work
in /home/tommaso/work/trusted/*)... ok, I went too far :-)

My2c,

T.

Re: Gnuplot format & converter script

2017-06-05 Thread Tommaso Cucinotta


from the help message this sounds interesting, I also thought
we could have had various terminal types, and get .gp to .pdf,
to .eps, etc, albeit it's sufficient to convert to one
vectorial format, and LyX knows how to get to the others.
Perhaps among the most useful things, is that 1) you can use this
script to render on screen through the .gp interactive capabilities,
using the same unmodified .gp script that is included in .lyx;
2) the terminal options, but theburden is that each document
(even each use of .gp scripts) might need slightly different options...

not really much time in these days, but this script might easily
replace the gnuplot2pdf.py one, with proper invokation in configure.py.
(-t pdf). However, I can't get at a glance the why of all of this
lines' parsing & mangling. guess it's a try to be robust w.r.t. various
different contents of the .gp script.

T.

On 25/05/2017 19:36, Scott Kostyshak wrote:

On Fri, May 05, 2017 at 08:03:02PM -0400, Scott Kostyshak wrote:


Tommaso,

I found an old script from Koji for gnuplot. Is it useful now?


Tommaso,

No worries if you don't have time, but I just wanted to make sure you
saw this.

Scott

Re: [LyX/master] skip graphics conversion when runparams.dryrun is true

2017-06-05 Thread Tommaso Cucinotta


On 04/06/2017 03:40, Enrico Forestieri wrote:

On Fri, Jun 02, 2017 at 12:55:47AM +0200, Enrico Forestieri wrote:


So, output_file is always empty, whatever op.dryrun. I think that
the following change was instead meant:

-   string const output_file = prepareHTMLFile(op);
+   string const output_file = op.dryrun ? string() : prepareHTMLFile(op);


Good catch. I agree. Please commit.


Done at 8ae652eb.


thx Enrico !

T.

Re: Adv. F. & R.

2017-05-17 Thread Tommaso Cucinotta


All valuable comments, thanks for the detailed report.
This deserves a TT on trac.lyx.org.

I'd agree with 2. and 5., and for the others the issue
is that the regexp inset is a kind of math inset ATM,
but at the same time it is not, in that it matches non-math
text. The quickest thing I can think of, is to disable/forbid
boldface, emphasized and other font variations within it,
because the idea is that you use it to be generic in terms of
text to find, but you handle any formatting outside that inset.

If people start inserting full-fledged LyX capabilities within
a \regexp{...} (images, external materials, titles, captions,
references, bibliographies), everything will happen :-), as
it wasn't designed for such uses.

Would that make sense ?

T.

On 18/05/2017 03:31, Andrew Parsloe wrote:

(alpha1-1, windows 7) Some "road testing" of adv. find & replace:

UI matters (mainly minor): 1. With a document open and the Adv. F & R
panel open, click on the document to ensure the focus is there. Now
click the X on the Adv F & R panel to close it. It closes but the
toolbar button remains depressed. (But: as soon as you click back in
the document, the button is released.)

2. Menu entry "Insert > Insert Regular Expression" for consistency
with the other entries in the menu should be "Insert > Regular
Expression".

3. User's Guide: Section 6.13.3. "This is done with the context menu
Insert > Insert Regular Expression" should be "This is done with the
context menu Insert Regular Expression or from the main menu "Insert
> Regular Expression".

4. Is there a reason for *not* having a regular expression checkbox
(on the Settings panel presumably)? It feels unintuitive to me to
have all the other settings on the two Adv F & R panels but to have
to divert to the Insert menu or right click to insert a regular
expression. I think it would fit compatibly at the bottom of the
lower group of checkboxes on the Settings panel.

5. I find "Ignore format" an unnatural way of thinking. It is a
negative statement & clearing the checkbox gives a double negative:
not ignore format. Why not "Format sensitive" a positive statement
(cleared by default, corresponding to "Ignore format" set by
default)? "Case sensitive" is already used and the User's Guide,
Section 6.13.1 (2nd bullet) uses "format-sensitive" and
"format-insensitive".

* Regular expressions.

This is broken in my installation of LyX 2.3.0a1-1.

6. Enter .* into a regexp then bold it (ctrl+B). The Code Preview
Pane (CPP) shows \regexp{\boldsymbol{.*}\endregexp{}} which suggests
it is not going to find any boldfaced *text*. But it also doesn't
find boldfaced maths.

7.1 Enter .* in a regexp and then emphasize it with the toolbar
button. CPP shows \regexp{\mathcal{.*}\endregexp{}}. Oops. (The Find
pane shows mathcal symbols.) 7.2 The emphasis button also is not
depressed (on).

8. Varying the regexp to [a-z]+ with the a-z emphasized the CPP shows
\regexp{[\mathcal{a-z}]+\endregexp{}}. With Ignore format cleared,
clicking Find Next, this expression gobbled the words of my test
document *except* for the emphasized ones. This has happened three or
four times while I've been testing, but I can't reliably reproduce.

There were various other "happenings" during my testing which made me
feel the regexp part of Adv F & R was very fragile. One thing that
intrigued me was when trying to clear bolding from a regexp by
toggling ctrl+B. If the cursor is not exactly correctly positioned it
is easy to get two \boldsymbol commands in the regexp. On one
occasion, by about the third toggle, the regexp suddenly changed into
an ERT-like inset, displaying \boldsymbol, braces etc. rather than
the bolded characters. I found I could edit this and the CPP
reflected my edits. People often ask about editing the CPP. It struck
me that that was what I was doing. (Alas, I haven't been able to
reproduce this particular effect.)

Andrew






--- This email has been checked for viruses by Avast antivirus
software. https://www.avast.com/antivirus

.

Re: libreoffice conversion issues

2017-05-09 Thread Tommaso Cucinotta


On 09/05/2017 22:56, Uwe Stöhr wrote:

Have you tried the latest LibreOffice? Some of the recent releases had 
conversion problems. Until I upgraded to 5.3.2 LibreOffice works fine for me. 
Of course the wrong option syntax should be fixed.


That was among the queued tries after my Ubuntu upgrade. Now I have LibreOffice 
5.3.1.2 and the problem seems to have gone.

This is the full output I get on screen, in addition to the "--" vs "-" syntax 
issue:

$ ./src/lyx ~/newfile1.lyx
Warning: -headless is deprecated.  Use --headless instead.
Warning: -nologo is deprecated.  Use --nologo instead.
Warning: eps is deprecated.  Use -eps instead.
convert /tmp/lyx_tmpdir.TCUECaQ20934/gconverto20934.odg -> 
/tmp/lyx_tmpdir.TCUECaQ20934/gconverto20934.eps using filter : draw_eps_Export
Input files: -
Processing: -
Rendering with existing %%BoundingBox: 0 0 539 785
Calculating Bounding Box...ready. %%BoundingBox: 96 589 236 697
Creating output file - ... ready.

Perhaps some of these lines might also be hidden...

T.

Re: libreoffice conversion issues

2017-05-09 Thread Tommaso Cucinotta


On 09/05/2017 22:56, Uwe Stöhr wrote:

Of course the wrong option syntax should be fixed.


[47cd1e23/lyxgit].

T.

Re: [LyX/master] Add needauth option to gnuplot->PDF converter introduced in [066edd3c/lyxgit].

2017-05-08 Thread Tommaso Cucinotta


On 08/05/2017 09:12, Pavel Sanda wrote:

Richard Heck wrote:

On 05/05/2017 06:51 PM, Tommaso Cucinotta wrote:

On 05/05/2017 17:26, Pavel Sanda wrote:

Finally after all the years! :)


someone counted 8y :)


Perhaps some example lyx file to give users idea how to use it?


how to make such examples available ?


lib/examples/ folder.


+


done & pushed, check out example_gnuplot.lyx

T.

Re: keytest.py

2017-05-08 Thread Tommaso Cucinotta


On 07/05/2017 19:46, Kornel Benko wrote:

Do they fail if retested separately too?


I tested again just a couple of these, one was keeping failing, another one 
succeeded, but I had no time for a proper investigation. Also, AFAICR my laptop 
started buzzing or smth while the overall tests batch was running, I suspect 
some keypresses didn't really result in what they were supposed to (I wasn't 
looking at the screen in that very moment), likely the cause of some of these 
failures.

I just tried removing the custom xvkbd from my dir (need the attached patch for 
that), and I've got this:

Running test cases . . .
bug-7673b-in.txt: Ok
 bug-7673-in.txt: Ok
 bug-8370-in.txt: Ok
 bug-8482-in.txt: Ok
 bug-8540-in.txt: Ok
 bug-8684-in.txt: Ok
 bug-export-latex-in.txt: Ok
bug-math-undo-in.txt: Ok
   findadv-01-in.txt: Ok
   findadv-02-in.txt: Ok
   findadv-03-in.txt: Ok
   findadv-04-in.txt: Ok
   findadv-05-in.txt: Ok
   findadv-06-in.txt: Ok
   findadv-07-in.txt: Ok
   findadv-08-in.txt: FAILED
   findadv-09-in.txt: Ok
   findadv-10-in.txt: Ok
   findadv-11-in.txt: Ok
   findadv-12-in.txt: Ok
   findadv-13-in.txt: Ok
   findadv-14-in.txt: FAILED
   findadv-15-in.txt: Ok
   findadv-16-in.txt: Ok
   findadv-17-in.txt: Ok
   findadv-18-in.txt: Ok
   findadv-19-in.txt: Ok
   findadv-20-in.txt: Ok
   findadv-21-in.txt: FAILED
findadv-crash-in.txt: Ok
 findadv-logo-in.txt: Ok
findadv-re-01-in.txt: Ok
findadv-re-02-in.txt: Ok
findadv-re-03-in.txt: FAILED
findadv-re-04-in.txt: Ok
findadv-re-05-in.txt: Ok
findadv-re-06-in.txt: FAILED
 tabular-footnote-in.txt: Ok

There were 5 FAILED tests

However, I saw the buzzing this time: the new Ubuntu sends my screen dark 
earlier due to inactivity, and I start hearing these buzz, after entering my 
pwd, I saw the tests were still running, but I'm not sure they were affected by 
the screen saver ... need to check whether there's a way to prevent screen 
saver during these tests.


Which QT version?


5.7.1

T.

diff --git a/development/autotests/run-tests.sh b/development/autotests/run-tests.sh
index 25d29340..23ab41cf 100755
--- a/development/autotests/run-tests.sh
+++ b/development/autotests/run-tests.sh
@@ -7,13 +7,13 @@
 export LYX_ROOT=../../..
 export LYX_EXE=$LYX_ROOT/src/lyx
 
-if [ "$XVKBD_HACKED" != "" ]; then
-export XVKBD_EXE=${XVKBD:-./xvkbd/xvkbd};
-if [ ! -x $XVKBD_EXE ]; then
-	echo "You need to build XVKBD first, try: cd xvkbd && xmkmf && make"
-	exit -1;
-fi
-fi
+# if [ "$XVKBD_HACKED" != "" ]; then
+# export XVKBD_EXE=${XVKBD:-./xvkbd/xvkbd};
+# if [ ! -x $XVKBD_EXE ]; then
+# 	echo "You need to build XVKBD first, try: cd xvkbd && xmkmf && make"
+# 	exit -1;
+# fi
+# fi
 
 if [ "$(which wmctrl)" == "" ]; then
 echo "You need to install wmctrl first, try:"
@@ -29,7 +29,7 @@ fi
 
 PROGRAM_SUFFIX=$(grep -e '#define PACKAGE ' ../../config.h | sed -e 's/#define PACKAGE "lyx\(.*\)"/\1/')
 
-export XVKBD_EXE=../$XVKBD_EXE
+export XVKBD_EXE=xvkbd
 export KEYTEST=../keytest.py
 LYX_HOME=out-home
 export LYX_USERDIR=$(pwd)/$LYX_HOME/.lyx$PROGRAM_SUFFIX

Re: keytest.py

2017-05-07 Thread Tommaso Cucinotta


On 07/05/2017 14:35, Kornel Benko wrote:

Am Sonntag, 7. Mai 2017 um 02:46:31, schrieb Tommaso Cucinotta 

On 06/05/2017 14:15, Kornel Benko wrote:

Hi Tommaso,
could you please test this version?


sure, as soon as I get back to a successful compile of LyX on Ubuntu 17.04 :(!
(see separate thread)

T.



OK.
I am committing. I feel confident that nothing is going wrong with alpha 
release. And since
I am probably the only one (apart from you) testing keytests now, it seems to 
not bother anyone.


Ubuntu 17.04 (perhaps my hw is slower [8 y.o. Latitude E6400], so it gives more 
troubles :-) ):

   findadv-07-in.txt: FAILED
   findadv-08-in.txt: Ok
   findadv-09-in.txt: Ok
   findadv-10-in.txt: Ok
   findadv-11-in.txt: Ok
   findadv-12-in.txt: Ok
   findadv-13-in.txt: FAILED
   findadv-14-in.txt: Ok
   findadv-15-in.txt: FAILED
   findadv-16-in.txt: FAILED
   findadv-17-in.txt: Ok
   findadv-18-in.txt: Ok
   findadv-19-in.txt: Ok
   findadv-20-in.txt: Ok
   findadv-21-in.txt: FAILED
findadv-crash-in.txt: Ok
 findadv-logo-in.txt: FAILED
findadv-re-01-in.txt: FAILED
findadv-re-02-in.txt: Ok
findadv-re-03-in.txt: FAILED
findadv-re-04-in.txt: FAILED
findadv-re-05-in.txt: Ok
findadv-re-06-in.txt: FAILED
 tabular-footnote-in.txt: Ok

There were 10 FAILED tests

T.

Re: compiling on Ubuntu 17.04

2017-05-07 Thread Tommaso Cucinotta


On 07/05/2017 14:39, Kornel Benko wrote:

Can it be due to some missing packages?


it was a --no-create --no-recurse that was there in my quite long ./configure 
line (which I recurrently copy from config.log).

Managed to successfully compile and run with qt4, trying now with qt5, thanks.

T.

compiling on Ubuntu 17.04

2017-05-06 Thread Tommaso Cucinotta


just made that OS upgrade, and can't compile LyX anymore

the usual autogen.sh; configure [params] doesn't output a Makefile, so make 
won't run anything

Anyone with a clue ? is there a different recipe to restart compilation from 
scratch ? (did also a make distclean, didn't help)

Thanks,

T.

Re: further add-ons for 2.3.0 ?

2017-05-06 Thread Tommaso Cucinotta


On 06/05/2017 02:02, Scott Kostyshak wrote:

Yes please revert. I remember there being some disagreement about the
file formats patch. I could be wrong though. Please see my questions
here:


reverted, those 2 patches remain available in my

  tommaso features/2.3.0alpha

branch, thanks.

T.

Re: [LyX/master] Add needauth option to gnuplot->PDF converter introduced in [066edd3c/lyxgit].


On 06/05/2017 00:51, Tommaso Cucinotta wrote:

and it should be also noted in news or rel notes as well... P


done in [af49aaa9/lyxgit], feel free to amend if not clear.


also in NewInLyX23:

  http://wiki.lyx.org/LyX/NewInLyX23#sLyX.NewInLyX23_27

T.

libreoffice conversion issues


Hi,

when trying use of .odg drawings in LyX, I'm seeing these two annoyances:

1) LyX fails to convert .odg -> eps if libreoffice is open already elsewhere, 
with any other document (I had a .odp presentation open in the background), 
because it tries to run
   if os.system(r'libreoffice -headless -nologo -convert-to eps ' + '"' + infile + 
'"' + '') != 0:
   
   which unfortunately seems to exit immediately without doing any conversion 
(Ubuntu 16.10).
   Closing the already running libreoffice instance fixes the issue: now you 
can open the same .lyx file, which gets properly rendered on screen

2) libreoffice complains about deprecated "-XYZ" syntax, when it should be 
"--XYZ"

For 1), we have also the unoconv converter in configure.py, which seems to work 
instead, shall we define unoconv as the first default ?

Thanks,

T.

Re: further add-ons for 2.3.0 ?


On 06/05/2017 00:38, Tommaso Cucinotta wrote:

rebased and pushed to my feature branch features/2.3.0alpha


my bad, in pushing the release notes for gnuplot scripts, these
two also slipped along, I forgot my local master was temporarily
"topped-up" ... shall we revert ?

or, would you like to play with these and leave them for 2.3.0alpha,
just in case ?

Thanks,

T.

Re: [LyX/master] Add needauth option to gnuplot->PDF converter introduced in [066edd3c/lyxgit].


On 05/05/2017 17:26, Pavel Sanda wrote:

Finally after all the years! :)


someone counted 8y :)


Perhaps some example lyx file to give users idea how to use it?


how to make such examples available ? wiki page ? yeah, probably the best 
thing, easily found with a search engine -- who reads the manuals nowadays :P ?


and it should be also noted in news or rel notes as well... P


done in [af49aaa9/lyxgit], feel free to amend if not clear.

Thanks,

T.

Re: further add-ons for 2.3.0 ?


On 04/05/2017 17:54, Kornel Benko wrote:

2) commit cb7a69b1
Author: Tommaso Cucinotta 
Date:   Wed Oct 19 11:18:10 2016 +0200

 Tolerate formats that are not supported by lyx2lyx.>


Yes.


rebased and pushed to my feature branch features/2.3.0alpha


3) commit bf3cda7b
Author: Tommaso Cucinotta 
Date:   Sat Oct 15 01:14:02 2016 +0200

 Create new graphics from within LyX choosing a sample file to copy
 from


Feels interesting.


rebased and pushed to same feature branch as above

Anyone willing to try these before merging ?

T.

Re: Gnuplot format & converter script

2017-05-04 Thread Tommaso Cucinotta


On 04/05/2017 11:53, Scott Kostyshak wrote:

as a consequence, I came up with the attached patch that solves the


pushed that anyway -- even if copying with images and their conversion is
desired, we shouldn't need dryrun then, should we ?

T.

Re: Gnuplot format & converter script

2017-05-04 Thread Tommaso Cucinotta


On 04/05/2017 11:53, Scott Kostyshak wrote:

Assuming we do want to embed the picture on the clipboard, the next


but I don't think any of this is actually working, copying from
an inset containing a real image file (.jpg) into LibreOffice,
it just inserts an XML header line, along with a seemingly useless
link to a file:///path/to/image-source.jpg, which is not displayed
on screen nor rendered in pdf (from within LibreOffice).
Perhaps M$Word users on Win are having a different experience ?

T.

Re: further add-ons for 2.3.0 ?

2017-05-04 Thread Tommaso Cucinotta


On 04/05/2017 01:45, Scott Kostyshak wrote:

I'd really love to have 3)...


Is it polished? Is there a trac ticket for this one? Or an archived
discussion?


http://www.lyx.org/trac/ticket/5962

thanks,

T.

Re: Gnuplot format & converter script


On 04/05/2017 01:36, Scott Kostyshak wrote:

Works well! I found what I think is unexpected behavior: if I select the
graphics inset (e.g. after I have followed your instructions and clicked
on "run" and everything looks good), and copy it, I am presented with
the authorization dialog.  [...] Can you reproduce?


yes, and it is unexpected that the converter question comes on the Copy
operation, not the Paste one!

In my stack trace I'm seeing

convert - Converter.cpp:462
prepareHTMLFile - InsetGraphics.cpp:940
...
writeLyXHTMLSource - Buffer.cpp:2239
putClipboard - CutAndPaste.cpp:574
copySelection - CutAndPaste.cpp:1042

interestingly, just above the copySelection() line, we can see:

// We do not need to produce images, etc.
runparams.dryrun = true;

as a consequence, I came up with the attached patch that solves the
annoyance for me, but I'm not expert of this area of the code, so
can please someone review?
(I just checked that I can still copy and paste the graphics as before)

Thanks,

T.

>From f3ec463a3a422dbdf98f61add03f68580ccda653 Mon Sep 17 00:00:00 2001
From: Tommaso Cucinotta 
Date: Thu, 4 May 2017 07:49:07 +0200
Subject: [PATCH] skip graphics conversion when runparams.dryrun is try

... so we avoid the "run converter ?" confirmation dialog for needauth
converters simply when copying (Ctrl+C) an InsetGraphics inset.
---
 src/insets/InsetGraphics.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/src/insets/InsetGraphics.cpp b/src/insets/InsetGraphics.cpp
index 04ed735f..16a482f1 100644
--- a/src/insets/InsetGraphics.cpp
+++ b/src/insets/InsetGraphics.cpp
@@ -947,7 +947,9 @@ string InsetGraphics::prepareHTMLFile(OutputParams const & runparams) const
 
 docstring InsetGraphics::xhtml(XHTMLStream & xs, OutputParams const & op) const
 {
-	string const output_file = prepareHTMLFile(op);
+	string const output_file;
+	if (!op.dryrun)
+		prepareHTMLFile(op);
 
 	if (output_file.empty()) {
 		LYXERR0("InsetGraphics::xhtml: Unable to prepare file `" 
-- 
2.9.3

further add-ons for 2.3.0 ? (was: Re: Gnuplot format & converter script)


As we're on this, a few other things I had in my tommaso/master [1], out of 
which:
I'd really love to have 3)...
about 2) I'm stuck with recurring to Emacs-editing the first line of a .lyx 
file every time I face that issue :-)...
just pushed 1), needs to be tried by someone
guess 4) might be debatable already for a 3.0 :-).

... if there's interest in any of these, it may not be impossible to rebase 
over the w.e. :-)

[1]

1) commit be98c5e1
Author: Tommaso Cucinotta 
Date:   Mon Oct 17 08:44:16 2016 +0200

Enable graphics generation from external gnuplot scripts.

2) commit cb7a69b1
Author: Tommaso Cucinotta 
Date:   Wed Oct 19 11:18:10 2016 +0200

Tolerate formats that are not supported by lyx2lyx.

3) commit bf3cda7b
Author: Tommaso Cucinotta 
Date:   Sat Oct 15 01:14:02 2016 +0200

Create new graphics from within LyX choosing a sample file to copy from.

4) commit 410ce81e
Author: Tommaso Cucinotta 
Date:   Wed Oct 16 22:55:40 2013 +0100

LyX XMPP Chat

This patch enables XMPP-based chatting within LyX.

With contributions from Kornel Benko 


Thanks,

T.

On 04/05/2017 00:54, Tommaso Cucinotta wrote:

On 03/05/2017 15:00, Scott Kostyshak wrote:

Tommaso, I'm still curious if you are planning to implement the gnuplot
patch for 2.3.0? Or perhaps you already did and I missed it?

The reason I ask is that time is running out to get features into 2.3.0
before we should focus completely on bug fixes.


I had it in my own tommaso/master branch, rebased up to last summer 
(LinuxDay@Pisa).

I thought it was late, but as you're asking, I took the chance to rebase the 
patch (trivial), check it's still working as expected, and push [1].

I wish someone else could try it out, in case you're short of gnuplot scripts, you can use 
the attached sample. Just run LyX, and do "Insert->Graphics...", then choose 
the .gp file, then enjoy the plot preview on screen, as well as in the PDF formatted output.

Thx,

    T.

[1]

commit b474aa5d
Author: Tommaso Cucinotta 
Date:   Thu May 4 00:49:18 2017 +0200

Add needauth option to gnuplot->PDF converter introduced in 
[066edd3c/lyxgit].

commit 066edd3c
Author: Tommaso Cucinotta 
Date:   Mon Oct 17 08:44:16 2016 +0200

Enable graphics generation from external gnuplot scripts.

Re: Gnuplot format & converter script


On 03/05/2017 15:00, Scott Kostyshak wrote:

Tommaso, I'm still curious if you are planning to implement the gnuplot
patch for 2.3.0? Or perhaps you already did and I missed it?

The reason I ask is that time is running out to get features into 2.3.0
before we should focus completely on bug fixes.


I had it in my own tommaso/master branch, rebased up to last summer 
(LinuxDay@Pisa).

I thought it was late, but as you're asking, I took the chance to rebase the 
patch (trivial), check it's still working as expected, and push [1].

I wish someone else could try it out, in case you're short of gnuplot scripts, you can use 
the attached sample. Just run LyX, and do "Insert->Graphics...", then choose 
the .gp file, then enjoy the plot preview on screen, as well as in the PDF formatted output.

Thx,

T.

[1]

commit b474aa5d
Author: Tommaso Cucinotta 
Date:   Thu May 4 00:49:18 2017 +0200

Add needauth option to gnuplot->PDF converter introduced in 
[066edd3c/lyxgit].

commit 066edd3c
Author: Tommaso Cucinotta 
Date:   Mon Oct 17 08:44:16 2016 +0200

Enable graphics generation from external gnuplot scripts.




sample.gp
Description: application/gnuplot

Re: [LyX/master] findadv: hide word-findadv verb from the mini-buffer


On 03/05/2017 17:42, Tommaso Cucinotta wrote:

On 03/05/2017 17:37, Tommaso Cucinotta wrote:

Kornel, would you mind to re-run the tests ?


I have this one consistently failing:

findadv-02-in.txt: FAILED

will dig shortly (I'm going off-line a few hours).


this is what I found:

1) started from re-running all the findadv-*.txt tests, and got
   all passing, EXCEPT 02, 08, 20, 21, re-04

2) I need this [1], in order to have findadv-02 and -08 pass;
   the problem is that multiple [Enter] or others seem to hit LyX too fast,
   and, e.g., the second [Enter] arrives while focus was not restored yet on
   the findadv pane, so it deletes the selected found segment from the previous
   [Enter] key press; guess this is related to [515ca693/lyxgit] et al.

3) all others are "flickering" pass/fail, except findadv-21-in.txt that
   keeps consistently failing, because, AFAICR, this one should be the
   bug about lists matching enums and vice-versa, which is still open
   (no on-line access while writing this e-mail)

AFAICS, issues just related to the autotests machinery and keytest.py, so clear
to go for the release, there's no actual problem, as these tests would all pass
if run manually by a user (except -21).

Thanks,

  T.

[1]

$ git diff
diff --git a/development/autotests/findadv-02-in.txt 
b/development/autotests/findadv-02-in.txt
index 7fb840e6..9954b499 100644
--- a/development/autotests/findadv-02-in.txt
+++ b/development/autotests/findadv-02-in.txt
@@ -7,6 +7,7 @@ KK: x^(a) +\\frac 1+x^(a) \[Down]1-x^(a)  \C\[Home]
 KK: \CF
 KK: \Cmx^(a) \[Tab]
 KK: \Cmx_a
+KD: 100
 KK: \[Return]\[Return]\[Return]
 KD: 60
 KK: \Cs
diff --git a/development/autotests/findadv-08-in.txt 
b/development/autotests/findadv-08-in.txt
index a0f6d952..eb0699cf 100644
--- a/development/autotests/findadv-08-in.txt
+++ b/development/autotests/findadv-08-in.txt
@@ -20,6 +20,7 @@ KK: o
 KK: \Axregexp-mode\[Return]
 KK: [\\w]* a
 # select whole words
+KD: 100
 KK: \Ac\Ac
 # search next
 KK: \Al
@@ -33,7 +34,6 @@ KK: \C\[Home]\Axdialog-show findreplaceadv\[Return]
 KK: \Axregexp-mode\[Return]
 KK: .* \Ac\Ac
 KK: \Al\Al\Al
-KK: \[Tab]\[Return]
 TestEnd
 Lang C
 Assert pcregrep -M 'Putting selection at .*idx: 0 par: 0 pos: 10\n with len: 
3' lyx-log3.txt

Re: [LyX/master] findadv: hide word-findadv verb from the mini-buffer


On 03/05/2017 17:42, Tommaso Cucinotta wrote:

On 03/05/2017 17:37, Tommaso Cucinotta wrote:

Kornel, would you mind to re-run the tests ?


I have this one consistently failing:

findadv-02-in.txt: FAILED

will dig shortly (I'm going off-line a few hours).


this is what I found:

1) started from re-running all the findadv-*.txt tests, and got
   all passing, EXCEPT 02, 08, 20, 21, re-04

2) I need this [1], in order to have findadv-02 and -08 pass;
   the problem is that multiple [Enter] or others seem to hit LyX too fast,
   and, e.g., the second [Enter] arrives while focus was not restored yet on
   the findadv pane, so it deletes the selected found segment from the previous
   [Enter] key press; guess this is related to [515ca693/lyxgit] et al.

3) all others are "flickering" pass/fail, except findadv-21-in.txt that
   keeps consistently failing, because, AFAICR, this one should be the
   bug about lists matching enums and vice-versa, which is still open
   (no on-line access while writing this e-mail)

AFAICS, issues just related to the autotests machinery and keytest.py, so clear
to go for the release, there's no actual problem, as these tests would all pass
if run manually by a user (except -21).

Thanks,

  T.

[1]

$ git diff
diff --git a/development/autotests/findadv-02-in.txt 
b/development/autotests/findadv-02-in.txt
index 7fb840e6..9954b499 100644
--- a/development/autotests/findadv-02-in.txt
+++ b/development/autotests/findadv-02-in.txt
@@ -7,6 +7,7 @@ KK: x^(a) +\\frac 1+x^(a) \[Down]1-x^(a)  \C\[Home]
 KK: \CF
 KK: \Cmx^(a) \[Tab]
 KK: \Cmx_a
+KD: 100
 KK: \[Return]\[Return]\[Return]
 KD: 60
 KK: \Cs
diff --git a/development/autotests/findadv-08-in.txt 
b/development/autotests/findadv-08-in.txt
index a0f6d952..eb0699cf 100644
--- a/development/autotests/findadv-08-in.txt
+++ b/development/autotests/findadv-08-in.txt
@@ -20,6 +20,7 @@ KK: o
 KK: \Axregexp-mode\[Return]
 KK: [\\w]* a
 # select whole words
+KD: 100
 KK: \Ac\Ac
 # search next
 KK: \Al
@@ -33,7 +34,6 @@ KK: \C\[Home]\Axdialog-show findreplaceadv\[Return]
 KK: \Axregexp-mode\[Return]
 KK: .* \Ac\Ac
 KK: \Al\Al\Al
-KK: \[Tab]\[Return]
 TestEnd
 Lang C
 Assert pcregrep -M 'Putting selection at .*idx: 0 par: 0 pos: 10\n with len: 
3' lyx-log3.txt

Re: [LyX/master] findadv: hide word-findadv verb from the mini-buffer


On 03/05/2017 17:37, Tommaso Cucinotta wrote:

Kornel, would you mind to re-run the tests ?


I have this one consistently failing:

findadv-02-in.txt: FAILED

will dig shortly (I'm going off-line a few hours).

T.

Re: [LyX/master] findadv: hide word-findadv verb from the mini-buffer


On 03/05/2017 11:26, Jean-Marc Lasgouttes wrote:

3610cdf66944dc790e2e3df666e99f33c45b1ede


Yes, this is much less surprising than to two other commits.


this is now (correctly) fixed in

commit cf6bbe21
Author: Tommaso Cucinotta 
Date:   Wed May 3 17:32:31 2017 +0200

findadv: amend [8c101829/lyxgit] check that opt.find_buf_name is found in 
theBuffers().


and we have back a manually working findadv pane.

Kornel, would you mind to re-run the tests ?

As for the "word-findadv" verb, I don't have strong opinions about hiding that, 
and
I don't know for sure what these "Edit" vs "Hidden" etc. flags are for :(...

Similar for "word-find" which seems to suffer of a similar usability issue 
(from the mini-buf).

Thanks,

T.

Re: [LyX/master] findadv: hide word-findadv verb from the mini-buffer


my bad, quite blind commits without testing, I just reverted the suppression of the 
"word-findadv" lfun name,
but, as you just pointed out, it was actually the crash fix to break all tests 
:-)! (just confirmed that
reverting that as well recovers a working findadv pane)

T.

On 03/05/2017 11:26, Jean-Marc Lasgouttes wrote:

Le 03/05/2017 à 10:44, Kornel Benko a écrit :

Testing manually, I see, that replace is not working. (E.g. findadv does not 
find anything)
Hm, test passes after reverting also 3610cdf66944dc790e2e3df666e99f33c45b1ede


Yes, this is much less surprising than to two other commits.

JMarc

Re: [LyX/master] Added testcase for crash with using function word-findadv


hmm.., not sure if [8c101829/lyxgit] would need a rollback then...

T.

On 02/05/2017 12:23, Kornel Benko wrote:

commit c4c989b9c8d687e6f52d44fc411cf2ad14b2f8c1
Author: Kornel Benko 
Date:   Tue May 2 12:22:09 2017 +0200

Added testcase for crash with using function word-findadv
---
 development/autotests/findadv-crash-in.txt |   10 ++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/development/autotests/findadv-crash-in.txt 
b/development/autotests/findadv-crash-in.txt
new file mode 100644
index 000..bc3affe
--- /dev/null
+++ b/development/autotests/findadv-crash-in.txt
@@ -0,0 +1,10 @@
+# Test for searching with regular expressions
+# Crash in using function 'word-findadv'
+
+Lang C
+TestBegin test.lyx -dbg find > lyx-log.txt 2>&1
+KK: abcd
+KK: \Axword-findadv a\[Return]
+TestEnd
+Lang C
+Assert ! pcregrep 'SIGSEGV signal caught' lyx-log.txt

Re: Crash with using lyx-function word-findadv


On 02/05/2017 18:10, Kornel Benko wrote:

It would still be nice to have the possibility to use minibuffer for scripted 
findadv tests.
(The reason is that the minibuffer commands are language independent)


would make sense as a feature enh request, needs a syntax rethinking.

This is what we have now with the regular find (word-find):

// data is of the form
// "
//"
docstring search;
docstring howto = split(ev.argument(), search, '\n');

bool casesensitive = parse_bool(howto);
bool matchword = parse_bool(howto);
bool forward   = parse_bool(howto);

return findOne(bv, search, casesensitive, matchword, forward, true, 
true);

the multi-line encoding allows for searching single-line arbitrary text without 
having special encoding for that.

The replace options has something more (replaced text).
   // data is of the form
// "
//  
//  "

and this is the crap :-) coming out of a FindAdv request (either find adv and 
replace adv):

ostringstream & operator<<(ostringstream & os, FindAndReplaceOptions const & 
opt)
{
os << to_utf8(opt.find_buf_name) << "\nEOSS\n"
   << opt.casesensitive << ' '
   << opt.matchword << ' '
   << opt.forward << ' '
   << opt.expandmacros << ' '
   << opt.ignoreformat << ' '
   << to_utf8(opt.repl_buf_name) << "\nEOSS\n"
   << opt.keep_case << ' '
   << int(opt.scope) << ' '
   << int(opt.restr);

LYXERR(Debug::FIND, "built: " << os.str());

return os;
}

:-P.

T.

Re: Crash with using lyx-function word-findadv


On 02/05/2017 18:00, Jean-Marc Lasgouttes wrote:

Why not make the function internal, i.e. give it an empty name like 
LFUN_FINISHED_BACKWARD?


u mean, just this [1] ? or, do we need "Hidden" ?

T.

[1]

tommaso@tommylap:~/lyx-trunk-ws/lyx$ git diff
diff --git a/src/LyXAction.cpp b/src/LyXAction.cpp
index e9f45f58..4f65266d 100644
--- a/src/LyXAction.cpp
+++ b/src/LyXAction.cpp
@@ -4069,7 +4069,7 @@ void LyXAction::init()
  * \li Origin: Tommaso, Nov 15 2007
  * \endvar
  */
-   { LFUN_WORD_FINDADV, "word-findadv", ReadOnly | NoBuffer, Edit 
},
+   { LFUN_WORD_FINDADV, "", ReadOnly | NoBuffer, Edit },
 
 /*!

  * \var lyx::FuncCode lyx::LFUN_WORD_FIND_BACKWARD

Re: Crash with using lyx-function word-findadv


On 02/05/2017 11:19, Kornel Benko wrote:

Start lyx
select new file
type anything e.g. 'abcd'
Goto home position
in minibuffer type word-findadv a
Any (or no at all) parameter to this function leads to crash.
This is independent of active search dialog.


(would be great to have an autotest for this :-) )
the attached patch fixes the crash for me, can you please check ?

Can we commit to master now ?


For comparision, the function 'word-find' does not crash, but doesn't find 
anything either.


these LFUNs use a special syntax to be provided, encoding the set of available 
search options as a string; the regular word-find can be hand-made, but the one 
needed by the word-findadv() requires a multi-line options string including the 
name of the search and replace internal buffers within theBuffers(), which are 
generally things internal to LyX and not exposed to the user...
... if we'd like to have a "word-findadv a" command to find "a" with default 
options, that can be achieved by tweaking the

  istringstream & operator>>(istringstream & is, FindAndReplaceOptions & opt)

function in lyxfind.cpp. Can't remember why this is done this way (passing the 
names of these 2 buffers), probably at that time I couldn't find a 
better/simpler way to find those 2 special buffers from the lyxfind.cpp context 
:-(!

T.

>From f2819a9daf5014434f2b39636268cad8455e0bd9 Mon Sep 17 00:00:00 2001
From: Tommaso Cucinotta 
Date: Tue, 2 May 2017 17:46:38 +0200
Subject: [PATCH] findadv: fix crash on wrong syntax/usage of word-findadv LFUN
 from mini-command.

---
 src/lyxfind.cpp | 5 +
 1 file changed, 5 insertions(+)

diff --git a/src/lyxfind.cpp b/src/lyxfind.cpp
index c743fb81..d789c012 100644
--- a/src/lyxfind.cpp
+++ b/src/lyxfind.cpp
@@ -1512,6 +1512,11 @@ bool findAdv(BufferView * bv, FindAndReplaceOptions const & opt)
 	DocIterator cur;
 	int match_len = 0;
 
+	// e.g., when invoking word-findadv from mini-buffer wither with
+	//   wrong options syntax or before ever opening advanced F&R pane
+	if (theBufferList().getBuffer(FileName(to_utf8(opt.find_buf_name))) == 0)
+		return false;
+
 	try {
 		MatchStringAdv matchAdv(bv->buffer(), opt);
 		int length = bv->cursor().selectionEnd().pos() - bv->cursor().selectionBegin().pos();
-- 
2.9.3

Re: Crash with using lyx-function word-findadv