Re: [tex4ht] [bug #345] regression testing is needed

[Michal Hoftich]

Hi Deimi,

> 
> yes, I misunderstood you. Sorry about that.

That's fine :)

> However, starting from the scratch may be not so bad. 
> The system from  scratch should be more clean that 
> tweaked. (l3build can always serve us  as a reference)
> But first we should  understand our needs, and maybe 
> some solutions (I still don't know how to deal with multiple outputs)
> I need to reflect about it.

Maybe the process could be following:

- it should be possible to save the state of correct test run
  automatically
  - we can get list of all generated files from the .lg file
  - save these files in some subdir
  - save the file list in a file
- before test run, delete all generated files - we can get them from the
  .lg file + temporary, aux, dvi
- run the compilation 
  - we can alter $TEXINPUTS to include the trunk/lit
directory, so we don't need to copy the .4ht files to the test dir.
  - ht-fonts dir can be passed to tex4ht with -i option, so it should be
possible to test also .htf files. But I haven't tried that
  - use make4ht for the compilation - we can use build files to run
biber, xindy, makeglossaries etc.
  - we don't need to use directly make4ht executable, but use it as a
library, so we can incorporate it directly to the testing script
- test the log files for the errors
  - stop if they are there
- test the generated files
  - test the existence of all files that should be created
  - test that no other files are referenced in the .lg file
  - use diff for the text files
  - use imagemagick's compare for the pictures
  - we can run specific tests on files with some extension
(validator for HTML?)
  - we can test using regexes
  - we can test using XML tools
  - generally, only the diff test is really needed?
- if there are differences between the saved and actual states which are
  on purpose, save the new state

So this is what I am thinking at the moment, not all of that is really
needed, but the tool should be flexible enough to enable that :)

Here is a sample script, it will get the current $TEXINPUTS variable,
include `tmp` subdirectory of the actual dir at the beginning and
compile the LaTeX document the new value of $TEXINPUTS. Note that it
also solves the issue that path separator in $TEXINPUTS is different on
Unix systems and on Windows (at least I think so)


 kpse.set_program_name "luatex"
 local dir = lfs.currentdir()
 local separator = os.type == "unix"  and ":" or ";"
 local stydir = dir .. "/tmp/"
 
 local texinputs = table.concat({stydir, kpse.expand_var("$TEXINPUTS")}, 
separator)
 print(stydir, ":"..texinputs..":")
 
 os.execute(string.format('export TEXINPUTS=%s && pdflatex sample ', 
texinputs))

Best regards,
Michal

Re: [tex4ht] [bug #345] regression testing is needed

[Michal Hoftich]

Hi Deimi, 

> Hi Michal, 
> 
> Very good.  I've removed  l3build files from the repo.  
> Starting from the scratch  :)
> 

no, you misunderstood me, I meant the opposite, that it can be useful
with some tweaking :) Maybe it sounded in a different way, I was quite
tired when I wrote that, so my writing was maybe even more unclear than
usual.

Best,
Michal

Re: [tex4ht] [bug #345] regression testing is needed

[Deimantas Galcius]

Hi Michal, 

Very good.  I've removed  l3build files from the repo.  
Starting from the scratch  :)

regards, -- deimi 


On Dec 16, 2016, at 4:51 PM, Michal Hoftich wrote:

> Hi all
> 
>> 
>> tex4ht testing needs to deal not with log files, but output files which may
>> be
>> text files (html, xml, multiply html) and/or binary files (e.g. images).
>> I am not sure if it would be good idea to extend l3build to support these
>> features.
>> Rather I think about a different script:
>> build.lua -> (run.lua)
>> l3build.lua -> test4ht.lua (?)
>> regression-test.tex -> regression-test-4ht.tex (?)
>> 
>> Does it sound reasonable?
>> Alternatively, we can always start from scratch.
>> 
>> 
> 
> It indeed sounds reasonable, we have a little bit different needs than
> LaTeX 3, where  they test only the log file, but with all three main
> engines. We need to test several files for each tested file (at least
> the XML and CSS), we also need to be able to pass different options
> for particular tests. I haven't read l3build documentation and source
> code yet, so I have no idea how difficult it will be to get that to
> work. But I am sure that it can't be harder than to start from
> scratch.
> 
> Best regards,
> Michal

Re: [tex4ht] [bug #345] regression testing is needed

[Michal Hoftich]

Hi all

>
> tex4ht testing needs to deal not with log files, but output files which may
> be
> text files (html, xml, multiply html) and/or binary files (e.g. images).
> I am not sure if it would be good idea to extend l3build to support these
> features.
> Rather I think about a different script:
> build.lua -> (run.lua)
> l3build.lua -> test4ht.lua (?)
> regression-test.tex -> regression-test-4ht.tex (?)
>
> Does it sound reasonable?
> Alternatively, we can always start from scratch.
>
>

It indeed sounds reasonable, we have a little bit different needs than
LaTeX 3, where  they test only the log file, but with all three main
engines. We need to test several files for each tested file (at least
the XML and CSS), we also need to be able to pass different options
for particular tests. I haven't read l3build documentation and source
code yet, so I have no idea how difficult it will be to get that to
work. But I am sure that it can't be harder than to start from
scratch.

Best regards,
Michal

Re: [tex4ht] [bug #345] regression testing is needed

[Deimantas Galcius]

Hi Karl,

I took l3build.lua from latex3 repo on github [1].
l3build script has extra features that are irrelevant for tex4ht
- dtx processing, creating doc pdf, preparing release packages for CTAN.
What we are interested in is testing and l3build provides a good infrastructure 
for  testing.
I think about l3build.lua as a starting point building testing system for 
tex4ht:
dropping irrelevant features and adding what is needed.

l3build.lua testing focus on log file and its normalized output.
The supporting file regression-test.tex provides with macros
such as \START.. \STOP,  \OMIT ..\TIMO, that helps to get normalized output.

tex4ht testing needs to deal not with log files, but output files which may be
text files (html, xml, multiply html) and/or binary files (e.g. images).
I am not sure if it would be good idea to extend l3build to support these 
features.
Rather I think about a different script:
build.lua -> (run.lua)
l3build.lua -> test4ht.lua (?)
regression-test.tex -> regression-test-4ht.tex (?)

Does it sound reasonable?
Alternatively, we can always start from scratch.


[1] https://github.com/latex3/latex3/blob/master/l3build/l3build.lua


best regards, -- deimi



On 12/16/2016 01:55 AM, Karl Berry wrote:

 build.lua is a driver file with settings for the main l3build.lua script.

I know.  My question was whether l3build.lua is a fork (modified by you)
or a copy of the original l3build.lua file from the l3build
package. Which you answered below -- you made changes. Ok.

 Probably it would be better to give  it a different name,

build.lua -> l3test4ht.lua comes to mind.

As for l3build.lua, it would be much better not to have it committed in
our repository at all, but to use their installed version. We don't want
to continually be merging new versions, and sticking with an old version
"forever" doesn't sound good either.

 just simple fix in l3build.lua:
os_diffexe  = os.getenv("diffexe") or "diff -c --strip-trailing-cr 
--ignore-matching-lines='

--
--
Deimantas

Re: [tex4ht] [bug #345] regression testing is needed

[Karl Berry]

build.lua is a driver file with settings for the main l3build.lua script.

I know.  My question was whether l3build.lua is a fork (modified by you)
or a copy of the original l3build.lua file from the l3build
package. Which you answered below -- you made changes. Ok.

Probably it would be better to give  it a different name, 

build.lua -> l3test4ht.lua comes to mind.

As for l3build.lua, it would be much better not to have it committed in
our repository at all, but to use their installed version. We don't want
to continually be merging new versions, and sticking with an old version
"forever" doesn't sound good either.

just simple fix in l3build.lua:
   os_diffexe  = os.getenv("diffexe") or "diff -c --strip-trailing-cr 
--ignore-matching-lines='

Re: [tex4ht] [bug #345] regression testing is needed

[Deimantas Galcius]

On 12/15/2016 02:46 AM, Karl Berry wrote:

But we don't really want/need a copy of build.lua in the repo, do we?
Did you have to modify it? I can't tell if the differences from the
distributed version are just different versions, or something you needed
to do.

build.lua is a driver file with settings for the main l3build.lua script.
Probably it would be better to give  it a different name, as
we will want to amend it.


I see it somehow avoids testing the date line?

just simple fix in l3build.lua:

  os_diffexe  = os.getenv("diffexe") or "diff -c --strip-trailing-cr 
--ignore-matching-lines='

Re: [tex4ht] [bug #345] regression testing is needed

[Karl Berry]

Hi Deimi and all,

My idea is to adapt l3build system for tex4ht needs.

I think using l3build for test infrastructure is all to the good.
But we don't really want/need a copy of build.lua in the repo, do we?
Did you have to modify it? I can't tell if the differences from the
distributed version are just different versions, or something you needed
to do.

I see it somehow avoids testing the date line? If so, that alone is
an excellent reason to use it :).

Thanks much,
Karl

Re: [tex4ht] [bug #345] regression testing is needed

[Deimantas Galcius]

Hi Karl, Michal,

I've committed quick and dirty showcase of testing using
l3build system, which l3team has developed and is using for their
testing needs. If you find it suitable, we can adapt it for tex4ht.

Sample files are
testfiles:
  t-suits.lvt  - tex file
  t-suits.tlg - html file (pre-saved)

To run a test (in tests/ folder):
$ make check

This runs "htlatex" on tex file (lvt) and makes diff  of generated html file 
against pre-saved html (tlg).


My idea is to adapt l3build system for tex4ht needs.
But I'm sure you have better ideas.


-- regards, deimi




On 12/14/2016 02:04 AM, Karl Berry wrote:

Follow-up Comment #1, bug #345 (project tex4ht):

I added a top-level tests/ directory with the assorted files I've collected
over the last few years -- hello,world stuff plus some bug examples.

Unfortunately I have done nothing to actually run/regress them, but it's what
I've got. Michal or Deimi, maybe one of you would be able to make a start on
it? Just hoping.

Also, there's a subdirectory tests/biblatex/ with the biblatex tests as I last
received them from CVR. There's been a new biblatex release a few days ago, so
they probably need to be updated.

 ___

Reply to this item at:

   

___
   Message sent via/by Puszcza
   http://puszcza.gnu.org.ua/



--
--
Deimantas

[tex4ht] [bug #345] regression testing is needed

[Karl Berry]

Follow-up Comment #1, bug #345 (project tex4ht):

I added a top-level tests/ directory with the assorted files I've collected
over the last few years -- hello,world stuff plus some bug examples.

Unfortunately I have done nothing to actually run/regress them, but it's what
I've got. Michal or Deimi, maybe one of you would be able to make a start on
it? Just hoping.

Also, there's a subdirectory tests/biblatex/ with the biblatex tests as I last
received them from CVR. There's been a new biblatex release a few days ago, so
they probably need to be updated.

___

Reply to this item at:

  

___
  Message sent via/by Puszcza
  http://puszcza.gnu.org.ua/

[tex4ht] [bug #345] regression testing is needed

[Karl Berry]

URL:
  

 Summary: regression testing is needed
 Project: tex4ht
Submitted by: karl
Submitted on: Sun 11 Dec 2016 01:45:56 AM EET
Category: None
Priority: 5 - Normal
Severity: 5 - Normal
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email: 
 Open/Closed: Open
 Discussion Lock: Any

___

Details:

As in, "make check" should do something. Anything would be better than what we
have now (nothing).

I have some random attempts at test files I could commit, but it might be more
confusing than helpful. I never got to the point of having anything
coherent/consistently useful. Sigh.





___

Reply to this item at:

  

___
  Message sent via/by Puszcza
  http://puszcza.gnu.org.ua/