Re: [NTG-context] roadmap
On 5/15/2018 8:15 AM, Christoph Reller wrote: On Tue, May 15, 2018 at 12:53 AM, wrote: Message: 3 Date: Mon, 14 May 2018 23:26:28 +0200 From: Hans Hagen To: Henning Hraban Ramm , mailing list for ConTeXt users Subject: Re: [NTG-context] roadmap Message-ID: Content-Type: text/plain; charset=utf-8; format=flowed On 5/14/2018 9:36 PM, Henning Hraban Ramm wrote: Hi Hans, thank you so much! Keep up the good work! * PDF/* (X, A, UA - probably only reasonable with external tools, but it seems like there’s not a lot that’s usable) we support various formats (to some extend the backend even adapts to it) ... tagged pdf ... already there for quite a while but i never had any demand for it so i never really check the current state (also because imo it's a it weird feature ... only there because publishers don't want to distribute sources that are suitable for rendering variations ... so we're stuck with some pseudo structure related to visuals) (i might upgrade tagging and exports as they closely relate but it's not easy to get motivated for something that one has no real use for so at most it will cold winter evening stuff) To us, tagged PDF is important. Specifically, for PDF/A with compliance level a, tagging is mandatory. ConTeXt is the only (!) TeX-based PDF creation suite that can produce correct and reliable tagging. Our company is producing (and weekly updating) 67 manuals and technical documents from more than 900 ConTeXt source files for our software products. The output PDFs are converted to PDF/A-2a, which is only possible due to ConTeXt's tagging. From what we hear of our customers, PDF/A level a and PDF/UA are becoming increasingly important, be it because of archiving issues or because of legislation requirements. So please, maintain this invaluable feature! Ah, it's good to hear that it's working ok. (Of course it is/will be maintained ... it's just that because I don't need it myself it's up to folks like you to check if it's ok.) Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___
Re: [NTG-context] roadmap
On Tue, May 15, 2018 at 8:15 AM, Christoph Reller < christoph.rel...@gmail.com> wrote: > > Our company is producing (and weekly updating) 67 manuals and > technical documents from more than 900 ConTeXt source files for our > software products. The output PDFs are converted to PDF/A-2a, which is > only possible due to ConTeXt's tagging. > > What do you think of verapdf ? -- luigi ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___
Re: [NTG-context] roadmap
On Tue, May 15, 2018 at 12:53 AM, wrote: > > > Message: 3 > Date: Mon, 14 May 2018 23:26:28 +0200 > From: Hans Hagen > To: Henning Hraban Ramm , mailing list for ConTeXt > users > Subject: Re: [NTG-context] roadmap > Message-ID: > Content-Type: text/plain; charset=utf-8; format=flowed > > On 5/14/2018 9:36 PM, Henning Hraban Ramm wrote: > > Hi Hans, thank you so much! Keep up the good work! > > * PDF/* (X, A, UA - probably only reasonable with external tools, but it > > seems like there’s not a lot that’s usable) > > we support various formats (to some extend the backend even adapts to > it) ... tagged pdf ... already there for quite a while but i never had > any demand for it so i never really check the current state (also > because imo it's a it weird feature ... only there because publishers > don't want to distribute sources that are suitable for rendering > variations ... so we're stuck with some pseudo structure related to > visuals) > > (i might upgrade tagging and exports as they closely relate but it's not > easy to get motivated for something that one has no real use for so at > most it will cold winter evening stuff) To us, tagged PDF is important. Specifically, for PDF/A with compliance level a, tagging is mandatory. ConTeXt is the only (!) TeX-based PDF creation suite that can produce correct and reliable tagging. Our company is producing (and weekly updating) 67 manuals and technical documents from more than 900 ConTeXt source files for our software products. The output PDFs are converted to PDF/A-2a, which is only possible due to ConTeXt's tagging. From what we hear of our customers, PDF/A level a and PDF/UA are becoming increasingly important, be it because of archiving issues or because of legislation requirements. So please, maintain this invaluable feature! Cheers, and thank you for all the effort. Christoph ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___
Re: [NTG-context] roadmap
On Tue, May 15, 2018 at 12:52 AM, Henri Menke wrote: > > Math typesetting is really crappy in ConTeXt > hm -- luigi ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___
Re: [NTG-context] roadmap
On 5/15/2018 12:52 AM, Henri Menke wrote:
Math typesetting is really crappy in ConTeXt, but I get that this is
beyond your priorities. I plan to develop a module which resembles the
features and syntax of the amsmath LaTeX package for my PhD thesis. I'm
not sure how well this will integrate with the existing mechanisms.
hm. i have no clue what you refer to ... afaik most is configureable
- columnsets, the new ones have considerably fewer features than the old
ones.
like ... but adding some is no problem (only predictable stuff) .. no
column handler suits all (we now also have page columns btw)
- rowwise setups in xtables and maybe columnwise, but that is
computationally expensive.
indeed so that's why we have categories instead
- We can add more trickery for fonts and scripts. There are some
pending extensions.
- Maybe we should provide a few more general styles.
What does that mean? Things like the TUGboat style?
no, e.g. some basic educational stuff
More callbacks. I'm missing callbacks into error handling (i.e.
intercept errors) not just into error reporting like show_error_hook.
if you want to intercept errors then that has to happen at the macro
level, because once tex starts expanding the error can be anywhere
(so, in a macro package one can set at the tex level flags that one can
act upon in the error callback)
(the eror messages themselves might become a layer but that's for later)
Throw out all non-Lua-related primitives and ntg-context@ntg.nlreplace them with Lua
functions. People can then define those primitives themselves, e.g.
way too slow ... in that case i'd drop tex completely (i.e. do all in lua)
also, you can right now (re)define primitives if you like (depending on
the definition of primitive)
\suppressoutererror
becomes
\protected\def\suppressoutererror{%
\directlua{errors.suppressoutererror()}}
This makes it much easier to access that stuff from Lua. Also interface
all the \pdfvariable and \pdfextension stuff to Lua.
all pdf stuff is already doable in lua (context doesn't even use \pdf*
for quite a while)
This should have maybe been done before 1.0 but I guess with 2.0 you can
introduce “breaking” features.
well, a 2.0 (if ever) will probably only be useable for context ...
LuaJIT will always be 5.1 compatible. That is one of the declared goals
of the project. However there exist compatibility layers for Lua which
implement recent features for older interpreters.
https://github.com/keplerproject/lua-compat-5.3
in that case in the end it will be dropped ...
I would rather not see LuaJIT support being dropped. The VM by itself
(without JIT) is already a lot faster than regular Lua and I feel that
the ConTeXt runs benefit from that quite a lot. I use contextjit as my
daily driver.
hm, at most 20% which is also what i get when i buy a new laptop
keep in mind that luajit has some limitations (stack and such)
(and the last few years i managed to squeeze out a lot from lua, and
with lua 5.3 the gaps became narrower)
Hans
-
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-
___
If your question is of interest to others as well, please add an entry to the
Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___
Re: [NTG-context] roadmap
On 15/05/18 03:17, Hans Hagen wrote:
Hi,
The ConTeXt meeting is - as usual - the right place and moment to
discuss the roadmap. We never had real binding roadmaps, more informal
ones. Anyway, here are some thoughts on the two main components: MkIV
and LuaTeX.
ConTeXt MkIV:
- Check if some mechanism can (by now) be simplified due to LuaTeX
extension introduced the last few years that can be considered stable by
now. This has a low impact as we already use Lua a lot.
- Figure out what mechanism in ConTeXt are bottlenecks in performance if
there are such bottlenecks at all. We need user input on this.
- Get rid of inconsistencies in the user interface e.g. by introducing
new commands with settings.
- Check what additional features users want (miss) and decide to what
extent and with what priority we will put effort in this. We've reached
a point where interference prevents more complex extensions.
Math typesetting is really crappy in ConTeXt, but I get that this is
beyond your priorities. I plan to develop a module which resembles the
features and syntax of the amsmath LaTeX package for my PhD thesis. I'm
not sure how well this will integrate with the existing mechanisms.
- Try to improve tricky mechanisms, like columns and tables.
Improvements are of course always on the agenda.
- columnsets, the new ones have considerably fewer features than the old
ones.
- rowwise setups in xtables and maybe columnwise, but that is
computationally expensive.
- We can add more trickery for fonts and scripts. There are some pending
extensions.
- Maybe we should provide a few more general styles.
What does that mean? Things like the TUGboat style?
- Are there reasonable challenges left.
LuaTeX 1.09:
- This version is pretty close to what is the final version (seen from
the functional point of view). We're still debating where to move after
this. LuaTeX 2.0? A stripped down (lean and mean) version specific for
ConTeXt? Keep in mind that we cannot fundamentally change something,
even if we want to, because other macro packages use it and don't expect
it to change much.
More callbacks. I'm missing callbacks into error handling (i.e.
intercept errors) not just into error reporting like show_error_hook.
Throw out all non-Lua-related primitives and replace them with Lua
functions. People can then define those primitives themselves, e.g.
\suppressoutererror
becomes
\protected\def\suppressoutererror{%
\directlua{errors.suppressoutererror()}}
This makes it much easier to access that stuff from Lua. Also interface
all the \pdfvariable and \pdfextension stuff to Lua.
This should have maybe been done before 1.0 but I guess with 2.0 you can
introduce “breaking” features.
- There will probably be some more options in controlling math (given
issues with fonts). We have to accept that not everything has a generic
programmable solution (which is why we have Lua on board).
- There might be a few more callbacks but probably nothing fundamental
is planned.
- We keep cleaning up the code base (less code is better, less
dependencies too, some documentation is missing or not yet adapted to
the new code). For instance the pdf inclusion code will soon be redone
(and then tested in the ConTeXt distribution as usual).
- When possible we will try to improve performance but there is not much
to gain to be expected there.
- We will also keep up with Lua (currently 5.3, some day 5.4). It is
unclear to what extent LuaJit follows. When it stays behind we need to
decide if support in ConTeXt will stay (to some extent we can have dual
code paths as we have now).
LuaJIT will always be 5.1 compatible. That is one of the declared goals
of the project. However there exist compatibility layers for Lua which
implement recent features for older interpreters.
https://github.com/keplerproject/lua-compat-5.3
I would rather not see LuaJIT support being dropped. The VM by itself
(without JIT) is already a lot faster than regular Lua and I feel that
the ConTeXt runs benefit from that quite a lot. I use contextjit as my
daily driver.
- We expect the ffi interface to external libraries to become more
stable over time. ConTeXt will not introduce dependencies (what can be
done in Lua will happen in Lua) but on the other hand we might put some
libraries in the distribution e.g. for database support.
- We might add some extensions to MetaPost in MPLib.
In addition we could formulate ideas with respect to the distribution,
garden, documentation and so on.
You can react on this list but if you come to the meeting, you can
participate in discussions.
So far for now,
Hans
-
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
---
Re: [NTG-context] roadmap
On 5/14/2018 9:36 PM, Henning Hraban Ramm wrote: Hi Hans, thank you so much! Am 2018-05-14 um 17:17 schrieb Hans Hagen : - Get rid of inconsistencies in the user interface e.g. by introducing new commands with settings. +1 - Check what additional features users want (miss) and decide to what extent and with what priority we will put effort in this. We've reached a point where interference prevents more complex extensions. * Recently I had some difficulties with (foot)notes, you’ll remember the margin notes thread. There are still some things that don’t work/look as I’d like them to, and I hope also others would welcome more flexible/simpler placement options. Notes and margins will always be somewhat tricky esp used with other features because in the end tex has to make a choice .. in that respect i think that fully automated typesetting will never be ok. * User/list/structure variables: Probably it’s just me, but there are a few difficulties. hm, they're just data carried with whatever items that have accessors ... and thre tightly bound so real problems cannot happen there * References: see Massimiliano’s question, apparently there are options missing for code before/after the list of pages These hooks were added; adding a few more hooks is normally no big deal ... the main problem with such additions is that they seldom get documented (in fact that imo is also a it up to the one requesting such features). * ePub: I got requests for ePubs again and will look into what’s still wrong/difficult. Last time I checked, footnote markers were accumulating in strange places. Maybe some more default constructs could get a representation in export XML, e.g. \bf, \it It's hard to comment on that one without examples ... keep in mind that the xport is *not* meant as visual companion to the pdf but as a kind of representation of structure as seen by context ... any reordering due to typesetting can reflect that unless one does the obvious: for the expor make a special run with dedicated settings (like making notes end notes) ... notes (and other inserts) are a separate thing in the tex engine and have an asynchronous character. The better the input structure the better the export, and fro real robust multiple usage just code in xml. Visual properties (e.g. fonts) are not part of a structure, but you can just use highlights and such as these get tagged (with classes) so one then can relate them to e.g. fonts or colors. Keep in mind that \bf is not a construct, but a highlight named 'important' is. We can't make structure from non-structure. (FWIW: as long as we don't have projects that need this the priority for such features - the export basically started as a proof of concept - they get a low priority.) * There are a few things missing in image handling - when I wrote my placement macros I needed image size calculations that already exist in ConTeXt but were not easily accessible. hm, there's a lot available (actually also already in mkii) but i fear that there are too many variantions in demands .. there is a rather extensive plugin mechanism too (but not that documented) ... if some info is missing it can be added (although i don't think that there is more available in the engine that we expose) * Would it be possible (or is it already?) to place stuff on layers and let "everything else" run around it? E.g. for half-page images I’m having a hard time using the float placement, while I could simply place images on a layer. I'm not sure what you mean here but for the main flow we only hav eside floats. However, normally everything can be put on / moved to layers (iir the details manual shows some of that) but tex will never be loks dtp. Again, when i'd need that (given that I know what that means) for a project thaen i'd probably look into it. I've learned that much can be done in tex but not all without interference with other mechanisms. - Are there reasonable challenges left. * ePub the export + a (dedicated) css ... exports have to be generic ... i must admit that i gave away my ebook reader as i never use(d) it and bying a new one is not on the agenda/budget (now) * PDF/* (X, A, UA - probably only reasonable with external tools, but it seems like there’s not a lot that’s usable) we support various formats (to some extend the backend even adapts to it) ... tagged pdf ... already there for quite a while but i never had any demand for it so i never really check the current state (also because imo it's a it weird feature ... only there because publishers don't want to distribute sources that are suitable for rendering variations ... so we're stuck with some pseudo structure related to visuals) (i might upgrade tagging and exports as they closely relate but it's not easy to get motivated for something that one has no real use for so at most it will cold winter evening stuff) * even better error messages -
Re: [NTG-context] roadmap
Hi Hans, thank you so much! Am 2018-05-14 um 17:17 schrieb Hans Hagen : > - Get rid of inconsistencies in the user interface e.g. by introducing new > commands with settings. +1 > - Check what additional features users want (miss) and decide to what extent > and with what priority we will put effort in this. We've reached a point > where interference prevents more complex extensions. * Recently I had some difficulties with (foot)notes, you’ll remember the margin notes thread. There are still some things that don’t work/look as I’d like them to, and I hope also others would welcome more flexible/simpler placement options. * User/list/structure variables: Probably it’s just me, but there are a few difficulties. * References: see Massimiliano’s question, apparently there are options missing for code before/after the list of pages * ePub: I got requests for ePubs again and will look into what’s still wrong/difficult. Last time I checked, footnote markers were accumulating in strange places. Maybe some more default constructs could get a representation in export XML, e.g. \bf, \it * There are a few things missing in image handling - when I wrote my placement macros I needed image size calculations that already exist in ConTeXt but were not easily accessible. * Would it be possible (or is it already?) to place stuff on layers and let "everything else" run around it? E.g. for half-page images I’m having a hard time using the float placement, while I could simply place images on a layer. > - Are there reasonable challenges left. * ePub * PDF/* (X, A, UA - probably only reasonable with external tools, but it seems like there’s not a lot that’s usable) * even better error messages - often you get some obscure errors if you just miss a brace or bracket somewhere. I guess that’s a big hurdle for beginners. * documentation... (sigh) While I’m trying to enhance our wiki and my book on ConTeXt, I often need to search the code base for undocumented options or usage examples, sometimes on stuff that looks simple or would be simple to do in InDesign... > LuaTeX 1.09: > - We expect the ffi interface to external libraries to become more stable > over time. ConTeXt will not introduce dependencies (what can be done in Lua > will happen in Lua) but on the other hand we might put some libraries in the > distribution e.g. for database support. Sounds good. I could use JSON, SQLite and/or MySQL support and some user interface stuff – at the moment my invoicing solution is a mixture of Bash, Python, Lua und ConTeXt, I’m already using a minimal OO library (classy.lua) and was looking into tekui, but got stuck... Maybe I should better look into the ConTeXt web framework as GUI. Or finally finish my (Django based) web shop and write a ConTeXt backend for PDF generation... Anyway. Some default imaging solution would be nice (GraphicsMagick library). Greetlings, Hraban --- https://www.fiee.net http://wiki.contextgarden.net GPG Key ID 1C9B22FD ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___
[NTG-context] roadmap
Hi, The ConTeXt meeting is - as usual - the right place and moment to discuss the roadmap. We never had real binding roadmaps, more informal ones. Anyway, here are some thoughts on the two main components: MkIV and LuaTeX. ConTeXt MkIV: - Check if some mechanism can (by now) be simplified due to LuaTeX extension introduced the last few years that can be considered stable by now. This has a low impact as we already use Lua a lot. - Figure out what mechanism in ConTeXt are bottlenecks in performance if there are such bottlenecks at all. We need user input on this. - Get rid of inconsistencies in the user interface e.g. by introducing new commands with settings. - Check what additional features users want (miss) and decide to what extent and with what priority we will put effort in this. We've reached a point where interference prevents more complex extensions. - Try to improve tricky mechanisms, like columns and tables. Improvements are of course always on the agenda. - We can add more trickery for fonts and scripts. There are some pending extensions. - Maybe we should provide a few more general styles. - Are there reasonable challenges left. LuaTeX 1.09: - This version is pretty close to what is the final version (seen from the functional point of view). We're still debating where to move after this. LuaTeX 2.0? A stripped down (lean and mean) version specific for ConTeXt? Keep in mind that we cannot fundamentally change something, even if we want to, because other macro packages use it and don't expect it to change much. - There will probably be some more options in controlling math (given issues with fonts). We have to accept that not everything has a generic programmable solution (which is why we have Lua on board). - There might be a few more callbacks but probably nothing fundamental is planned. - We keep cleaning up the code base (less code is better, less dependencies too, some documentation is missing or not yet adapted to the new code). For instance the pdf inclusion code will soon be redone (and then tested in the ConTeXt distribution as usual). - When possible we will try to improve performance but there is not much to gain to be expected there. - We will also keep up with Lua (currently 5.3, some day 5.4). It is unclear to what extent LuaJit follows. When it stays behind we need to decide if support in ConTeXt will stay (to some extent we can have dual code paths as we have now). - We expect the ffi interface to external libraries to become more stable over time. ConTeXt will not introduce dependencies (what can be done in Lua will happen in Lua) but on the other hand we might put some libraries in the distribution e.g. for database support. - We might add some extensions to MetaPost in MPLib. In addition we could formulate ideas with respect to the distribution, garden, documentation and so on. You can react on this list but if you come to the meeting, you can participate in discussions. So far for now, Hans - Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl - ___ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___
