Re: [Amforth] Ref Card Generation quo vadis?
Dear AmForthers, I tend to disagree on some of this ... What you argue for is a "architecture and project-specific" "reduced" refcard. At least this is, what I understand. I have used the refcard extensively in the beginnings of my AmForth journey. And I would argue any time, that the refcard should please include /all/ /available/ things --- whether or not they are in my build. This is akin to the "Complete Instruction Set Summary" found in the document describing the AVR assembly instructions. Whether or not a given word is /in my build/ ... " words " to rescue. Or just trying this /interactively/ on the serial prompt. Or reading the .lst file from the build (for .asm definitions). The examples by Tristan >> https://tjnw.co.uk//amforth-6.92/appl/arduino/uno.html >> https://tjnw.co.uk//amforth-6.92/appl/arduino/custom.html look nice and clean. However, /I/ did not notice, the words are actually links --- only after Marks comment. For me attributes like this are pretty much /invisible/, although they are quite common (They can be topped by buttons that appear only /after/ I have selected something -- sigh). IMHO a worthwhile extension of the refcard would include a pointer to the source (in readable form), something like > WORD ( stack -- effect ) > - one-line-description > - colon definition equivalent, if the word is in .asm > - .../path/to/file.frt -- one entry if this is a /common/ .frt file > - .../path/to/arch/file.asm -- several entries (one per line) if > this is a target-specific .asm file This way it is clear, where to look, if the word is missing in my build. It even indirectly tells you, on what targets this WORD is available. And all this information should be visible on a printed version of the refcard, too. The pre-made .hex files for the arduino targets are a double edged sword imho: On one hand they considerably lower the bar to get to your first AmForth-prompt. On the other hand, they do not teach users about fuses, about .lst and .map and include files, etc. This is where the "can we please include words X and Y"- wishlist comes from. But with pre-made files, users give up control over their project --- to some extent. I would encourage everyone get on top of the complete tool chain immediately after the first pre-made file runs successfully. But lets assume for this paragraph we want these "target and build specific" refcards (which formats is another question imho). How are we going about them? Is their production included in the project Makefile? Are they produced by /every/ build? How long does that take and what would be an acceptable time? And how much tools do we add as a dependency? The minimal toolchain is already substantial (from my Linux centric point of view): - the sources (via tarball, subversion/git, ...) - the assembler (AvrAsm2 plus wine, unfortunately) - an editor (/any!/) - make to help with the build (could be done with shell commands in principle) - a programmer (hardware) - a program to talk to the programmer (avrdude or similar) - a serial cable of some sort (hardware) - a program to talk to the serial interface (minicom et al., amforth-upload.py, amforth-shell.py) - possibly some programm (evince, xpdf, ...) to read the documentation offline. Even if we produce the refcard using an upgraded perl script, imho this is going the wrong way. One other thing I find interesting to note in this lengthy thread: So far we have only looked at "How to generate the documentation from the sources?" But there is another option: "How to generate the source from a (large) description of the whole system?" This is called "literate programming", as everyone and their dog knows, of course :-) I envision something like "every word has a description, including any reasoning why it is implemented in such a way, plus a description of the source code in some form. From this description a .frt and/or .asm (variable asm formats)" is generated, along with the Technical Handbook, the RefCard and what not. HOWEVER! I have tried to produce code like this. This is a not so modest pain up the backside for a project, which is on the move, which is not fully implemented. Literate Programming alone is unable to track the evolution of the code, unless you rewrite the description every time. I still think there is value to this idea, but it is not the one answer to rule all questions. I will not even try to convert AmForth to such a structure, because it requires a lot of work PLUS an even larger tool chain. :-) Too Long; Didn't Read: - I argue for a full refcard, not a reduced one - I argue for a bit more information on the refcard - I argue against pre-made .hex files - I argue for less dependencies - I do not argue for "produce the code from documentation" :-) - my current priority is "how to make a release". And I'm not there yet. - I'm willing to accept patches fixing the headers of .asm files. - I'm willing to rewrite the refcard scripts, since
Re: [Amforth] Ref Card Generation quo vadis?
Hello Tristan, Erich and AmForthers around the world. Tristan, I really like the direction you have gone in. Coupling the build process with the local refcard really is a solution that I can put my support behind. When push comes to shove what really matters to the end user? The words that have been installed into their system. From what I can see, it just works. AHHH and I just noticed that you linked the names to the source file. That is the solution to having the info you might need easy to find. Super. Since the build process would create individual refcards, it would just be a matter of making group of refcards based on the builds. But that begs the question of what other platforms is AmForth being used on? Is it enough to just make a project that loads every single common and avr8 file and use that for the website. I think maybe yes. Sort of the one refcard to rule them all. In any case, I really like what you've done so far Tristan. It is a big improvement upon the oundated version. Cleaning up the source to give a better card will be a minor issue in the long run since it will only have to be done once. When it is done locally during the build, the created html file should be dumped somewhere convenient, perhaps even in the build directory so one could just click on it to load it into a browser. All the best, Mark On Sat, Sep 19, 2020 at 6:30 PM Tristan Williams wrote: > Hello Mark, Erich, AmForthers, > > I made some more modifications to the perl reference card script so > that it will write out an AVR8 build specific reference card in > html. Below are example outputs for the stock UNO build and > a custom build > > https://tjnw.co.uk//amforth-6.92/appl/arduino/uno.html > https://tjnw.co.uk//amforth-6.92/appl/arduino/custom.html > > The issues relating to the presence and correctness of the > documentation in the .asm files still remain. > > Best wishes, > Tristan > > On 09Sep20 08:38, Tristan Williams wrote: > > Hello Mark, Erich, AmForthers, > > > > Yes, I completely agree the format of my refcard excerpt has some > > issues. I just wanted to show that, with the hard work done by the > > perl script, all of the documentation data fields for AVR8 built-in > > words with compliant doc headers are readily available for > > output using the .lst file. That data could be formatted as desired > > and then written out in html, rst, LaTeX, Lout, etc. > > > > For the above to be useful, the .asm source files need to have > > compliant (and correct) doc headers. Lots of files are, but sufficient > > are not that some coordinated way of doing this is needed. > > > > How is this best done? > > > > Part of my motivation for pursuing this is that I think there is some > > value in having "fuller" pre-built AVR8 hex files in the distribution > > and giving them greater prominence in the documentation[1]. A build > > specific reference card would be helpful in such cases and it ought to > > be created by the same process that creates the hex files. Whilst I > > agree with [2] that it would be impractical to build hex files for an > > extensive combinations of clock frequency/mcu/baud rate, appl/arduino > > already exists and caters for arguably the most popular AVR8 > > development boards plus a few other interesting ones - perhaps ~6 > > pairs of hex files in total. Adding assembler words such as bm-set, > > bm-clear, bm-toggle, sleep, spirw, store-wdc etc. to the default build > > for these larger flash devices would just make the default build more > > useful. > > > > > This does drastically change what the current refcard is. That is to > say, > > > right now it was just a dump of the base system without regard to > usage, > > > more to availability. > > > > It does not have to be a replacement, just something that I think fits > > well with "fuller" pre-built AVR8 hex files and I see as achievable. > > > > > For the website there would have to be cards generated for different > > > architectures and perhaps even NWWR sizes. So, what to do? > > > > Limiting this to AVR8 and those boards that already are in > > appl/arduino (or perhaps should be) etc. makes it simpler. > > > > For existing non AVR8 pre-built hex/binary then having a matching > > refcard would make sense. However, I don't know enough about the details > > of the non AVR8 build process to say whether the same approach would > > work. Also risc-v/words/1-minus.s does not use the same doc headers as > > avr8/words/1minus.asm which suggests problems. > > > > > Maybe the easiest would be to have some generic setups in the appl > > > dir (most likely many already exist) and run the refcard script against > > > them while building for the website (or perhaps even locally, it won't > take > > > much time) then using that to create an array of refcards that could be > > > grouped together as web pages. The point is, that amforth.asm will > dictate > > > what a more or less default system will be and that can be used for the > > > site. > > >
Re: [Amforth] Ref Card Generation quo vadis?
Hello Mark, Erich, AmForthers, I made some more modifications to the perl reference card script so that it will write out an AVR8 build specific reference card in html. Below are example outputs for the stock UNO build and a custom build https://tjnw.co.uk//amforth-6.92/appl/arduino/uno.html https://tjnw.co.uk//amforth-6.92/appl/arduino/custom.html The issues relating to the presence and correctness of the documentation in the .asm files still remain. Best wishes, Tristan On 09Sep20 08:38, Tristan Williams wrote: > Hello Mark, Erich, AmForthers, > > Yes, I completely agree the format of my refcard excerpt has some > issues. I just wanted to show that, with the hard work done by the > perl script, all of the documentation data fields for AVR8 built-in > words with compliant doc headers are readily available for > output using the .lst file. That data could be formatted as desired > and then written out in html, rst, LaTeX, Lout, etc. > > For the above to be useful, the .asm source files need to have > compliant (and correct) doc headers. Lots of files are, but sufficient > are not that some coordinated way of doing this is needed. > > How is this best done? > > Part of my motivation for pursuing this is that I think there is some > value in having "fuller" pre-built AVR8 hex files in the distribution > and giving them greater prominence in the documentation[1]. A build > specific reference card would be helpful in such cases and it ought to > be created by the same process that creates the hex files. Whilst I > agree with [2] that it would be impractical to build hex files for an > extensive combinations of clock frequency/mcu/baud rate, appl/arduino > already exists and caters for arguably the most popular AVR8 > development boards plus a few other interesting ones - perhaps ~6 > pairs of hex files in total. Adding assembler words such as bm-set, > bm-clear, bm-toggle, sleep, spirw, store-wdc etc. to the default build > for these larger flash devices would just make the default build more > useful. > > > This does drastically change what the current refcard is. That is to say, > > right now it was just a dump of the base system without regard to usage, > > more to availability. > > It does not have to be a replacement, just something that I think fits > well with "fuller" pre-built AVR8 hex files and I see as achievable. > > > For the website there would have to be cards generated for different > > architectures and perhaps even NWWR sizes. So, what to do? > > Limiting this to AVR8 and those boards that already are in > appl/arduino (or perhaps should be) etc. makes it simpler. > > For existing non AVR8 pre-built hex/binary then having a matching > refcard would make sense. However, I don't know enough about the details > of the non AVR8 build process to say whether the same approach would > work. Also risc-v/words/1-minus.s does not use the same doc headers as > avr8/words/1minus.asm which suggests problems. > > > Maybe the easiest would be to have some generic setups in the appl > > dir (most likely many already exist) and run the refcard script against > > them while building for the website (or perhaps even locally, it won't take > > much time) then using that to create an array of refcards that could be > > grouped together as web pages. The point is, that amforth.asm will dictate > > what a more or less default system will be and that can be used for the > > site. > > Yes, I would agree - with board level customisation in uno.asm etc as > it is currently. A refcard reflecting the built-in words of hex file > created, be that built locally or pre-built on the website. > > With the current AmForth and AVR8 it is likely the built-in refcards for > appl/arduino boards would be very similar - if not identical. > > So for an AVR8 builtin-ref card I think this is needed > > 1. Compliant doc headers for all the .asm files > 2. Modify the perl script to write out refcard as desired in the desired > formats > 3. Connect this to the build system > 4. Connect the pre-built hex files and their refcards to the main > documentation > > All can be done for a local build setup, but most value would be with > 1 and 4 done at the distribution/website level. > > Best wishes, > Tristan > > [1] https://sourceforge.net/p/amforth/mailman/message/37054617/ > [2] > http://amforth.sourceforge.net/faq.html#there-are-no-hexfiles-in-the-distribution-archive > > > On 08Sep20 13:14, Mark Roth wrote: > > Hello Tristan, Erich and fellow lurking AmForthers (I really do like this > > intro Tristan) :) > > > > It really does seem that you may have caught the tiger by the tail Tristan. > > For better or for worse even! > > > > For the better (hey, you caught the tiger) : > > I think your layout really goes a long way toward documenting the used > > words. The last few days before I saw your mail I had been thinking of this > > very thing, to use the local build for the refcard. I hadn't, however,
Re: [Amforth] Ref Card Generation quo vadis?
Hello Mark, Erich, AmForthers, I agreed with Mark's comment below >> It seems that the intent of the refcard was to document the things that >> are compiled into the system. and commented > For me, the scope of the/each refcard is defined by the > distribution build for each architecture (AVR8, msp430, etc.). If the > refcard script were part of the hex build process then a custom > refcard could be a product of the build process also. For AVR8, the .lst file produced as part of the build process lists all the .asm files used in building the hex files. Modifying Mark's perl script from https://sourceforge.net/p/amforth/mailman/message/37060541/ so that it uses only the included files listed in the .lst file, a list of words with the their documentation fields can output. These are specific to the individual build, rather than to the general assembly source tree. Giving something like this (see end of message). As Mark pointed out >> There is still the issue of files that don't have the 3 lines at the top >> of stack effects, category and description. Making these (assembly) files compliant would require some coordinated effort - some of which has been already done I believe, but after that a build specific ref card documenting built-in words could just be another automated part of the hex build process. Not perhaps the perfect ref card - whatever that is, but achievable and with a clearly defined scope. Certainly something I would make use of. Best wishes, Tristan Edited example (text) output of the modified script for uno.lst Arithmetics --- VOC : 1- DSTACK : ( n1 -- n2 ) RSTACK : CSTACK : DESC : optimized decrement CATEGORY : Arithmetics ASM_FILE : amforth-6.92/avr8/words/1minus.asm VOC : 1+ DSTACK : ( n1|u1 -- n2|u2 ) RSTACK : CSTACK : DESC : optimized increment CATEGORY : Arithmetics ASM_FILE : amforth-6.92/avr8/words/1plus.asm VOC : 2/ DSTACK : ( n1 -- n2 ) RSTACK : CSTACK : DESC : arithmetic shift right CATEGORY : Arithmetics ASM_FILE : amforth-6.92/avr8/words/2slash.asm Compiler VOC : 2literal DSTACK : ( -- x1 x2 ) RSTACK : CSTACK : (C: x1 x2 -- ) DESC : compile a cell pair literal in colon definitions CATEGORY : Compiler ASM_FILE : amforth-6.92/common/words/2literal.asm VOC : again DSTACK : ( -- ) RSTACK : CSTACK : (C: dest -- ) DESC : compile a jump back to dest CATEGORY : Compiler ASM_FILE : amforth-6.92/common/words/again.asm VOC : ahead DSTACK : ( f -- ) RSTACK : CSTACK : (C: -- orig ) DESC : do a unconditional branch CATEGORY : Compiler ASM_FILE : amforth-6.92/common/words/ahead.asm ___ Amforth-devel mailing list for http://amforth.sf.net/ Amforth-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/amforth-devel
Re: [Amforth] Ref Card Generation quo vadis?
Hello Mark, Erich, AmForthers, Mark - I have chopped and reordered selected parts of your message to form my reply. I hope this is OK. > It seems that the intent of the refcard was to document the things that > are compiled into the system. +1 For me, the scope of the/each refcard is defined by the distribution build for each architecture (AVR8, msp430, etc.). If the refcard script were part of the hex build process then a custom refcard could be a product of the build process also. > Since the Forth source code could be nearly anything AND has to be > uploaded after the hex files are uploaded to the device you are now > entering into the realm of full generated documentation. +1 I think AmForth's documentation is very good, but as Mark's draft 6.8+ refcard pointed out to me, there are words I did not know existed. There is more to AmForth than made it to the existing documentation. Complete, exhaustive and automated documentation would clearly be great, but as has already been noted, doing this across AVR8/msp430/risc-v/arm + various assembler formats + common/uncommon Forth is challenging. As previously mentioned, part of the challenge being agreeing on what a refcard should be. My vote would be to 1. Limit the refcard to the words written in assembler that contribute to the distribution build of the hex files. 2. Limit the refcard improvements for AmForth 6.9 to AVR8 only 3. Take this as an opportunity to review the distribution builds for AVR8 (which words to include) and make the existence and location of the hex files more prominent in the documentation. > I think that we will really need a roadmap to head for v6.9 and more > importantly v7.0. I think the distinction between 6.9 and 7.0 is a very helpful one. My opinion is that AVR8 has been and currently is the core platform for AmForth. From the from the mailing list archive it is certainly the one that generates the most activity. Whilst I probably have more than enough AVR8 development boards in the parts box to see out my list of projects, I think Matthias saw a longer term future for AmForth beyond AVR8. Perhaps answers to these two questions go someway towards defining a roadmap for 7.0 1) which, if any, of the other mcus AmForth can run on do AmForthers think is the one to focus on? 2) is there a desire/will for AmForth on AVR8 to be extended to run on newer AVR8 such as the ATMEGA4809[1] ? Best wishes, Tristan [1] http://ww1.microchip.com/downloads/en/DeviceDoc/ATmega4808-09-DataSheet-DS40002173B.pdf On 02Aug20 09:53, Mark Roth wrote: > On Sat, Aug 1, 2020 at 10:50 PM Erich Wälde wrote: > > > thread hijacked intentionally. > > > 'Thar be pirates! > > > > > > today I spent some time trying to understand the "make-refcard*" > > scripts in some detail. > > > > The script works roughly like this: > > > > > > > - the first three lines are comments expected to produce > > 1. the stack effects (data, return, compile stacks) > > 2. the category to which this word belongs > > 3. a one line description, what this word is supposed to do. > > > > > Indeed. It also happens to be the reason the perl script works so well, as > well as being so easy to break. It's just too rigid. One thing that the > original has as well is that the 4th line needs to be the "VE" or "XT" > part. If it isn't it will fail. That was the reason for poking around with > the 3 prior lines part of the the script and hardwiring it for the first 3 > lines. > > > > > > Now I could commit Marks patches and not look further. However, there > > are several shortcomings with the current state. > > > > Yeah, don't do that. I really just put up the diff as a reference for > someone that knows Perl. At best right now I would say you could replace > the very outdated refcard.html with the one my changes generate. It is an > improvement just because of the age difference (5.5 to 6.8). For the long > term however, this tool needs to either be fixed entirely or replaced with > something that the build system can use. Having a refcard generated > directly from the source tree is fantastic IMHO. > > > > > > - The script will currently only read "one" directory, whereas we have > > several directories with /different/ asm styles! > > - arm/words/-- gnu asm style > > - avr8/words/ -- avr asm style > > - common/words/ -- avr, msp430 asm style with .if directives > > - msp430/words/ -- msp430 asm style > > - risc-v/words/ -- riscv asm style > > - shared/words/ -- looks like some macro style for generators > > > > - The generated output (LaTeX, ReST) is done with two different > > scripts. > > > > - The generated output does currently not have any indication, on > > which ports a particular word is available. > > > > This gets to the guts of how to march from here (Google's rendition of the > Latin in the title). Each of the flavors could be determined from their > location in the source tree. The additional information would
Re: [Amforth] Ref Card Generation quo vadis?
On Sat, Aug 1, 2020 at 10:50 PM Erich Wälde wrote: > thread hijacked intentionally. > 'Thar be pirates! > > today I spent some time trying to understand the "make-refcard*" > scripts in some detail. > > The script works roughly like this: > > > - the first three lines are comments expected to produce > 1. the stack effects (data, return, compile stacks) > 2. the category to which this word belongs > 3. a one line description, what this word is supposed to do. > > Indeed. It also happens to be the reason the perl script works so well, as well as being so easy to break. It's just too rigid. One thing that the original has as well is that the 4th line needs to be the "VE" or "XT" part. If it isn't it will fail. That was the reason for poking around with the 3 prior lines part of the the script and hardwiring it for the first 3 lines. > > Now I could commit Marks patches and not look further. However, there > are several shortcomings with the current state. > > Yeah, don't do that. I really just put up the diff as a reference for someone that knows Perl. At best right now I would say you could replace the very outdated refcard.html with the one my changes generate. It is an improvement just because of the age difference (5.5 to 6.8). For the long term however, this tool needs to either be fixed entirely or replaced with something that the build system can use. Having a refcard generated directly from the source tree is fantastic IMHO. > > - The script will currently only read "one" directory, whereas we have > several directories with /different/ asm styles! > - arm/words/-- gnu asm style > - avr8/words/ -- avr asm style > - common/words/ -- avr, msp430 asm style with .if directives > - msp430/words/ -- msp430 asm style > - risc-v/words/ -- riscv asm style > - shared/words/ -- looks like some macro style for generators > > - The generated output (LaTeX, ReST) is done with two different > scripts. > > - The generated output does currently not have any indication, on > which ports a particular word is available. > > This gets to the guts of how to march from here (Google's rendition of the Latin in the title). Each of the flavors could be determined from their location in the source tree. The additional information would have to be added to the hash tables and the generation end would have to be made aware of all this of course. But in those cases it would seem that you would just end up with a mess of duplication. The script would need to determine if something exists already then act accordingly. However, what would be appropriate? How do you deal with differences in the stack effects because of the platform? It seems that the end user would be better served if they could just pull up their flavor of refcard and be done with it. As you mentioned, right now it is the AVR8 refcard. I don't know how different each platform is since I am personally only concerned with the AVR one. But there is of course a wider audience out there and each needs to be dealt with equally. Having individual refcards would seem to align with the way that the website documents already work for the different platforms (sections for linux and windows etc) so that is where I'd cast my vote. > - The script will currently not read any forth code. And words like > "value" or "c," should show up in the refcard as well, shouldn't > they? And should the refcard not have the information that you have > to include one of these files: > :avr8/lib/forth2012/core/c-comma.frt > : msp430/lib/forth-2012/core/c-comma.frt > > I kind of remember that Matthias decided to move some code from the > pre-assembled form /back/ to pure Forth. This was in order to help > dealing with the new, additional architectures. > > - in the .asm files I would also like to see a pure Forth equivalent > as a comment. I have missed this in the past already. > > It seems that the intent of the refcard was to document the things that are compiled into the system. Since the Forth source code could be nearly anything AND has to be uploaded after the hex files are uploaded to the device you are now entering into the realm of full generated documentation. In that case it may be better to look into something like Doxygen (or whatever the kids are using these days) and properly generate all that. Of course that would take a one time scouring of all the sources to align with something of that nature. As for the Forth source in the asm file I'm not sure about that. I think I'd rather see the .frt file alongside the .asm file. That is just personal preference though and I can see the value in having them together. It does seem like it could complicate the build and/or uploading process. ;tldnr Perl script needs to be updated entirely by someone fluent. Sources need to be properly commented for the refcard/doc generation. Maybe define the style to be used (indentation, tabs, spaces etc) for the sources. Forth source for
