Re: [M100] Questions about tokenizing BASIC in UNIX

[Daryl Tester]

On 26/9/22 15:52, John R. Hogerhuis wrote:


Enabled by default. Huh. Weird considering both ends are going to be sending 
their own xon xoff.


Not necessarily.  How the Linux OS will respond to a program that has a port 
open to xon/xoff
will depend on the type of port e.g. if it's a pty, it'll have that interface 
that responds to
stty, and buffer locally.  I don't think the flow control characters get passed 
through -
it's gobbled up by either that terminal control layer (whose name temporarily 
escapes me,
pre coffee) or, as it turns out, the ftdi chip.  The process will be put to 
sleep if it
fills its local buffer, unless the serial port was opened in blocking mode.


Maybe it's a lookahead to interrupt transmission at the hardware level if an 
xoff is received.


Interrupt as in paused - the transmitter just stops sending until it receives 
the start
character.  The chip would be implementing the same mechanism as it does for 
hardware
flow control.  Which is neat.  I just wish it was better documented (still 
hunting for
the exact documentation that enables/disables this, without resorting to their 
API).
There must be some mechanism the USB uses stops sending as well, due to the 
potentially
tiny buffers (which reminds me that I'm still waiting on my USB sniffer - 
another victim
of the silicon shortage).


Cheers,
  --dt

Re: [M100] Questions about tokenizing BASIC in UNIX

[Daryl Tester]

On 26/9/22 18:44, Joshua O'Keefe wrote:


If the kernel has DYNAMIC_DEBUG=y [1] then:
https://stackoverflow.com/a/63682160 


Thanks for that, will also have a poke later (gotta find my other FT2232 board 
as well).


DYNAMIC_DEBUG is pretty neat!  It didn't exist last time I touched driver code.


Yeah, every time I leave for a bit, something changes underneath.  "Where did 
systemtap
go?  Oh, it's eBPF now ..."

Cheers,
  --dt

Re: [M100] Questions about tokenizing BASIC in UNIX

[Joshua O'Keefe]

> On Sep 25, 2022, at 8:59 PM, Daryl Tester 
>  wrote:
> 
> I've got a todo item to see how
> to enable the dev_dbg() messages

I looked, hoping to spare you from rebooting with debug enabled the old 
fashioned way.

If the kernel has DYNAMIC_DEBUG=y [1] then:
https://stackoverflow.com/a/63682160

TL;DR: mount debugfs if not mounted, tell 
/sys/kernel/debug/dynamic_debug/control the kernel file you want output from, 
look in kern.log

Just tested the debug control procedure against ftdi_sio.c and it worked fine 
to produce some output.  Unfortunately said serial chip is downstairs and I'm 
stuck upstairs for several more hours so it's a bit inconvenient to 
reinitialize the chip from here to watch it come up.

DYNAMIC_DEBUG is pretty neat!  It didn't exist last time I touched driver code.

[1] https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html

Re: [M100] Questions about tokenizing BASIC in UNIX

[John R. Hogerhuis]

Enabled by default. Huh. Weird considering both ends are going to be
sending their own xon xoff.

I guess whatever functionality it is creating must coexist with software
flow control happening generated by the two parties.

Or I am just thinking of the feature wrong altogether. Maybe it's a
lookahead to interrupt transmission at the hardware level if an xoff is
received.  Would reduce latency of USB and going up through software
layers.

-- John.

Re: [M100] Questions about tokenizing BASIC in UNIX

[Daryl Tester]

On 24/9/22 02:39, John R. Hogerhuis wrote:


On Fri, Sep 23, 2022 at 2:59 AM Brian White mailto:b.kenyo...@gmail.com>> wrote:



 I believe all the right chip does is help, and make more code more likely to 
make it through



Chip/driver wise the two things I've observed:


Here's some extra Linux/FTDI related stuff.  Abort now if you're driving and 
prone to sleepiness.

Someone (B 9?) mentioned that the FTDI had on chip soft flow control, and I 
hadn't heard of such a
thing either, so I went spelunking through the source code and datasheets.  
Sure enough, the FT2232R
and variants (which is something I use for JTAG/SPI work, as well as serial) 
datasheets mention
"Fully assisted hardware or X-On/X-Off software handshaking", and that's *all* 
they mention of it.
The FT232R datasheet however doesn't mention it, so I wonder if it was 
introduced with their MPSSE
tech (Multi Purpose Synchronous Serial Engine), but I believe that's only used 
for the non-UART
transfers (and couldn't find any mention of it here [1]).  However [2] mentions 
that the FT232R
*is* capable of soft flow control on the chip side, so ¯\_(ツ)_/¯.  Damn sparse 
datasheets.

The Linux FTDI driver (ftdi_sio) does have support for this on-chip software 
handshaking if available
though[3], and is enabled by default if using software flow control.  I've got 
a todo item to see how
to enable the dev_dbg() messages to ensure that it's being enabled, or at least 
poke the chip to read
its current settings via USB.

Without the hardware assist, software flow control will be subject to USB poll 
latency, as the USB
host controller has to poll the serial chip for waiting characters (which is 
then processed by the
appropriate serial driver).  The poll latency is requested by the chip:

$ lsusb | grep Future
Bus 003 Device 044: ID 0403:6010 Future Technology Devices International, Ltd 
FT2232C/D/H Dual UART/FIFO IC
$ lsusb -vd 0403:6010 | egrep 'id[VP]|bInterval'
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
  idVendor   0x0403 Future Technology Devices International, Ltd
  idProduct  0x6010 FT2232C/D/H Dual UART/FIFO IC
bInterval   0
bInterval   0
bInterval   0
bInterval   0

And as bInterval is zero, the driver uses its default:

$ cat 
/sys/devices/pci:00/:00:14.0/usb3/3-1/3-1:1.0/ttyUSB0/latency_timer
16

which I believe is in mS, so this is the potential maximum (not including 
processing overhead)
that the X-OFF character will take to traverse just the USB subsystem.  This is 
adjustable to
the end user, and might be something to try if you, or anyone you love, is 
afflicted by any of
these issues that we've discussed in this email.  Apparently, this matters to 
gamers - there's
a lot of chat around USB gaming mice and USB latency on the interwebz, but it's 
applicable to
a broader range of devices than just mice.  Given an extremely loose 
back-of-envelope figure
of 1mS per character at 9600 bps, that's roughly up to 16 characters (worst 
case) before the
OS knows it has to stop sending, without hardware assist.

That's not taking into account the on chip Rx and Tx buffer sizes that vary 
between chips; the
older FT232R has an Rx (to serial device) of 128 bytes, the newer MPSSE capable 
ones are up
around 4K.

I've also on my todo list to testbench this setup (prior to knowing about the 
software flow
control shenanigans), as I'm interested in other USB serial chips as well, but 
that particular
item is well down on priority.


WAKE UP!


Footnotes:

1. 

2. 

3. 



Cheers,
  --dt

Re: [M100] Questions about tokenizing BASIC in UNIX

[Stephen Adolph]

you are right,  that never really got sorted out.  Would be good to pick
that up again, and try to sort out, using low cost components, how to make
that work.

On Fri, Sep 23, 2022 at 2:00 PM Mike Stein  wrote:

> When Steve and I were playing with M100 to M100 communication over the
> internet way back when, he spent a fair bit of time trying to overcome the
> small buffer in the Model Ts but I don't think he could ever get it to work
> 100%. IIRC reducing the buffer size in the ethernet 'modem' down to one did
> solve the problem but wasn't an option with all versions of the adapters.
>
> m
>
> On Fri, Sep 23, 2022 at 1:09 PM John R. Hogerhuis 
> wrote:
>
>>
>>
>> On Fri, Sep 23, 2022 at 2:59 AM Brian White  wrote:
>>
>>>
>>> I believe all the right chip does is help, and make more code more
>>> likely to make it through
>>>
>>
>> Chip/driver wise the two things I've observed:
>>
>> You can configure the FTDI driver buffering threshold on Windows all the
>> way down to 1 byte, which I think is what we want. This is really about
>> decreasing transmit latency on the Windows side in case of a sensitive
>> receiver on the Model T side... that is, TS-DOS.
>> The prolific drive has bugs under the Windows API, last I checked. There
>> is a facility under Win32 and the .Net Framework for reading a block of
>> data with a timeout. It works with the FTDI, it doesn't work with Prolific.
>> I believe it is a driver issue. I can work around it in software, and have,
>> but just generally spec it out of commercial projects because it is not
>> correct.
>>
>> On Linux, I've never really noticed a difference between Prolific and
>> FTDI. But my experience is the same as yours, that software flow
>> control does not work with our laptop. Linux is too slow to react to a
>> flow-off presumably since it is in-band in the data and gets buffered, and
>> overwhelms the tiny 64-byte, interrupt driven receive buffer on the Model
>> T. And as Mike mentions, USB, Wifi-Serial add create even more overhead and
>> burstiness.
>>
>> I could imagine some low level hardware scheme (specific
>> hardware+driver+configuration) overcoming the buffering of XON/XOFF
>> specifically, in the wired case... but this is the first I've heard of it.
>> And it really seems like it would have to be configured. Hardware isn't
>> normally involved in software flow control.
>>
>> -- John.
>>
>

Re: [M100] Questions about tokenizing BASIC in UNIX

[Mike Stein]

When Steve and I were playing with M100 to M100 communication over the
internet way back when, he spent a fair bit of time trying to overcome the
small buffer in the Model Ts but I don't think he could ever get it to work
100%. IIRC reducing the buffer size in the ethernet 'modem' down to one did
solve the problem but wasn't an option with all versions of the adapters.

m

On Fri, Sep 23, 2022 at 1:09 PM John R. Hogerhuis  wrote:

>
>
> On Fri, Sep 23, 2022 at 2:59 AM Brian White  wrote:
>
>>
>> I believe all the right chip does is help, and make more code more likely
>> to make it through
>>
>
> Chip/driver wise the two things I've observed:
>
> You can configure the FTDI driver buffering threshold on Windows all the
> way down to 1 byte, which I think is what we want. This is really about
> decreasing transmit latency on the Windows side in case of a sensitive
> receiver on the Model T side... that is, TS-DOS.
> The prolific drive has bugs under the Windows API, last I checked. There
> is a facility under Win32 and the .Net Framework for reading a block of
> data with a timeout. It works with the FTDI, it doesn't work with Prolific.
> I believe it is a driver issue. I can work around it in software, and have,
> but just generally spec it out of commercial projects because it is not
> correct.
>
> On Linux, I've never really noticed a difference between Prolific and
> FTDI. But my experience is the same as yours, that software flow
> control does not work with our laptop. Linux is too slow to react to a
> flow-off presumably since it is in-band in the data and gets buffered, and
> overwhelms the tiny 64-byte, interrupt driven receive buffer on the Model
> T. And as Mike mentions, USB, Wifi-Serial add create even more overhead and
> burstiness.
>
> I could imagine some low level hardware scheme (specific
> hardware+driver+configuration) overcoming the buffering of XON/XOFF
> specifically, in the wired case... but this is the first I've heard of it.
> And it really seems like it would have to be configured. Hardware isn't
> normally involved in software flow control.
>
> -- John.
>

Re: [M100] Questions about tokenizing BASIC in UNIX

[John R. Hogerhuis]

On Fri, Sep 23, 2022 at 2:59 AM Brian White  wrote:

>
> I believe all the right chip does is help, and make more code more likely
> to make it through
>

Chip/driver wise the two things I've observed:

You can configure the FTDI driver buffering threshold on Windows all the
way down to 1 byte, which I think is what we want. This is really about
decreasing transmit latency on the Windows side in case of a sensitive
receiver on the Model T side... that is, TS-DOS.
The prolific drive has bugs under the Windows API, last I checked. There is
a facility under Win32 and the .Net Framework for reading a block of data
with a timeout. It works with the FTDI, it doesn't work with Prolific. I
believe it is a driver issue. I can work around it in software, and have,
but just generally spec it out of commercial projects because it is not
correct.

On Linux, I've never really noticed a difference between Prolific and FTDI.
But my experience is the same as yours, that software flow control does not
work with our laptop. Linux is too slow to react to a flow-off presumably
since it is in-band in the data and gets buffered, and overwhelms the tiny
64-byte, interrupt driven receive buffer on the Model T. And as Mike
mentions, USB, Wifi-Serial add create even more overhead and burstiness.

I could imagine some low level hardware scheme (specific
hardware+driver+configuration) overcoming the buffering of XON/XOFF
specifically, in the wired case... but this is the first I've heard of it.
And it really seems like it would have to be configured. Hardware isn't
normally involved in software flow control.

-- John.

Re: [M100] Questions about tokenizing BASIC in UNIX

[Mike Stein]

Terminal sessions aren't usually a problem unless you're sending a file to
the Model T; manual typing in either direction isn't likely to overrun a
buffer and whatever the T is connected to will probably be able to handle
the data stream.

Even receiving a file is usually not a problem if you're connected through
'real' RS232 ports unless the T has to scroll the screen; you may be able
to avoid that.

The main issue is when the T has to spend time processing the incoming data
somehow and it's in packets as it is when using a USB, BT or WiFi<>serial
converter; by the time the sending computer sees an XOFF there are probably
enough characters already on the way to overflow the buffer. Some drivers
have the option to set the packet length, which may solve the issue.

m

On Fri, Sep 23, 2022 at 11:47 AM Brian Brindle  wrote:

> One way under Linux I have gotten around the issues of software flow
> control is to use screen for my serial terminal sessions. It has software
> flow control support built in. A must if you want to use ssh or another app
> that doesn't play well with stty.
>
> Brian
>
>
> On Fri, Sep 23, 2022, 5:59 AM Brian White  wrote:
>
>> I don't believe this is enough to ensure clean data. I must have a dozen
>> real ftdi adapters from different manufacturers and I keep having to
>> increase the per-byte sleep in dlplus' bootstrapper and tsend.ps1 and
>> PDDuino as I discover new basic programs to feed in.
>>
>> Some programs will ingest fine at 5ms/byte, some will fail on some
>> particular part of one line, always the same spot, unless the sleep is
>> increased to 6ms.
>>
>> Then I discover some other code that fails at 6 but succeeds at 7.
>>
>> Some of the problem code I just found in the M100SIG, some were new
>> things like the REX setup utils, some I created the new problem by
>> refactoring existing tpdd client loaders to make them smaller. This made
>> them apparently more difficult to ingest, but they are still legal correct
>> BASIC, so any procedure needs to be able to handle the worst case or else
>> it's no good.
>>
>> For an example, one of the loaders I re-worked, used to work fine before,
>> and then the reworked version would always be missing a : in the middle of
>> one long line of code. Always the same byte missing. And it didn't cause
>> the transfer to fail immediately, it tokenized on the fly without apparent
>> error. But the program was broken and didn't work when you try to run it.
>>
>> This is all with real ftdi adapters and with xon/off enabled on both ends.
>>
>> It may work at 1200 or something where the low baud rate does the same
>> job as a sleep between bytes, but no way at 19200.
>>
>> I will put together a few test cases that expose the problem. And if this
>> method does somehow work then that would indeed be pretty interesting and
>> good to know.
>>
>> I believe all the right chip does is help, and make more code more likely
>> to make it through, but not actually guarantee it. It probably does make a
>> large improvement, and you can probably show numerous examples of data that
>> fails on one adapter and succeeds on an ftdi chip.
>>
>> --
>> bkw
>>
>> On Thu, Sep 22, 2022, 8:57 PM John R. Hogerhuis  wrote:
>>
>>>
>>>
>>> On Thu, Sep 22, 2022 at 5:32 PM B 9  wrote:
>>>

 Of particular note for troubleshooting is that, if some of the data
 gets transferred, but it is garbled or you get a ?DS ERROR, then the
 problem is that your PC's serial port does not support "ON CHIP SOFTWARE
 FLOW CONTROL".  One solution is to buy a serial card or USB cable with an
 FTDI chip in it. (Other companies make ICs that support on chip software
 flow control, I even have a cheap Prolific 2303 device that does, but FTDI
 is the only company I know of whose chips are supposed to always work. )


>>> I've never heard of this on-chip software flow control.All I knew was
>>> that software flow does not work under Linux. My experience is software
>>> flow control with a Model T *does* work on Windows.
>>>
>>> Is that something you have to enable? Seems awfully strange to have the
>>> chip interceding in-band. Automatic hardware flow control I have heard of.
>>>
>>> This failure on Linux is why I made HTERM to use hardware flow control
>>>
>>> If there's a consistent way to make it work on Linux, it would be good
>>> to know. Even just with FTDI chipset. Windows prolific drivers are trash. I
>>> don't know about Prolific on Linux.
>>>
>>> Is the stty command you put supposed to enable on-chip software flow
>>> control? Which part?
>>>
>>> *s**tty -F /dev/ttyUSB0 ixon ixoff  stop ^S start ^Q  -onlcr -icrnl
>>>  eof ^Z  19200*
>>>
>>> -- John.
>>>

Re: [M100] Questions about tokenizing BASIC in UNIX

[Brian Brindle]

One way under Linux I have gotten around the issues of software flow
control is to use screen for my serial terminal sessions. It has software
flow control support built in. A must if you want to use ssh or another app
that doesn't play well with stty.

Brian


On Fri, Sep 23, 2022, 5:59 AM Brian White  wrote:

> I don't believe this is enough to ensure clean data. I must have a dozen
> real ftdi adapters from different manufacturers and I keep having to
> increase the per-byte sleep in dlplus' bootstrapper and tsend.ps1 and
> PDDuino as I discover new basic programs to feed in.
>
> Some programs will ingest fine at 5ms/byte, some will fail on some
> particular part of one line, always the same spot, unless the sleep is
> increased to 6ms.
>
> Then I discover some other code that fails at 6 but succeeds at 7.
>
> Some of the problem code I just found in the M100SIG, some were new things
> like the REX setup utils, some I created the new problem by refactoring
> existing tpdd client loaders to make them smaller. This made them
> apparently more difficult to ingest, but they are still legal correct
> BASIC, so any procedure needs to be able to handle the worst case or else
> it's no good.
>
> For an example, one of the loaders I re-worked, used to work fine before,
> and then the reworked version would always be missing a : in the middle of
> one long line of code. Always the same byte missing. And it didn't cause
> the transfer to fail immediately, it tokenized on the fly without apparent
> error. But the program was broken and didn't work when you try to run it.
>
> This is all with real ftdi adapters and with xon/off enabled on both ends.
>
> It may work at 1200 or something where the low baud rate does the same job
> as a sleep between bytes, but no way at 19200.
>
> I will put together a few test cases that expose the problem. And if this
> method does somehow work then that would indeed be pretty interesting and
> good to know.
>
> I believe all the right chip does is help, and make more code more likely
> to make it through, but not actually guarantee it. It probably does make a
> large improvement, and you can probably show numerous examples of data that
> fails on one adapter and succeeds on an ftdi chip.
>
> --
> bkw
>
> On Thu, Sep 22, 2022, 8:57 PM John R. Hogerhuis  wrote:
>
>>
>>
>> On Thu, Sep 22, 2022 at 5:32 PM B 9  wrote:
>>
>>>
>>> Of particular note for troubleshooting is that, if some of the data gets
>>> transferred, but it is garbled or you get a ?DS ERROR, then the problem
>>> is that your PC's serial port does not support "ON CHIP SOFTWARE FLOW
>>> CONTROL".  One solution is to buy a serial card or USB cable with an FTDI
>>> chip in it. (Other companies make ICs that support on chip software flow
>>> control, I even have a cheap Prolific 2303 device that does, but FTDI is
>>> the only company I know of whose chips are supposed to always work. )
>>>
>>>
>> I've never heard of this on-chip software flow control.All I knew was
>> that software flow does not work under Linux. My experience is software
>> flow control with a Model T *does* work on Windows.
>>
>> Is that something you have to enable? Seems awfully strange to have the
>> chip interceding in-band. Automatic hardware flow control I have heard of.
>>
>> This failure on Linux is why I made HTERM to use hardware flow control
>>
>> If there's a consistent way to make it work on Linux, it would be good to
>> know. Even just with FTDI chipset. Windows prolific drivers are trash. I
>> don't know about Prolific on Linux.
>>
>> Is the stty command you put supposed to enable on-chip software flow
>> control? Which part?
>>
>> *s**tty -F /dev/ttyUSB0 ixon ixoff  stop ^S start ^Q  -onlcr -icrnl  eof
>> ^Z  19200*
>>
>> -- John.
>>
>>>

Re: [M100] Questions about tokenizing BASIC in UNIX

[Brian White]

I don't believe this is enough to ensure clean data. I must have a dozen
real ftdi adapters from different manufacturers and I keep having to
increase the per-byte sleep in dlplus' bootstrapper and tsend.ps1 and
PDDuino as I discover new basic programs to feed in.

Some programs will ingest fine at 5ms/byte, some will fail on some
particular part of one line, always the same spot, unless the sleep is
increased to 6ms.

Then I discover some other code that fails at 6 but succeeds at 7.

Some of the problem code I just found in the M100SIG, some were new things
like the REX setup utils, some I created the new problem by refactoring
existing tpdd client loaders to make them smaller. This made them
apparently more difficult to ingest, but they are still legal correct
BASIC, so any procedure needs to be able to handle the worst case or else
it's no good.

For an example, one of the loaders I re-worked, used to work fine before,
and then the reworked version would always be missing a : in the middle of
one long line of code. Always the same byte missing. And it didn't cause
the transfer to fail immediately, it tokenized on the fly without apparent
error. But the program was broken and didn't work when you try to run it.

This is all with real ftdi adapters and with xon/off enabled on both ends.

It may work at 1200 or something where the low baud rate does the same job
as a sleep between bytes, but no way at 19200.

I will put together a few test cases that expose the problem. And if this
method does somehow work then that would indeed be pretty interesting and
good to know.

I believe all the right chip does is help, and make more code more likely
to make it through, but not actually guarantee it. It probably does make a
large improvement, and you can probably show numerous examples of data that
fails on one adapter and succeeds on an ftdi chip.

-- 
bkw

On Thu, Sep 22, 2022, 8:57 PM John R. Hogerhuis  wrote:

>
>
> On Thu, Sep 22, 2022 at 5:32 PM B 9  wrote:
>
>>
>> Of particular note for troubleshooting is that, if some of the data gets
>> transferred, but it is garbled or you get a ?DS ERROR, then the problem
>> is that your PC's serial port does not support "ON CHIP SOFTWARE FLOW
>> CONTROL".  One solution is to buy a serial card or USB cable with an FTDI
>> chip in it. (Other companies make ICs that support on chip software flow
>> control, I even have a cheap Prolific 2303 device that does, but FTDI is
>> the only company I know of whose chips are supposed to always work. )
>>
>>
> I've never heard of this on-chip software flow control.All I knew was that
> software flow does not work under Linux. My experience is software flow
> control with a Model T *does* work on Windows.
>
> Is that something you have to enable? Seems awfully strange to have the
> chip interceding in-band. Automatic hardware flow control I have heard of.
>
> This failure on Linux is why I made HTERM to use hardware flow control
>
> If there's a consistent way to make it work on Linux, it would be good to
> know. Even just with FTDI chipset. Windows prolific drivers are trash. I
> don't know about Prolific on Linux.
>
> Is the stty command you put supposed to enable on-chip software flow
> control? Which part?
>
> *s**tty -F /dev/ttyUSB0 ixon ixoff  stop ^S start ^Q  -onlcr -icrnl  eof
> ^Z  19200*
>
> -- John.
>
>>

Re: [M100] Questions about tokenizing BASIC in UNIX

[John R. Hogerhuis]

On Thu, Sep 22, 2022 at 5:32 PM B 9  wrote:

>
> Of particular note for troubleshooting is that, if some of the data gets
> transferred, but it is garbled or you get a ?DS ERROR, then the problem
> is that your PC's serial port does not support "ON CHIP SOFTWARE FLOW
> CONTROL".  One solution is to buy a serial card or USB cable with an FTDI
> chip in it. (Other companies make ICs that support on chip software flow
> control, I even have a cheap Prolific 2303 device that does, but FTDI is
> the only company I know of whose chips are supposed to always work. )
>
>
I've never heard of this on-chip software flow control.All I knew was that
software flow does not work under Linux. My experience is software flow
control with a Model T *does* work on Windows.

Is that something you have to enable? Seems awfully strange to have the
chip interceding in-band. Automatic hardware flow control I have heard of.

This failure on Linux is why I made HTERM to use hardware flow control

If there's a consistent way to make it work on Linux, it would be good to
know. Even just with FTDI chipset. Windows prolific drivers are trash. I
don't know about Prolific on Linux.

Is the stty command you put supposed to enable on-chip software flow
control? Which part?

*s**tty -F /dev/ttyUSB0 ixon ixoff  stop ^S start ^Q  -onlcr -icrnl  eof ^Z
 19200*

-- John.

>

Re: [M100] Questions about tokenizing BASIC in UNIX

[B 9]

Oops, small typo. I forgot to send the ^Z at the end of the file to the
Model T. Step 2 should be:


*(cat FOO.DO; echo $'\cZ'; read)  > /dev/ttyUSB0*

Little details like that are why I have the recipe wrapped up in a shell
script.

—b9

On Thu, Sep 22, 2022 at 5:31 PM B 9  wrote:

> Just a note for anyone interested: Birt was right. A Model T *can* tokenize
> a BASIC program one line at a time from any serial port. This is much
> better than converting a .DO file which can easily fill up RAM. If you
> weren't lucky enough to get a Backpack before they sold out, here's how
> I've been tokenizing on the fly:
>
>1. On my Tandy 200:
>LOAD "COM:98N1ENN"
>2. On my GNU/Linux box, hooked up via a NULL modem:
>*(cat FOO.DO; read)  > /dev/ttyUSB0*
>3. When the Model T says "Ok",
>   1. On my PC, I hit *Enter*
>   2. On my Tandy 200, I type: SAVE "FOO"
>
> This looks simple but the serial port handling for Linux is a bit funny. I
> needed the *read* command because the port data was getting cut off as
> soon as *cat* finished. Also, it is necessary to preconfigure the serial
> port using *stty* to make sure it has the right baudrate, etc. I do that
> once, in step 0
>
> 0. Configure serial port the way the Tandy portables like.
> *s*
> *tty -F /dev/ttyUSB0 ixon ixoff  stop ^S start ^Q  -onlcr -icrnl  eof ^Z
>  19200*
>
>
> Since that's a mouthful to type, I put all of the above steps in a shell
> script, which also includes commentary on what the settings mean and
> possible sources of errors. (Please see attachment).
>
> Of particular note for troubleshooting is that, if some of the data gets
> transferred, but it is garbled or you get a ?DS ERROR, then the problem
> is that your PC's serial port does not support "ON CHIP SOFTWARE FLOW
> CONTROL".  One solution is to buy a serial card or USB cable with an FTDI
> chip in it. (Other companies make ICs that support on chip software flow
> control, I even have a cheap Prolific 2303 device that does, but FTDI is
> the only company I know of whose chips are supposed to always work. )
>
> Of course, the simplest solution is to get a backpack once Birt has them
> back in stock.
>
> —b9
>
> P.S. Why on chip xon/xoff is needed: It took me a long time to figure this
> out, but software flow control no longer works on most modern PCs. For
> speed, serial ports use long FIFOs (buffers) which increases throughput at
> the cost of latency. By the time the ^S character (XOFF == "please stop
> transmitting") from the Model T gets to the PC, the PC will have already
> written a lot of data that the Model T has to discard.
>
>
>
> On Wed, Jul 27, 2022 at 5:15 AM  wrote:
>
>> Supply chain issues still abound for all. We are working on trying to get
>> some more Backpacks made. I think you could use this same idea for any sort
>> of serial loading of programs.
>>
>>
>>
>> Jeff Birt
>>
>>
>>
>> *From:* M100  *On Behalf Of *B 9
>> *Sent:* Wednesday, July 27, 2022 2:28 AM
>> *To:* m...@bitchin100.com
>> *Subject:* Re: [M100] Questions about tokenizing BASIC in UNIX
>>
>>
>>
>> What a nifty use of the Backpack's command line interface! I saw your
>> YouTube on the Backpack last year and thought they looked intriguing. Did
>> the supply chain issues ever get resolved?
>>
>>
>>
>> —b9
>>
>

Re: [M100] Questions about tokenizing BASIC in UNIX

[B 9]

Just a note for anyone interested: Birt was right. A Model T *can* tokenize
a BASIC program one line at a time from any serial port. This is much
better than converting a .DO file which can easily fill up RAM. If you
weren't lucky enough to get a Backpack before they sold out, here's how
I've been tokenizing on the fly:

   1. On my Tandy 200:
   LOAD "COM:98N1ENN"
   2. On my GNU/Linux box, hooked up via a NULL modem:
   *(cat FOO.DO; read)  > /dev/ttyUSB0*
   3. When the Model T says "Ok",
  1. On my PC, I hit *Enter*
  2. On my Tandy 200, I type: SAVE "FOO"

This looks simple but the serial port handling for Linux is a bit funny. I
needed the *read* command because the port data was getting cut off as soon
as *cat* finished. Also, it is necessary to preconfigure the serial port
using *stty* to make sure it has the right baudrate, etc. I do that once,
in step 0

0. Configure serial port the way the Tandy portables like.
*s*
*tty -F /dev/ttyUSB0 ixon ixoff  stop ^S start ^Q  -onlcr -icrnl  eof ^Z
 19200*


Since that's a mouthful to type, I put all of the above steps in a shell
script, which also includes commentary on what the settings mean and
possible sources of errors. (Please see attachment).

Of particular note for troubleshooting is that, if some of the data gets
transferred, but it is garbled or you get a ?DS ERROR, then the problem is
that your PC's serial port does not support "ON CHIP SOFTWARE FLOW
CONTROL".  One solution is to buy a serial card or USB cable with an FTDI
chip in it. (Other companies make ICs that support on chip software flow
control, I even have a cheap Prolific 2303 device that does, but FTDI is
the only company I know of whose chips are supposed to always work. )

Of course, the simplest solution is to get a backpack once Birt has them
back in stock.

—b9

P.S. Why on chip xon/xoff is needed: It took me a long time to figure this
out, but software flow control no longer works on most modern PCs. For
speed, serial ports use long FIFOs (buffers) which increases throughput at
the cost of latency. By the time the ^S character (XOFF == "please stop
transmitting") from the Model T gets to the PC, the PC will have already
written a lot of data that the Model T has to discard.



On Wed, Jul 27, 2022 at 5:15 AM  wrote:

> Supply chain issues still abound for all. We are working on trying to get
> some more Backpacks made. I think you could use this same idea for any sort
> of serial loading of programs.
>
>
>
> Jeff Birt
>
>
>
> *From:* M100  *On Behalf Of *B 9
> *Sent:* Wednesday, July 27, 2022 2:28 AM
> *To:* m...@bitchin100.com
> *Subject:* Re: [M100] Questions about tokenizing BASIC in UNIX
>
>
>
> What a nifty use of the Backpack's command line interface! I saw your
> YouTube on the Backpack last year and thought they looked intriguing. Did
> the supply chain issues ever get resolved?
>
>
>
> —b9
>


sendtomodelt
Description: Binary data

Re: [M100] Questions about tokenizing BASIC in UNIX

[birt_j]

Supply chain issues still abound for all. We are working on trying to get some 
more Backpacks made. I think you could use this same idea for any sort of 
serial loading of programs.

 

Jeff Birt

 

From: M100  On Behalf Of B 9
Sent: Wednesday, July 27, 2022 2:28 AM
To: m...@bitchin100.com
Subject: Re: [M100] Questions about tokenizing BASIC in UNIX

 

What a nifty use of the Backpack's command line interface! I saw your YouTube 
on the Backpack last year and thought they looked intriguing. Did the supply 
chain issues ever get resolved?

 

—b9

Re: [M100] Questions about tokenizing BASIC in UNIX

[B 9]

On Tue, Jul 26, 2022 at 5:15 AM  wrote:

> A typical issue when trying to load a DO and convert it to a BA is not
> having enough RAM for both copies in memory at the same time. One of the
> Backpack owners whipped up this little program to solve that problem. It
> loads in one line at a time from the DO file and tokenizes it. Listing
> below and program as a DO file (even though it is named BA) attached.
>
> Jeff Birt
>

What a nifty use of the Backpack's command line interface! I saw your
YouTube on the Backpack last year and thought they looked intriguing. Did
the supply chain issues ever get resolved?

—b9

Re: [M100] Questions about tokenizing BASIC in UNIX

[John R. Hogerhuis]

I was looking for a little pokes and machine code but it ain't there. All
BASIC. Probably super slow too. But clever nonetheless.

-- John.

Re: [M100] Questions about tokenizing BASIC in UNIX

[John R. Hogerhuis]

Laddiealpha and probably other file services only support BA and maybe B?
to indicate tokenized files.  Not 3 letters and no distinction between
platforms. It's up to the user to either load a DO file or make sure the BA
file is appropriate for their platform.

The only caveat is Laddiealpha does a tokenization check to see if a DO
file has been misnamed as BA to prevent memory corruption, and maps the
filename in the directory from BA to DO.

-- John.

Re: [M100] Questions about tokenizing BASIC in UNIX

[birt_j]

A typical issue when trying to load a DO and convert it to a BA is not having 
enough RAM for both copies in memory at the same time. One of the Backpack 
owners whipped up this little program to solve that problem. It loads in one 
line at a time from the DO file and tokenizes it. Listing below and program as 
a DO file (even though it is named BA) attached.

Jeff Birt

1 'DNLOAD.BA - LOAD a BASIC .DO program directly from the Backpack

5 By - Rocket Fueled

10 MAXFILES=2:DEFSTR A-F,Q:D="COM:98N1E"

20 OPEN D FOR OUTPUT AS 1

30 OPEN D FOR INPUT AS 2

40 C=CHR$(13):Q=CHR$(34)

50 PRINT "DO program filename to load";:INPUT F

60 PRINT #1,C;C;C;C;

70 A=INPUT$(1,2):PRINT A;:IF A<>"#" THEN 70

80 A=INPUT$(1,2):PRINT A

90 A="download "+F+C

100 FOR I=1 TO LEN(A):PRINT #1,MID$(A,I,1);:B=INPUT$(1,2):PRINT B;:NEXT I

110 FOR I=1 TO 3:GOSUB 200:NEXT I:PRINT ""

120 PRINT "":PRINT "Wait up to 60 sec for the Backpack LED"

130 PRINT "to start. When it stops blinking, press"

140 PRINT "SHIFT-BREAK and type SAVE ";Q;LEFT$(F,LEN(F)-3);".BA";Q

150 PRINT "LOAD ";Q;D;Q : LOAD D

160 END

200 A=INPUT$(1,2):PRINT A;:IF A<>C THEN 200 ELSE RETURN

 

From: M100  On Behalf Of B 9
Sent: Tuesday, July 26, 2022 1:46 AM
To: m...@bitchin100.com
Cc: Model 100 Discussion 
Subject: Re: [M100] Questions about tokenizing BASIC in UNIX

 

Thanks for all the tips, everyone!

 



DNLOAD.BA
Description: Binary data

Re: [M100] Questions about tokenizing BASIC in UNIX

[B 9]

Thanks for all the tips, everyone!

I had no idea VirtualT was scriptable via telnet. That is so convenient
that I almost didn't bother writing my own tokenizer. But I realized that,
now that I have a table of the tokens
, I
could just plug it into a lexer like Flex and have it generate a C program
for me. So that's what I did: https://github.com/hackerb9/tokenize. It
should work for Model 100/102 and Tandy 200. Creating a version for the
PC-8201/8300 could be as simple as plugging in the different token values
. I can't guarantee that it
works for everything, but I tried it on a couple programs and it is giving
me exactly the same file. (Okay, not "exactly" as I'm not calculating the
memory offsets which the Model 100 recomputes when the file is loaded.)

Now I have another question: Are there different filename extensions for
the different tokenized BASICs? I notice that the ROM2/Cleuseau
documentation mentions that Tandy's BASIC and NEC's BASIC are tokenized
differently. It has a "Strong machine file type mode" that forces the
filename extension on a TDD to be *.BA1* for Model 100/102, *.BA2* for
Tandy 200, and *.BA0* for PC-8201A. Are these well known enough extensions
that I should use them in a new project? Or should I come up with something
different, like *.BA.NEC.*

—b9

On Mon, Jul 25, 2022 at 4:52 AM Ken Pettit  wrote:

> M100/M102 and T200 all use the same token format, but 8201A/8300 is
> different from the other 3 variants.  You could try using M100 VirtualT
> with 32K to tokenize and save the .BA file, then load it into T200.
>
> Ken
>
> On 7/24/22 11:42 PM, John R. Hogerhuis wrote:
> > Not as far as I know. I think Robert Pigford's program is it.
> > PowerBASIC is a compiler though so I guess its binaries would run
> > under a Linux DOS emulator. I've certainly run DOS assemblers like
> > TASM as part of a GNU/Linux Makefile.
> >
> > You would need a tokenizer for each BASIC variant you want to support
> > (M100/102, 8201A/8300, and maybe t200?)
> >
> > The ROM of each machine is of course already capable of doing it
> > properly by definition via LOAD/CLOAD. So emulators effectively
> > support tokenization as well. VirtualT is fully scriptable over
> > telnet/socket.
> >
> > For myself I just use my laptop or emulator to created tokenized files.
> >
> > One note... there are valid, specially constructed BA  files that do
> > not fully LIST, and are not convertible to ASCII BASIC text and back
> > again, with high numbered lines and/or machine language embedded in
> > it. To create such files goes beyond simple tokenization, but it may
> > be useful and interesting to create advanced special BA binary file
> > generators. Most such files require special measures to create.
> >
> > -- John.
>
>

Re: [M100] Questions about tokenizing BASIC in UNIX

[MikeS]

If B9's UNIX BASIC is somewhat compatible and he promises not to laugh or 
criticize I can try to find the Tbasic source code.

m
  - Original Message - 
  From: Brad Grier 
  To: m...@bitchin100.com 
  Sent: Monday, July 25, 2022 3:32 PM
  Subject: Re: [M100] Questions about tokenizing BASIC in UNIX


  Club100 Member directory for Mike Stein has Entoke.exe - 
http://www.club100.org/memfiles/index.php?=0=nom=Mike%20Stein


  And a wayback machine crawl reveals detoke.exe - 
https://web.archive.org/web/2017155020/http://www.xibalba.com/website100/software.html




  On Mon, Jul 25, 2022 at 11:58 AM Peter Vollan  wrote:

There are tokenise and detokenise programs for DOS, in the Compuserve 
archive, I think.


On Sun, 24 Jul 2022 at 23:42, John R. Hogerhuis  wrote:

  Not as far as I know. I think Robert Pigford's program is it. PowerBASIC 
is a compiler though so I guess its binaries would run under a Linux DOS 
emulator. I've certainly run DOS assemblers like TASM as part of a GNU/Linux 
Makefile.

  You would need a tokenizer for each BASIC variant you want to support 
(M100/102, 8201A/8300, and maybe t200?)

  The ROM of each machine is of course already capable of doing it properly 
by definition via LOAD/CLOAD. So emulators effectively support tokenization as 
well. VirtualT is fully scriptable over telnet/socket.

  For myself I just use my laptop or emulator to created tokenized files.

  One note... there are valid, specially constructed BA  files that do not 
fully LIST, and are not convertible to ASCII BASIC text and back again, with 
high numbered lines and/or machine language embedded in it. To create such 
files goes beyond simple tokenization, but it may be useful and interesting to 
create advanced special BA binary file generators. Most such files require 
special measures to create.


  -- John.


  -- 

  -- 
  Brad Grier

Re: [M100] Questions about tokenizing BASIC in UNIX

[Brad Grier]

Club100 Member directory for Mike Stein has Entoke.exe -
http://www.club100.org/memfiles/index.php?=0=nom=Mike%20Stein

And a wayback machine crawl reveals detoke.exe -
https://web.archive.org/web/2017155020/http://www.xibalba.com/website100/software.html


On Mon, Jul 25, 2022 at 11:58 AM Peter Vollan  wrote:

> There are tokenise and detokenise programs for DOS, in the Compuserve
> archive, I think.
>
> On Sun, 24 Jul 2022 at 23:42, John R. Hogerhuis  wrote:
>
>> Not as far as I know. I think Robert Pigford's program is it. PowerBASIC
>> is a compiler though so I guess its binaries would run under a Linux DOS
>> emulator. I've certainly run DOS assemblers like TASM as part of a
>> GNU/Linux Makefile.
>>
>> You would need a tokenizer for each BASIC variant you want to support
>> (M100/102, 8201A/8300, and maybe t200?)
>>
>> The ROM of each machine is of course already capable of doing it properly
>> by definition via LOAD/CLOAD. So emulators effectively support tokenization
>> as well. VirtualT is fully scriptable over telnet/socket.
>>
>> For myself I just use my laptop or emulator to created tokenized files.
>>
>> One note... there are valid, specially constructed BA  files that do not
>> fully LIST, and are not convertible to ASCII BASIC text and back again,
>> with high numbered lines and/or machine language embedded in it. To create
>> such files goes beyond simple tokenization, but it may be useful and
>> interesting to create advanced special BA binary file generators. Most such
>> files require special measures to create.
>>
>> -- John.
>>
>

-- 
-- 
Brad Grier

Re: [M100] Questions about tokenizing BASIC in UNIX

[Peter Vollan]

There are tokenise and detokenise programs for DOS, in the Compuserve
archive, I think.

On Sun, 24 Jul 2022 at 23:42, John R. Hogerhuis  wrote:

> Not as far as I know. I think Robert Pigford's program is it. PowerBASIC
> is a compiler though so I guess its binaries would run under a Linux DOS
> emulator. I've certainly run DOS assemblers like TASM as part of a
> GNU/Linux Makefile.
>
> You would need a tokenizer for each BASIC variant you want to support
> (M100/102, 8201A/8300, and maybe t200?)
>
> The ROM of each machine is of course already capable of doing it properly
> by definition via LOAD/CLOAD. So emulators effectively support tokenization
> as well. VirtualT is fully scriptable over telnet/socket.
>
> For myself I just use my laptop or emulator to created tokenized files.
>
> One note... there are valid, specially constructed BA  files that do not
> fully LIST, and are not convertible to ASCII BASIC text and back again,
> with high numbered lines and/or machine language embedded in it. To create
> such files goes beyond simple tokenization, but it may be useful and
> interesting to create advanced special BA binary file generators. Most such
> files require special measures to create.
>
> -- John.
>

Re: [M100] Questions about tokenizing BASIC in UNIX

[Ken Pettit]

M100/M102 and T200 all use the same token format, but 8201A/8300 is 
different from the other 3 variants.  You could try using M100 VirtualT 
with 32K to tokenize and save the .BA file, then load it into T200.


Ken

On 7/24/22 11:42 PM, John R. Hogerhuis wrote:
Not as far as I know. I think Robert Pigford's program is it. 
PowerBASIC is a compiler though so I guess its binaries would run 
under a Linux DOS emulator. I've certainly run DOS assemblers like 
TASM as part of a GNU/Linux Makefile.


You would need a tokenizer for each BASIC variant you want to support 
(M100/102, 8201A/8300, and maybe t200?)


The ROM of each machine is of course already capable of doing it 
properly by definition via LOAD/CLOAD. So emulators effectively 
support tokenization as well. VirtualT is fully scriptable over 
telnet/socket.


For myself I just use my laptop or emulator to created tokenized files.

One note... there are valid, specially constructed BA  files that do 
not fully LIST, and are not convertible to ASCII BASIC text and back 
again, with high numbered lines and/or machine language embedded in 
it. To create such files goes beyond simple tokenization, but it may 
be useful and interesting to create advanced special BA binary file 
generators. Most such files require special measures to create.


-- John.

Re: [M100] Questions about tokenizing BASIC in UNIX

[John R. Hogerhuis]

Not as far as I know. I think Robert Pigford's program is it. PowerBASIC is
a compiler though so I guess its binaries would run under a Linux DOS
emulator. I've certainly run DOS assemblers like TASM as part of a
GNU/Linux Makefile.

You would need a tokenizer for each BASIC variant you want to support
(M100/102, 8201A/8300, and maybe t200?)

The ROM of each machine is of course already capable of doing it properly
by definition via LOAD/CLOAD. So emulators effectively support tokenization
as well. VirtualT is fully scriptable over telnet/socket.

For myself I just use my laptop or emulator to created tokenized files.

One note... there are valid, specially constructed BA  files that do not
fully LIST, and are not convertible to ASCII BASIC text and back again,
with high numbered lines and/or machine language embedded in it. To create
such files goes beyond simple tokenization, but it may be useful and
interesting to create advanced special BA binary file generators. Most such
files require special measures to create.

-- John.

[M100] Questions about tokenizing BASIC in UNIX

[B 9]

Hi! I keep running out of space when downloading an ASCII BASIC program and
then using *LOAD*, *SAVE* to tokenize it on the Tandy 200. I'd like to be
able to tokenize on my UNIX host system before downloading it to my T200.

What I've found so far:

   - Robert Pigford wrote a Model 100 Tokenizer that runs in PowerBasic
   

   but requires Microsoft Windows, which I do not have.
   - Almost 20 years ago, John Hogerhuis was talking about making a Linux
   program that would handle detokenization
   

   when talking to a PDD. He even released a Forth program to do it , but I
   don't see anything that handles tokenization.
   - The tokenization code in the Tandy 100 ROM disassembly
   

   seems reasonably short.

Questions remaining:

   - Has someone already written a BASIC tokenizer that runs on UNIX
   systems?
   - Is the tokenized BASIC for the Model 100/102 and Tandy 200 identical?
   What about for variants like the NEC PC-8201 and 8300?

I'll probably just write my own BASIC tokenizer since it seems simple
enough. To facilitate that, I've been gathering the information I've
learned on Archive Team's wiki
:.
(Please let me know if you see any errors).

—b9