Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
Hi Tom, > JUMPERS! anybody remembers JUMPERS (and associated problems) ? Yes, I remember jumpers, but I don't remember any problems. (Resource conflicts are caused by humans, but not jumpers.) Robert -- +++ BTTR Software +++ Home page: https://www.bttr-software.de/ DOS ain't dead: https://www.bttr-software.de/forum/ ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
>>> you can design APIs all day long, but if nobody is using these APIs >>> it's a bit pointless. and the BIOS certainly will not suddenly call >>> any DMA reservation API. >> These APIs will be used by anybody who needs to virtualize things >> like IRQs, I/O ports, and DMA channels. > after more then 18 years of virtualization we somehow learn right now that > we need a 'virtualization API'. sometimes Tom writes nonsense. he was speaking about VirtualPC and the like. for the intended purpose - presenting a virtual sound card, serial port, paralell port etc. this might make sense. however only a sound card would use DMA. > And hard drives and Ethernet cards and anything else that needs to transfer a > lot of data at once. none of them use ISA DMA which is the one you would have to synchronize. Tom ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
>> you can design APIs all day long, but if nobody is using these APIs >> it's a bit pointless. and the BIOS certainly will not suddenly call >> any DMA reservation API. > These APIs will be used by anybody who needs to virtualize things > like IRQs, I/O ports, and DMA channels. after more then 18 years of virtualization we somehow learn right now that we need a 'virtualization API'. congratulation to make this new detection. Tom ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
> you can design APIs all day long, but if nobody is using these APIs > it's a bit pointless. and the BIOS certainly will not suddenly call > any DMA reservation API. These APIs will be used by anybody who needs to virtualize things like IRQs, I/O ports, and DMA channels. > besides this, MSDOS and associated programs got along for a couple > of dekades without such an API. That's because for decades there was no need for virtualization and DMA and IRQ and I/O were always handled by real hardware. Now we're dealing with virtualization everywhere we turn, and you need to build something that will simulate what the hardware does in every detail. You're looking at it from the application side instead of the hardware/BIOS side. The two sides need to coordinate but they have very different perspectives. > Conflicts on DMA channels can often cause very strange system > problems and at times can be very difficult to diagnose. Which is exactly why the emulation/virtualization needs to be done properly. > DMA channels are used most commonly by floppy and tape drives and > sound cards. And hard drives and Ethernet cards and anything else that needs to transfer a lot of data at once. > so it looks like 'it would have been nice to have a reservation API > in the 80's', but they somehow navigated around these problems. 40 > years ago. Nope. As discussed above, 40 years ago virtualization didn't exist so there was no need for an API. ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
Hallo Herr tom ehlert, am Dienstag, 10. August 2021 um 11:04 schrieben Sie: >>> however if several applications want to access the same DMA controller >>> at the sam time things get complicated. there is no communication >>> between these applications. >> Exactly. That's what the API would somehow need to manage. > you can design APIs all day long, but if nobody is using these APIs > it's a bit pointless. and the BIOS certainly will not suddenly call any > DMA reservation API. > besides this, MSDOS and associated programs got along for a couple of dekades > without such an API. > that said (and I have to admit that I never did any motherboard DMA > programming), I found > http://www.dewassoc.com/support/bios/errors/dma_controllers.htm > 'Direct Memory Access (DMA) Channels > Direct memory access (DMA) channels are basically what the name implies, > the system pathway used by devices to transfer information directly to > and from memory. DMA channel problems are not nearly as frequent as IRQ > issues, > however they do occur. This is probably due to the fact that there are fewer > of them and they are used by far fewer devices. Hence they usually cause > fewer > problems with system setup. > ***Conflicts on DMA channels can often cause very > strange system problems and at times can be very difficult to > diagnose. *** > DMA channels are used most commonly by floppy and tape drives and sound > cards. > so it looks like 'it would have been nice to have a reservation API in > the 80's', but they somehow navigated around these problems. 40 years > ago. > Tom I probably found the reservation API. http://www.dewassoc.com/support/bios/errors/dma_controllers.htm DMA Request (DRQ) and DMA Acknowledgment (DACK) Each DMA channel is comprised of two signals: the DMA request signal (DRQ) and the DMA acknowledgment signal (DACK). Some peripheral cards have separate jumpers for these instead of a single DMA channel jumper. If this is the case, make sure that the DRQ and DACK are set to the same number, otherwise the device won't work! JUMPERS! anybody remembers JUMPERS (and associated problems) ? Tom ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
>> however if several applications want to access the same DMA controller >> at the sam time things get complicated. there is no communication >> between these applications. > Exactly. That's what the API would somehow need to manage. you can design APIs all day long, but if nobody is using these APIs it's a bit pointless. and the BIOS certainly will not suddenly call any DMA reservation API. besides this, MSDOS and associated programs got along for a couple of dekades without such an API. that said (and I have to admit that I never did any motherboard DMA programming), I found http://www.dewassoc.com/support/bios/errors/dma_controllers.htm 'Direct Memory Access (DMA) Channels Direct memory access (DMA) channels are basically what the name implies, the system pathway used by devices to transfer information directly to and from memory. DMA channel problems are not nearly as frequent as IRQ issues, however they do occur. This is probably due to the fact that there are fewer of them and they are used by far fewer devices. Hence they usually cause fewer problems with system setup. ***Conflicts on DMA channels can often cause very strange system problems and at times can be very difficult to diagnose. *** DMA channels are used most commonly by floppy and tape drives and sound cards. so it looks like 'it would have been nice to have a reservation API in the 80's', but they somehow navigated around these problems. 40 years ago. Tom ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
On 8/9/2021 1:06 PM, tom ehlert wrote: insofar DMA virtualisation is just another device to be understood and emulated, and not something entirely different. there is nothing special about DMA. however if several applications want to access the same DMA controller at the sam time things get complicated. there is no communication between these applications. Considering that by and large, (Free)DOS is a single tasking OS, this isn't such a big issue... ;-) Ralf -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
Hi Tom, > details of course matter, as there is a huge difference between > mainboard DMA (afaik mostly used for floppies) and PCI DMA (used by > everybody else). Mainboard DMA also sounds good for ISA SoundBlaster, I guess? > for the former onboard DMA, I'm not sure there was ever some > communication among DMA users. it was just 'set up DMA. use it. > forget it.' this should be emulatable as well. In a way yes, because you had to configure drivers to avoid DMA channel conflicts, or use ISA PnP for that, but then the 8237 DMA has some port overlaps: R- DMA channel 0 current address byte 0, then byte 1 -W DMA channel 0 base addressbyte 0, then byte 1 0001 RW DMA channel 0 word count byte 0, then byte 1 0002 R- DMA channel 1 current address byte 0, then byte 1 0002 -W DMA channel 1 base addressbyte 0, then byte 1 0003 RW DMA channel 1 word count byte 0, then byte 1 0004 R- DMA channel 2 current address byte 0, then byte 1 0004 -W DMA channel 2 base addressbyte 0, then byte 1 0005 RW DMA channel 2 word count byte 0, then byte 1 0006 R- DMA channel 3 current address byte 0, then byte 1 0006 -W DMA channel 3 base addressbyte 0, then byte 1 0007 RW DMA channel 3 word count byte 0, then byte 1 0008 R- DMA channel 0-3 status register (see #P0001) 0008 -W DMA channel 0-3 command register (see #P0002) 0009 -W DMA channel 0-3 write request register (see #P0003) 000A RW DMA channel 0-3 mask register (see #P0004) 000B -W DMA channel 0-3 mode register (see #P0005) 000C -W DMA channel 0-3 clear byte pointer flip-flop register any write clears LSB/MSB flip-flop of addr & counter reg 000D R- DMA channel 0-3 temporary register 000D -W DMA channel 0-3 master clear register any write causes reset of 8237 000E -W DMA channel 0-3 clear mask register any write clears masks for all channels 000F rW DMA channel 0-3 write mask register (see #P0006) As you can see in RBIL tables P0001 to P0005, some registers have the same port, but different data, for each channel. No problem if you have a central virtual 8237 controller. But if you want to write for example a virtual floppy with a built-in 8237 emulation for one channel and a virtual SB16 with a built-in 8237 for another channel, they will both have to virtualize port 8 to F. So it might be nice to have a central virtual 8237 with some sort of API. On the other hand, who needs a virtual floppy? Almost everybody only uses BIOS API to work with floppy drives. And if a virtual soundblaster uses several ISA DMA channels, it would still be for the same single SB16. Eric ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
> however if several applications want to access the same DMA controller > at the sam time things get complicated. there is no communication > between these applications. Exactly. That's what the API would somehow need to manage. I'm thinking a program would somehow need to register as the "owner" of the DMA channel (similar to how the API works now where a program registers as "owner" of an I/O port). But, there may some alternative way to handle it as well that doesn't involve a "registration" -- I'm not sure. In the end it's critical that there only be one "manager" of each resource or you'll end up with a big mess. ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
> insofar DMA virtualisation is just another device to be understood and > emulated, and not something entirely different. there is nothing > special about DMA. however if several applications want to access the same DMA controller at the sam time things get complicated. there is no communication between these applications. ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
> One thing that I think > still definitely needs to be added is DMA virtualization, though I > haven't yet worked through the details on how to do that. details of course matter, as there is a huge difference between mainboard DMA (afaik mostly used for floppies) and PCI DMA (used by everybody else). for the latter, that's straight forward. for your example of NE2000 (which you intend to emulate), it sets up the DMA to point to some memory, then makes some outport command 'please send these data'. sure you must also trap the DMA commands to the understand what memory is to be transmitted, but 'these data' are what is loaded into the DMA registers. for the former onboard DMA, I'm not sure there was ever some communication among DMA users. it was just 'set up DMA. use it. forget it.' this should be emulatable as well. insofar DMA virtualisation is just another device to be understood and emulated, and not something entirely different. there is nothing special about DMA. Tom ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
> RBIL sort of implies that all sizes should work, but dword is broken > and the byte/word distinction is only partially explained in RBIL? In my actual testing, the only thing that works properly is Bytes. > Have you seen apps which make use of the QPI way, apart from > SoftMPU? No, I haven't. But they could exist. I've ran across a dozen or so implementations of the MS API. Many of them are just "test" programs, though, and I never kept track of them. My USBJSTIK (for USB joysticks/gamepads) is a "real" implementation of the MS API. Is your implementation a superset of QPI? Or MS style? Both? Just MS. Qualitas provided a superset of the MS API also, and mine is a superset of the Qualitas superset. > How understandable is the API by reading source comments, as > no docs are there yet, so software beyond your USB drivers > can start to use the extra features? :-) As I think you are aware, I do a pretty good job of commenting my code so just reading the comments (especially the comments in the test program which tests all the features) will give you a good idea of what's going on. BTW, the test program is written for NASM. > I would suggest that you publish your code so people can look > around in it and ponder adding it to JEMM. Maybe even Japheth > himself would want to test it in the current state already :-) I really don't think that's a good idea, but maybe could be convinced otherwise. I don't think a "willy nilly" approach to this is a good idea -- it needs to be centralized so that there ends up only being one API. I know lots of folks don't like the idea of a "centralized" approach to something like this, but I think it's needed here. > Of course I would be happy to absorb some of your thoughts > and put them into some text file if you think that helps, > in comparison to you providing a readme yourself. I know you would, Eric, and I do appreciate everything you do. That's perhaps a little premature for now, though. I think I might be willing to publish the test program right away, though (without documentation). > To be honest, I am not aware of JLM (loadable protected mode > drivers) beyond the demo KEYB provided by Japheth. I'm not aware of any "real" JLMs either (other than test programs), but there may be some out there. > Also, is your goal to provide I/O trapping accessible by JLM, or > is there a general compatibility problem which breaks all JLM > as soon as the I/O trapping (your style, QPI or MS style?) My goal is simply be to not break existing JLM compatibility, though I don't see JLMs as the way to go in the future. That would particularly apply if Bob ever did publish the source code for QEMM since JLMs are strictly a JEMM thing. > Apart from the sound cards (although even those have VESA/AI) > you can probably virtualize all of those at the BIOS level or > packet driver level without too much compatibility loss with > common apps or drivers. The problem is that even where there are BIOS-level APIs for things (like keyboards, joysticks, mice, and even serial ports) the BIOS API's aren't always used (and in the case of joysticks and serial ports, are almost never used). To maintain compatibility with as many old programs as possible, I/O and, at least in some cases, DMA virtualization is required. > Of course, a BIOS would probably use SMM for port level > virtualization, which is not a style which user drivers like yours > can use. Exactly. That would need to be implemented by the BIOS (or EFI/UEFI) -- the can be only one SMM "manager". As you know, BIOS implementations of USB only address keyboards, mice, and/or disks. I've never seen a BIOS that supports USB COM ports, joysticks, sound cards, or Ethernet, though some Virtual Machines do. > But all of those would not yet be used by any other software ;-) The use of these is to set up a "virtualization layer" so other programs that don't understand the newer hardware can still work. E.g., a program can in fact be "talking to" a USB keyboard when it believes it is "talking to" a PS2 keyboard even when it is accessing the keyboard at the hardware (I/O port) level. > Thanks once again for your low level coding! Eric Thank You!!! ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
Hi Bret, > I've been working on adding an implementation of I/O port trapping to > JEMM using the EMM386/QEMM API (INT 2F.4A15). I have it basically working. Great news! > As far as the I/O port trapping, MS did a really crummy job of implementing > it. > Bob Smith of Qualitas found this out as he was implementing it in QEMM. > E.g., the MS implementation only supports byte I/O (not Word or DWord)... RBIL sort of implies that all sizes should work, but dword is broken and the byte/word distinction is only partially explained in RBIL? > The description in RBIL is incomplete and doesn't even detail the > de-installation > procedure. Qualitas' implementation fixed the MS problems and expanded its > features > (but I've never actually seen or tested the Qualitas implementation). Have you seen apps which make use of the QPI way, apart from SoftMPU? > I've added even more features in the JEMM implementation. Is your implementation a superset of QPI? Or MS style? Both? How understandable is the API by reading source comments, as no docs are there yet, so software beyond your USB drivers can start to use the extra features? :-) > If anybody wants to do it, I'll send you the code I've got so far and try to > explain my thinking. I would suggest that you publish your code so people can look around in it and ponder adding it to JEMM. Maybe even Japheth himself would want to test it in the current state already :-) Of course I would be happy to absorb some of your thoughts and put them into some text file if you think that helps, in comparison to you providing a readme yourself. > One of the main things missing is supporting JLMs (JLOAD needs > to be fixed to make it compatible). To be honest, I am not aware of JLM (loadable protected mode drivers) beyond the demo KEYB provided by Japheth. Also, is your goal to provide I/O trapping accessible by JLM, or is there a general compatibility problem which breaks all JLM as soon as the I/O trapping (your style, QPI or MS style?) would be invoked? I assume JEMM could block the respective other feature while either JLM or I/O trapping are in use, without bothering most users at all :-) > The main reason I started working on it was to support my > USB drivers, and in particular the ability to virtualize > the following kinds of USB devices: > > Keyboards > Mice > Joysticks/Game Pads > Serial/COM ports > Sound Cards > Ethernet Cards > Parallel ports (maybe -- these aren't as critical as Serial/COM ports) Apart from the sound cards (although even those have VESA/AI) you can probably virtualize all of those at the BIOS level or packet driver level without too much compatibility loss with common apps or drivers. Of course, a BIOS would probably use SMM for port level virtualization, which is not a style which user drivers like yours can use. Also, you already mentioned that packet drivers are no good abstraction level for USB LAN. https://en.wikipedia.org/wiki/System_Management_Mode#Problems > A couple of the things I've added in my JEMM implementation are > support for ports below 100h (though each port can only have a > single "manager" or "virtualizer") and IRQ virtualization (which > requires manipulation of ports 20h and 70h). One thing that I > think still definitely needs to be added is DMA virtualization But all of those would not yet be used by any other software ;-) Thanks once again for your low level coding! Eric ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
Re: [Freedos-devel] JEMM can't completely replace EMM386 (yet). Soluti ons?
I've been working on adding an implementation of I/O port trapping to JEMM using the EMM386/QEMM API (INT 2F.4A15). I have it basically working. I want to clear up some misconceptions, though. As far as the I/O port trapping, MS did a really crummy job of implementing it. Bob Smith of Qualitas found this out as he was implementing it in QEMM. E.g., the MS implementation only supports byte I/O (not Word or DWord), has a very limited number of port traps that it can implement, and doesn't always uninstall the I/O traps correctly. The description in RBIL is incomplete and doesn't even detail the de-installation procedure. Qualitas' implementation fixed the MS problems and expanded its features (but I've never actually seen or tested the Qualitas implementation). I've added even more features in the JEMM implementation. Even though I have it basically working, it's still far from where it needs to be and would really like someone else to take it and finish so I can concentrate on other things. If anybody wants to do it, I'll send you the code I've got so far and try to explain my thinking. One of the main things missing is supporting JLMs (JLOAD needs to be fixed to make it compatible). However, if this is going to be a "fork" of JEMM rather than an "upgrade", fixing JLOAD may not be necessary. The main reason I started working on it was to support my USB drivers, and in particular the ability to virtualize the following kinds of USB devices: Keyboards Mice Joysticks/Game Pads Serial/COM ports Sound Cards Ethernet Cards Parallel ports (maybe -- these aren't as critical as Serial/COM ports) E.g., for an Ethernet card, it's possible to implement a USB Ethernet drive as a simple packet driver, but virtualizing something like an NE2000 at the hardware level opens up many more possibilities for compatibility with older programs. A couple of the things I've added in my JEMM implementation are support for ports below 100h (though each port can only have a single "manager" or "virtualizer") and IRQ virtualization (which requires manipulation of ports 20h and 70h). One thing that I think still definitely needs to be added is DMA virtualization, though I haven't yet worked through the details on how to do that. And, of course, let's not forget the documentation which I haven't even started yet (and which I'm sure most implementers would just slip through the cracks). I've also created a test program to be able to test all the features to make sure they work. The version of JEMM I started with when I did this was 5.79, and I think Japheth has a newer version out and Japheth's latest changes would also need to be incorporated. I also went through and "cleaned up" some of the code to make it look more consistent. Anyway, does anybody else want to take this over and finish it? I know a lot of people want to see it happen (including me), and I think I've got a good start on it. ___ Freedos-devel mailing list Freedos-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/freedos-devel
