There a re a couple of systems that used a 68000 in a Unix type environment. I have worked on both.The first is an Altos ACS68000 system that used a 68000 and 4 (four, count'em) 68451 mmu units. The way it did the memory allocation was to use "stack probes" to trip up a memory fault interrupt. These probes were simple 'TST' instructions that indicated where the stack would grow to. These instructions were easily "restarted" since they didn't change the register contents. Subsequent instructions would have no memory faults and everything would be working nicely. The Rasio Shack Model 16 used the same system, but had a cruder MMU that had two offset and two limit registers. The operating system was Xenix and worked OK, but the methods for memory management were not the best, but considering the hardware worked OK.
The C compiler (from what I understand) was a precursor to GCC and had the stack probes code built in. It was part of the activation record code. So, yes you can run a Unix style operating system on a 68000, you just need to be creative. I don't know how others did it. Yes, the 68010 (or later) was the solution to all the problems.
