Linux-Advocacy Digest #459, Volume #28 Thu, 17 Aug 00 19:13:07 EDT
Contents:
Re: Windows stability: Alternate shells? (R.E.Ballard ( Rex Ballard ))
Open source: an idea whose time has come (phil hunt)
----------------------------------------------------------------------------
From: R.E.Ballard ( Rex Ballard ) <[EMAIL PROTECTED]>
Subject: Re: Windows stability: Alternate shells?
Date: Thu, 17 Aug 2000 22:23:32 GMT
In article <XDhl5.5147$[EMAIL PROTECTED]>,
"Erik Funkenbusch" <[EMAIL PROTECTED]> wrote:
> "R.E.Ballard ( Rex Ballard )" <[EMAIL PROTECTED]> wrote in message
> news:8n2uj7$r4$[EMAIL PROTECTED]...
> > AT&T decided to use UNIX to control it's switching systems. Since
> > they made this decision prior to divestiture, all of the Baby Bells
> > were dependent on UNIX too. What would happen if your company's
> > telephone system stopped working for an hour or two? How about
> > if an entire city came "unplugged"?
>
> You do realize that AT&T and the bells have always used redundancy.
> switches fail, but they've always had extensive cutovers and the
> ability to re-route around failures.
Yes. I was a developer on one of the first computerized Directory
Assistance systems to go nationwide. We had developed the original
using PDP-11s and a hacked up version of RT-11, with one of the
very first clustered systems. We ran 8 PDP-11/44 processors per
cluster, connected to up to 32 group controllers, each group
controller supported up to 24 8085/8088 processors and up 20 terminal
cluster controllers.
Each cluster controller served up to 20 terminals. Each data volume
was redundantly stored on 3 drives, more for better performance.
Keep in mind that a PDP-11/44 was about as fast and had about as
much memory as a PC/XT. With our additional support processors,
each SCU (pdp-11 with supporting 8008 group controller and 4004
drive controller (essentially a bridge between ESDI and something
like SCSI) we could get almost as much power as a PC/AT. These
systems had down-times of 15 minutes/year or 3 parts/million.
When we won the 2ADA bid, (just before devistature) we began moving
the whole thing to UNIX. Furthermore, we ended up deploying it
at British Telecomm. It was amazing. The entire UNIX development
took less time than a single upgrade revision (minor structure changes)
to the legacy system.
What made the job so much easier was that we didn't have to reinvent
the wheel to do simple things like queue jobs to the printer (lpr),
generate formatted reports (tbl | nroff), route messages (ip), connect
to remote computers (tcp), and create databases (awk, sed, grep). We
didn't even have to create relational databases (split, join). And
when we did have a huge table we didn't have to create an ISAM (dbm).
Keep in mind that this in 1983, about 7 years before the first SQL
databases appeared for UNIX.
> > The culture of UNIX was "It will fail, what will you do?".
> > As a result, code was much more carefully tested, reliable code
> > was packaged into self-contained units that could be linked together
> > without risk of breaking the components. UNIX developers also
> > came up with things like RAID, Clustering, and hot-standby or
> > active-redundant systems that could cutover in a matter of
> > milliseconds.
>
> RAID was developed as a means of producing faster,
> more inexpensive disk subsystems.
One of my supervisors, Anita Freeman, was on the SCSI draft
committee when the SASI standard was being formalized to the
ANSII SCSI standard. Almost from the very beginning, Anita
had the goal of supporting redundancy within SCSI. Anita had
worked on the projects decribed above. She knew that we needed
the ability to quickly update multiple drives concurrently.
The concept of RAID was implemented in ESDI and SASI long before
the formal buzzword/acronym evolved for the SCSI version. When
they first announced RAID-1 my CCI colegues chuckled and said
"Gee, I wonder where they got that idea from" - of course, they
were doing "RAID" back in 1978. Of course, back in those days,
it was RAED - Redundant Array of Expensive Drives.
> Redundancy was added because it made sense,
> but was not the primary motivation.
RAID-0 reduced the latency and fetch time for large files. RAID-1
was specifically designed for high performance, high availability
databases. RAID-5 became popular because you could get more storage
into fewer drives - and this was only practical when you kept the
drives pretty much synchronized (since you had to read all stripes
each time data was written to create the ECC drive.
> > Eventually, UNIX even found it's way into things like Air Traffic
> > Control, Military tactical systems, and even strategic systems like
> > Norad and SAC.
>
> Although these systems were primarily
> mainframe based until recent years.
Mainframe - as in OS/370?
Most of the tactical and strategic systems were implemented on
supercomputers using either ADA or a combination of ADA and UNIX.
I used to chuckle when ininformed managers would claim that TCP/IP
was unreliable and insecure. The didn't know that TCP/IP was designed
by the military (DOD) to survive a nuclear holocaust and be completely
jam-proof, with the addition of DES and spread routes, it was also
nearly uninterceptable as well. But managers assumed that because
it was "written by some college kid" (actually the BSD implementation
was a combination of ARPA and Bill Joy who was in college and later
went on to create Sun Microsystems.
> > Linux did have certain advantages over traditional UNIX. It was
> > modular, but not a "microkernel". This made it much easier to add
> > and debug driver software. Many hardware vendors even test their
> > Microsoft Windows drivers on Linux before switching from the Linux
> > wrapper to the Microsoft wrappers.
>
> Many? Name one.
I remember reading in some of the howtos that a few vendors were
implementing Linux drivers and simply changing the wrappers for
Microsoft. I believe this included 3Com, LinkSys, DLink, TI (TIGA),
S3, and Adaptec. I think this was mentioned in one of the howtos.
I'd read it about 3 years ago, and it correlated to an article in a
print publication (Info-World?) where vendors who were frustrated
with Microsoft for not supporting their Windows 3.51 and Windows 95
porting attempts, resorted to testing with Linux systems using
interchangable wrapper APIs. Many early winsock implementations
were based on the adapter software written by the same person who
wrote net3 and many of the packet drivers for interface cards.
> > Unfortunately, the very nature of the Windows APIs and OLE/COM
> > environment dictates that processes pass blocks of memory around
> > and to the event queues of thet main interpreter. If there is a
> > 1 microsecond window between when a process checks a variable and
> > when it attempts to modify that variable, and you do this 1 million
> > times a day, you will corrupt the system about once a day.
>
> What the hell are you talking about?
> OLE/COM uses memory copies between
> processes, not direct writes. This is done by LRPC on NT.
>
> What "main interpreter" are you talking about?
Sorry, event handler. Most Windows NT programs are based on passing
messages to an event handler which then calls the appropriate actions
and methods for the classes recieving the event.
Sorry for the confusion. I was unclear.
> There's no such thing.
>
> > A process would ask the kernel for service which would pass control
> > immediately back to the kernel. The result was that you never had
> > more than one system tinkering with shared memory at a time.
>
> I guess that's why mmap exists, right?
Both UNIX and Windows support shared memory. The difference is that
UNIX also provides a number of structures that reduce the need of
multiple applications to modify memory. Typically, when shared memory
is used in UNIX, a single service modifies the block and all others
read.
In windows, interprocess communication wasn't really formalized until
MSMQ. Prior to that, NT provided DCOM, but with a 95% loss of
performance (20 times slower) when switching from COM (in-process)
to DCOM (out of process), it was avoided like the plague.
As a result, a number of abstractions were used. A classic in Win3.1
was to create a DLL with a shared global. With Win95 you created
a VXD. With WinNT you created an OCX. Each allowed two programs
to pass messages between each other via shared memory - with all
the problems of race conditions, deadlocks, and gridlocks that
reduce uptime and decrease performance.
Again, Microsoft is aware of these problems in NT, and has made a
number of changes to Windows 2000 to eliminate them. In some cases,
they preserved the windows APIs while cleaning up the back-end. In
other cases, they introduced new APIs like COM+, MTS, and MSMQ.
Of course, these are proprietary interfaces, incompatible with any
other platform, which assures that you will have to do a major rewrite
to port to other platforms, depend on very expensive software for
Windows2000->UNIX migration, and because you are depending on 3rd
party software, you risk instability resulting from Service Packs that
may accidentally or deliberately render these 3rd party products
disfunctional.
> > Ironically, this makes the system faster, because you don't have
> > to arbitrate for resources as often. The cost is a few more context
> > switches, but Linux and UNIX have optimised the scheduler to reduce
> > the number of cache misses. The net result is that Linux can run
> > thousands of independent processes that
> > simply pass messages back and
> > forth via data-streams, more efficiently and more quickly than NT
> > can schedule it's apartment threads.
>
> Bullshit. pthreads and kernel threads would
> not exist if this were the case.
Posix threads are used by Linux and UNIX to provide multiple threads
within a single process. They are used very sparingly, typically with
only 2-3 threads (waiting for responses from multiple sources), and
most threads can be avoided by using message queues (SysV-IPC,
MQSeries, MOM, and MSMQ on Win2K). Keep in mind that I qualified
the above by specifying apartment threads, which have
the MMU protecting messages from each other. Posix Threads have
no such memory protection.
In UNIX, the equivalent of an apartment thread is the fork(),
which kicks off a new process, but the new "child" process
(actually only a process table entry) share all of it's
memory with the parent process.
In UNIX a much higher percentage of the executable and static memory
is shared by all processes. Futhermore, a larger percentage of
the buffer memory is also shared.
> > By the way, none of this was lost to Windows 2000. Microsoft's
> > "fabrics" very similar to the interaction between linux "fork"
> > processes (which clone the process but alter/remap only the buffer
> > space, which is minimal because streams are used). Unfortunately,
> > Microsoft still didn't understand the benefits of streams,
pipelines,
> > and processes that can be connected independently. As a result, the
> > Windows API still requires that applications be linked together into
> > huge monolithic applications that can't be reconfigured on-the-fly.
>
> Oh, give me a break. Like you can reconfigure
> "tin" or "elm" on the fly.
Again, UNIX systems are typically combinations of simple binaries
such as grep, compress, crypt, tar, sed, awk, and so on. These
binaries are combined using shell scripts which can be used to
connect multiple programs via pipelines, tees (to named pipes),
and via expect.
More complex programs, including those with graphical user interfaces,
typically use simple scripting languages like Python, TCL, or PERL
to combine these simple utilities - that have been tried and tested
in some of the harshest environments - and link them to a simple
graphical interface that leverages an equally hardened graphical
environment.
> Whatever "reconfigure" is supposed to mean here.
Don't like the security options, change the script.
Don't like the prompts, change the script.
Don't like the colors, write a script that changes the properties
(app-defaults). Don't like the user interface at all, use a
different script. Don't need the GUI for admin purposes, use
a script to put it all into a cron job so that it starts automatically
every night at 3:00 A.M.
> > Linux and UNIX provide tools that make dynamic configuration
> > trivial.
> > The shell, which seems like a primitive command line interface, also
> > provides a very simple way to interconnect a series of processes
> > that can be very quickly be reconfigured for different needs.
>
> You mean pipes?
Pipes, message queues, RPC, unix domain sockets, signals, and
"broadcast" shared memory (one writer, multiple readers).
Ironically, what was originally considered to be a shortcoming of
UNIX (lack of system level file record locking) eventually became
the driver of it's greatest strength (pass messages to servers via
a kernel).
> MS has had pipes for years.
Yes they did. From 1982 to 1995, you could pipe output from one
command into another command. Of course the implementation left
something to be desired. You ran the first program and stored
all of the output into a huge hidden file, when the first program
completed, the second program would read the hidden file. When
the receiver program was finished, it erased the file. Disk space
was critical.
Windows 95 improved things a bit, using memory between the processes.
Eventually, NT did it the "UNIX way" - filling a very small block of
memory, passing the small block to the receiver, and the receiver
would pass control back to the sender when the block was drained.
Actually, the "UNIX way" is to have two blocks of memory. This way,
the sender can fill blocks WHILE the receiver is draining them.
Unfortunately, unless you have the NT resource kit, most Windows
programs are still written to a paradigm based on huge monolithic
objects that must be read into memory in their entirity before the
methods of the object can be invoked to modify the object. If you
have a large object such as a BMP file, or a Word document, this
can involve megabytes between the processes.
> > Finally, Microsoft depends on proprietary file formats that can't
> > be parsed by either stream parsers or by human beings. This is
> > because Microsoft wants it's content to be managed as objects
> > created exclusively for and by Microsoft Applets rather than as
> > information created by and used by the end-user.
>
> Anything can be parsed.
Yes, but if you must know the length of the entire object before
you can send the first bytes of the header, which contain the
size of the rest of the entire object, parsing takes quite a bit
more memory. Furthermore, if you have to allocate memory for the
entire object and read the contents of the entire object into memory
before you can execute methods against the object, you have additional
latency as well as memory.
By using streams of smaller objects, such as words, lines, paragraphs,
and similarly delimited tokens, whether the objects are sent via
XML or unix strings, you can parse smaller tokens using very little
memory and pass information to the next stage of the chain while
you work on the chunk coming from the previous stage of the chain.
If you have enough processors, you can even put the machine into
a trivial multiproccessing mode that speeds the chain considerably.
> If text based file formats are so cool, why does
> the Red Hat packaging system use binary fiiles for it's file format?
> Check out the files in /var/lib/rpm.
Look at the details more closely. They use a combination of compressed
tar files, along with a series of scripts and dependency databases
(similar to make-files) that help the package manager identify which
dependencies have already been satisfied, which ones need to be
satisfied, and how to satisfy them.
Sure, some of the files are actually compressed binary executables.
More building blocks. Most of the packages however are essentially
a combination of simple components combined with some PYTHON, PERL,
or TCL scripts along with some BASH scripts.
> Fact is, text formats are orders of
> magnitude slower to process than binary formats are in most cases.
Very true. As Bill would say, "CPU Cycles are cheap". The fact is
that most UNIX programs convert extern text representations into
internal binary representations, then back into text representations.
A good example of this is the X11 resource database. When an X11
application starts, it check a resource database that is very similar
to the NT registry. This database internally uses a binary format
based on dbm or other ISAM. When the application is started, the file
system is checked for an app-defaults file with a related name. If the
file exists, and is newer than the app-defaults rdb entry for that file,
the app-defaults file is parsed, indexed and stored. Then, the 4
million iterations over the life-cycle of that object will be handled
by the "compiled" representation.
The flip side is that the same resource database can be used to save
the binary representation into a text form. Finally, you can load
and/or store you own preferences in you own .Xdefaults folder.
Microsoft has the capability to support this. You can export the
registry, even a branch at a time, and have the registry loaded
and saved incrementally. This would be very nice to have in the
event that a rebuild is needed, but most applications make no provision
for either loading or storing these segments. Some back-up programs
like veritas have the capability to store and load the segments required
to recover a failed system without messing up the configuration too
badly (very important if you are restoring to a replacement/upgrade
machine. Unfortunately, almost none of the values are documented,
even fewer are understood, and it's very easy to create a serious
mess. Even Microsoft warns against making modifications to the
registry using REGEDIT.
> > Linux and UNIX decided to document the formats used to pass
> > information between processes. Furthermore, they decided to
> > pass even GUI object messages as datastreams known as X-wire.
> > The result is that all messages can be passed between processes
> > in the same machine, or between processes in different countries,
> > with about the same level of ease.
>
> Linux developers did nothing of the sort.
> X was developed long before Linux was even a thought.
I said Linux *AND* UNIX. UNIX developers created X11, followed the
documentation. They even formalized it in the ICCCM. Linux
developers added more tools and features, but documented both the
text-stream and the gui-stream protocols in a manner similar to UNIX.
Again, this is largely a function of the UNIX culture, which Linux
inherited (along with about 95% of the code).
--
Rex Ballard - I/T Architect, MIS Director
Linux Advocate, Internet Pioneer
http://www.open4success.com
Linux - 42 million satisfied users worldwide
and growing at over 5%/month! (recalibrated 8/2/00)
Sent via Deja.com http://www.deja.com/
Before you buy.
------------------------------
From: [EMAIL PROTECTED] (phil hunt)
Crossposted-To: gnu.misc.discuss,comp.os.ms-windows.advocacy
Subject: Open source: an idea whose time has come
Date: Thu, 17 Aug 2000 23:05:38 +0100
Some recent events:
(1) Sun's decision to open source StarOffice.
(2) Formation of the Gnome Foundation, to develop the Gnome GUI, backed
by a consortium of major industry players (including IBM, Sun, Compaq,
HP, Red Hat, the FSF, VA Linux Systems).
(3) Microsoft's recent decision to force major corporate users to pay
twice for Windows 2000 if they want to have a common corporate setup.
What do these events have in common? They all indicate that Open Source
software is an example of the most powerful thing in the world: an idea
whose time has time.
Let's take these events one by one:
(1) Open-Sourcing StarOffice
StarOffice was previously available on a "free beer" licence. Now it is
being open sourced. This will make it future-proof: the software will
never go away, and will always get upgraded to work with new environments,
so long as a (small) minimal number of users are still using it.
(2) Gnome foundation
Why would major hardware companies set up the Gnome foundation? So they
can ship the Gnome desktop environment on computers, especially Unix
computers, that they sell. Note that there have been prvious efforts
to produce a common GUI for Unix boxes: the Common Desktop Environment.
But Gnome has an advantage over CDE: it is open source, which means that
firstly, it has got a lot of developer effort behind it, without
attracting large amounts of corporate money, and secondly open source
means that no company can shaft another member of the consortium by
stopping them using proprietary enhancements. Goodbye competition,
hello co-opetition.
The Gnome desktop they will produce includes the Gnome base software,
StarOffice (now called Gnome Office), IBM's Sash development tool,
and Evolution, a competitor to MS Outlook.
A beta release of this system is likely to be out later this year, with
a fully integrated version next year.
(3) Microsoft re-imaging rip off
If you are a MS corporate customer, you have Windows 2000 on a PC, and
you re-install the OS, MS want you to pay them again for the software.
The charge is $117 to $157. This is, of course, a blatant rip-off only
possible because of MS's dominant monopoly position.
Now, say you're an IT manager sometime next year, MS are up to their
old tricks, you know they will continue to shaft you as long as you let
them. The Gnome desktop looks an attractive alternative, doesn't it? --
no licencing costs, no double licencing costs, no upgrade treadmill,
no deliberate incompatibilities, & no supplier grabbing you by the
balls.
Be scared Microsoft, be very scared.
--
*****[ Phil Hunt ]*****
** The RIAA want to ban Napster -- so boycott the music industry! **
** Don't buy CDs during August; see http://boycott-riaa.com/ **
** Spread the word: Put this message in your sig. **
------------------------------
** FOR YOUR REFERENCE **
The service address, to which questions about the list itself and requests
to be added to or deleted from it should be directed, is:
Internet: [EMAIL PROTECTED]
You can send mail to the entire list (and comp.os.linux.advocacy) via:
Internet: [EMAIL PROTECTED]
Linux may be obtained via one of these FTP sites:
ftp.funet.fi pub/Linux
tsx-11.mit.edu pub/linux
sunsite.unc.edu pub/Linux
End of Linux-Advocacy Digest
******************************
