What's the real-world performance of a '486 or Pentium PC running NEXTSTEP? Which components and subsystems work best? What's the price range for a fully rigged developer system? What is the level of component integration? What kind of support do PC vendors provide for NEXTSTEP users?
These are the questions that NEXTSTEP customers are asking as they decide which PC model to adopt as a development platform. To get the answers, NeXTWORLD put out the call for evaluation units of the 17 developer systems listed in the June release of NeXT's hardware-compatibility guide, as well as to several other manufacturers who are not yet listed but are targeting the NEXTSTEP market.
The first thing we learned is that few of the advertised NEXTSTEP PC suppliers are fully up to speed in servicing the market. While 14 PC suppliers responded to our request for test systems (Compaq provided us with both a '486- and Pentium-based computer, for a total of 15 systems), most were still uncertain about the level of their support for NEXTSTEP, final configurations, channels of distribution, and, in some cases, pricing. Therefore, we are not yet able to provide finished evaluations of each product, complete with cube ratings.
Nevertheless, in putting these systems through their paces, we were able to reach important general conclusions that should be helpful to the first wave of developers faced with choosing systems.
This article includes our benchmark results, which readers can use to form their own opinions. Beginning with the next issue, we will provide rated reviews of individual machines that are ready for prime time. To be included in this review, systems had to have 32MB of RAM (those with more were limited to 32MB for the benchmarks) and a 340-MB or larger hard disk. Once the systems were in-house, we installed NEXTSTEP 3.1 and NEXTSTEP Developer on each Ð not always an easy process Ð and then ran our benchmarks (see the sidebar, "Installation Blues"). Note that we include benchmark results for only 12 systems. The systems from Advance 2000 and Digital Equipment, as well as the '486 box from Compaq, could not be tested reliably, for the reasons explained in the sidebar.
While all these factors are important, the top priority for most developers is performance. Even with today's top-of-the-line systems, developers spend a lot of time waiting. Speed up the system, and the developer be-comes more productive.
The performance of a NEXTSTEP machine is determined by a variety of factors, including the CPU type, graphics subsystem, hard-disk drive, disk controller, and expansion-bus design. Our tests look at the variability in each of these factors.
We did not test Ethernet performance, since it depends almost en-tirely on the computer's bus design and isn't critical to most NEXTSTEP developers, whose entire development systems are normally resident on the computer's hard disk. We also didn't look at support for sound or serial ports, since both are limited under NEXTSTEP 3.1.
Here's what we found:
CPU: With two exceptions, all of the machines we reviewed (see the chart "NeXTWORLD benchmarks") use Intel's i486 DX2/66 microprocessor, which runs at 66MHz and has an external-bus interface of 33MHz. Not surprisingly, the CPU performance among those PCs with DX2/66 processors varied by less than ten percent. This variance can be attributed to secondary RAM caches, which ranged from 0KB to 256KB, and to motherboard design.
The Compaq Deskpro 5/60M contains Intel's new Pentium processor. Running under NEXTSTEP, the Pentium chip made the Compaq about 50 percent faster than the '486-based systems we tested; however, other limitations prevented its real-world performance from improving correspondingly. Dell's 450DE/2 DGX, on the other hand, was equipped with a DX2/ 50, the slowest CPU of those tested.
We are intrigued by the performance claims for Advance 2000's 80MHz system, which pushes the external clock speed on a standard DX2/66 chip from 33MHz to 40MHz. Unfortunately, an installation problem prevented us from fully testing this prototype system.
As expected, all of the systems were faster on raw CPU performance than a NeXTstation Turbo Color, and the news on that score will only get better. By year-end, Intel is expected to release its clock-tripling chip, the DX3, which will sport an external speed of 33MHz and an internal speed of 99MHz. The DX3 will offer a noticeable performance improvement over other '486 chips but will not be as fast as the Pentium. Sometime in the first half of next year, Intel will introduce a '486/Pentium hybrid, the P24T, and Pentiums running at 100MHz. The P24T is designed to fit into a socket that exists on the motherboard of most of the machines we tested and will offer near-Pentium performance. For now, most NEXTSTEP developers will likely be satisfied with the compilation performance of the DX2/66.
Bus: Having a fast bus in your system speeds communication between the CPU and any cards that are plugged into the bus (see the sidebar, "Take the Bus"). For example, if your video card plugs into the bus (as is the case with most of the systems we tested), having a fast bus will speed video performance. A high-speed disk interface plugged into a fast bus will likewise speed compiling.
The importance of the bus diminishes with integrated systems, such as the NEC and Intel GX, which put the video and disk interface directly onto the CPU board.
Graphics: Having a fast graphics subsystem makes windows drag and scroll perceptibly faster. While fast graphics won't speed your compiles, they do make the system more enjoyable to use. Since NEXTSTEP treats the graphics subsystem as a dumb frame buffer, graphics performance depends largely on how fast the computer's CPU can transfer data to, and within, the system's video memory.
We found that local-bus video systems offered the best graphics performance. The zippiest graphics were to be found on the Dell DGX (thanks to Dell's JAWS video subsystem), followed by systems from eCesys, Epson, and Lucky-Goldstar that are equipped with Chips & Technologies' Wingine video chip. PCs using the ATI Mach-32 chip (AST, Continental, Data General, NCR, and Intel) were consistently slower, because a bug in the ATI chip causes wait states when writing directly to the frame buffer. The next revision of the ATI chip should be significantly faster.
All three Wingine-based PCs were jittery: Vertical lines oscillated left and right for no apparent reason. This sideways jitter made the systems somewhat annoying to use, despite the Wingine's faster graphics performance. We were unable to test NeXT's new driver for newer Epson Progressions (capable of displaying 16-bit color at 1120 by 832 pixels) that reportedly will eliminate this motion.
On the other hand, Compaq's QVision board provided the most stable video image of the machines tested. It was also the only PC that NEXT-STEP 3.1 supported at the 1280-by-1024 resolution (in 8-bit grayscale mode only). Unfortunately, the QVision's EISA-implementation helped make it the slowest performer of the bunch.
We were able to run the NEC Express/II in grayscale mode only, because NEXTSTEP 3.1 does not drive the computer's ET 4000 graphics subsystem in color.
With the notable exception of Dell's JAWS technology, these systems provided slower graphics performance at lower resolution than the comparison NeXTstation Turbo Color. We look forward to running NEXT-STEP in 16-bit-color mode at 1280-by-1024 resolution on some of the latest high-performance video adapters. Drivers for these adapters should be available later this year.
Disk: Since NEXTSTEP uses the hard disk for virtual-memory swap space, having a fast hard-disk drive and disk interface will do more than make your programs load faster; it will speed up the performance of your entire system.
The fastest SCSI interface cards we tested were DPT's 2012 and 2022 and BusLogic's BT747 EISA cards. Unfortunately, DPT's cards appear to have compatibility problems with certain CD-ROM drives.
DPT's SCSI adapter is unique among the SCSI cards we tested in al-lowing the user to add from 0.5MB to 16.5MB of additional RAM, which is used exclusively as a disk cache. We tested the DPT in the Data General, both with and without an additional 4.5MB of cache memory. Not surprisingly, we found that adding the cache RAM dramatically improved the overall performance as measured by the Compile benchmark. (Due to cache overhead, raw disk performance actually decreased with the additional RAM.)
Many of the PCs equipped with Adaptec's ISA-based 154X-series cards achieved different disk-transfer rates, even with the same SCSI hard disks. The explanation lies with a jumper switch on the Adaptec interface board that sets the board's DMA (direct memory access) transfer speed; in testing these PCs, some manufacturers set the speed to the maximum rate at which their system can reliably operate, giving it higher performance.
We also compared the performance of the Adaptec 1540C in the Advanced Logic Research machine with its synchronous SCSI turned both on and off. We found synchronous reads to be nearly 20 percent slower than asynchronous reads; overall performance of our disk-intensive Webster benchmark decreased by five percent in the synchronous mode.
The fastest disk performance we found was from the NEC Express/II, which was equipped with the integrated DPT 2022 SCSI controller.
None of the systems tested included local-bus SCSI controllers, and none are supported under NEXTSTEP 3.1. When drivers become available for local-bus SCSI adapters, we expect them to be at least as fast as the EISA controllers. NEXTSTEP 3.1 also doesn't include a driver for the Adaptec 6260 SCSI interface card (provided by Compaq and integrated on the motherboard of the Intel GX) or its successor, the 6360. A 6260/6360 driver is being developed.
Lastly, we found some PCs to suffer SCSI errors due to long, circuitously routed ribbon cables of poor quality. The biggest offenders in this category were Compaq and Dell.
If you're used to the sleek styling and solid construction of NeXT's black hardware, you won't like what the Intel world has to offer. We found the PC cases to be poorly designed; they were difficult to open and close and were held together by too many screws.
The machines the with best construction were Data General's '486 DX2/66 LE2 and Intel's GX. Both computers were put together with precision and obvious attention to workmanship. Data General's was the only computer we tested that could be opened without special tools. It had no screws; like a Chinese puzzle, though, it took us nearly half an hour to figure out how to take it apart.
None of the systems were equipped with floppy-disk drives that could notify the operating system when a floppy-disk was inserted or eject a floppy under program control.
For many buyers, factors like the preloading of NEXTSTEP, the support staff's level of knowledge, and the length of the warranty will be as important as either price or performance. We did not attempt to judge these factors for this article, but they will be a part of our rating criteria in future issues.
Our two real-world benchmarks, Webster and Compile, confirm that most of the machines are faster than a NeXTstation Turbo Color for typical tasks. Leaving aside the Pentium, the fastest compile times were achieved by the Continental, NEC, NCR, and Dell systems. The machines fastest at running the Webster benchmark were the Data General, Compaq, and NEC models.
Being fans of black hardware, we were attracted to Intel's GX, a tightly integrated system that looks remarkably like a white NeXTstation. The GX features an integrated SCSI and ATI video interface on its system board, as well as built-in 16-bit sound (sound and SCSI are not yet supported). Our chief complaint was that the GX's integrated SCSI interface lacks a connector for an internal drive. The NEC Express/II also integrates the SCSI and video controllers on the motherboard. Integrated designs are easier to configure and may cost less, but they could limit future expandability.
While Dell's 450DE/2 DGX offers the best graphics performance, its high price and disappointing CPU performance might cause developers to shy away from purchasing the system. We hope to see a Pentium PC with JAWS graphics. On the other hand, Epson's Progression might make a great developer's workstation Ð but its ISA bus results in slow disk access.
Considering the hype that has accompanied the introduction of Intel's Pentium chip, we were surprised by the poor performance of Compaq's Pentium machine. While this computer's CPU had the fastest MIPS rating, its IDE disk subsystem slowed it down when compiling. Its video performance was likewise compromised, thanks to Compaq's QVision board. It shows that having a fifth-generation Intel processor isn't enough to guarantee a killer system: All aspects of the computer must be fine-tuned to achieve high performance.
A few of these machines have not been approved by NeXT to run NEXTSTEP. Unauthorized systems will probably work, as long as NEXT-STEP drivers are available for the SCSI adapter and the video interface is installed. There are, however, hundreds of '486 motherboards on the market, and reports from the field indicate that some of them are not compatible. Buyer beware.
So how does Intel hardware stack up against the Motorola standard? As a group, today's NEXTSTEP-compatible PCs are slightly faster and considerably cheaper than NeXT's late hardware. On the other hand, they are not as well-designed or as easy-to-use. There is, as yet, no one system that is ideally suited for NEXTSTEP development. If you need white hardware for development projects today, we recommend purchasing a system with both EISA and VESA local buses: Such systems offer the greatest promise for upgrading during the coming years.