Several years ago, I spent a while working on a project that involved converting a huge pile of Objective-C, NIB files and C++ from NeXTSTEP to Mac OS X.
One tool that proved to be critical to the process was Virtual PC v5.0. In particular, because the applications to be converted used a bunch of custom palettes and NIB files, we had to write some code on both NS 3.3 and OS 4.2 to do the interface conversion.
I wrote a bunch of installation notes. These are a few years old and VPC has changed since, but these should still mostly work.
Yes, both NS 3.3 and OS 4.2. While Mac OS X will sometimes open some of the NIB files from NS or OS, you really want to open and save each NIB file on a version of each major release of the AppKit since the original platform.
And because of certain idiosyncrasies of the NIB files, we had to write IB palettes on OS 4.2 to do some conversion there, too.
We also modified the NS -> OS conversion scripts to turn them into OpenStep 4.2 -> Mac OS X conversion scripts, mostly by adding a phase to turn Display PostScript calls into NSBezierPath invocations.
Fun stuff. Someone just asked about this on the cocoa-dev list, so I figured I would pull the notes forward from my old weblogs.
Update: Ovidiu asks if you can boot OpenStep/NeXTSTEP directly on an Intel Mac. Honestly, I have no idea. Lacking a bootable floppy drive is the initial hurdle, though.
However, there is a better way. You can run NeXTSTEP and OpenStep under Parallels under Mac OS X. There were some issues with the 1.0 release of Parallels, but I think they have been addressed in later releases.