Unit Testing.

Yeah, I can’t avoid this one.

Wil Shipley wrote a great article about his opinions of unit testing. Of course, the community response has been the fairly typical “He’s all wrong” / “He’s totally right” extremes.

Whatever. I would rather relay an anecdote.

As Duncan wrote, we used OCUnit to unit test Core Data. I integrated OCUnit into the Core Data project within a few weeks of the first lines of code being written. The experience was so positive that we decided to integrate OCUnit into Xcode (A huge note of thanks to the Sen:Te folks) and released the result at WWDC ’05 in Xcode 2.1.

Core Data 1.0 is not perfect, but it is a rock solid product that I’m damned proud of. The quality and performance achieved could not have been done without the use of unit testing. Furthermore, we were able to perform highly disruptive operations to the codebase very late in the development cycle. The end result was a vast increase in performance, a much cleaner code base, and rock solid release.

During the development cycle, we saw nearly zero regressions. And, of course, Core Data was developed on top of and along with Tiger. That is, we were frequently running in a highly unstable environment. Time and again, the Core Data unit testing caught problems that we are actually in components below Core Data.

Not only did the unit tests ensure the quality of our code, but it helped other teams to quickly find and fix bugs in other components.

Sure, writing the unit tests was occasionally tedious and maintaining the tests consumed a bunch of engineering time. I can say with complete and total confidence that the return on investment has been huge.

And the ROI continues to be high. When a developer files a bug against Core Data, the bug is generally turned into a unit test to verify that the bug really exists and the exact details. This greatly streamlines the whole bug fixing process and ensures that bugs stay fixed.

The team is also free to pursue some radical and disruptive changes to the framework for the next version with the confidence that it will remain completely backwards compatible.

Why such a different experience than Wil’s?

Software can generally be said to fall somewhere along a line between “infrastructure” and “end user application”. Core Data clearly falls at the “infrastructure” end of the scale while Wil’s application — the rather awesome Delicious Library — clearly falls at the other end of the scale.

And that clearly is the focus of Wil’s testing philosophy. Combined with his “the best software is the software with the fewest lines of clearly written code” attitude, his approach to testing is 100% entirely user driven.

That makes sense for his software.

I spoke with Wil at WWDC and he was very complimentary of the Core Data 1.0 release. I am thoroughly flattered by that as I have a huge amount of respect for Wil and that he was impressed by something I worked on makes me very happy indeed.

Given his “least code possible” philosophy, one could easily imagine that Delicious Library 2.0 or 3.0 will use Core Data as doing so would eliminate vast swaths of code he must have today. I hope he does exactly that at the same time he decides its OK to ship a Tiger+ only product.

And that is the answer. A software developer like Wil — a totally user focused, user experience driven, engineer — may not see the value of using unit testing directly in their development efforts. However, that Delicious Library could use something like Core Data and does use all of the other technologies in Tiger would not be possible without the huge investment in Unit Testing by the various teams at Apple.

Unit Testing is awesome for guaranteeing consistent behavior of infrastructure. UT can be very effectively applied to user interface level code, too, but it is a little harder to do.

But Unit Testing cannot test User Experience and, in the end, a quality UE is what turns a brilliantly engineered piece of software only appreciated by other engineers into a brilliantly designed products used by millions.



26 Responses to “Unit Testing.”

  1. Lance says:

    hi-

    Automated testing is always good. You are right about using it to revalidate previous tests for new revs. I groan about having to manually test things on each rev. and the “try not to break things” to coding is very hard :-).

    However, coredata does suffer a bit from the lack of unit testing for Oracle adapters :-). Or, to put it another way, it might be better to write a Oracle adapter than to write a unit test to test the Oracle adapter. Or am I missing something or is my info out of date? These are the type of issues I think Wil was thinking of: With limited resources a developer needs to focus on user solutions, with unlimited resources a developer can do whatever.

    Plus, with a big team I have found that one developer breaks another developer’s code, unbeknownst to either developer, especially with concurrent development systems. Some developers prefer ownership of big sections of code and lock out other developers from direct access so things don’t break as often and humans can be responsible for not coding wrong thus making unit testing less important and satisfying user needs more important. It seems to be a balancing act.

    thanks for coredata!-

    -lance

  2. Lee Harvey Osmond says:

    Hey, if Lance can say “Oracle adaptors”, I wanna say “Sybase adaptors”.

    I love the stuff Wil writes in his blog, both the lifestyle stuff (“is helicopter an indoor toy”), and the deliberately contentious code-related stuff like “teh suck”. Clearly Wil has devised mechanical aids for seeing the wood for the trees — maybe Delicious Library can handle bark codes.

    I’d guess Wil has foresworn unit testing for his stuff not because he thinks it isn’t useful, but because GUI stuff can be, like, really awkward when you’re trying to fit it to a test harness, and any proposed incorporation of unit tests did not survive the first cost/benefit review. Libraries and server stuff — webapps figure big here — are easier to instrument, so all the pass/fail pithy goodness is available at an affordable price in man months. Hey, the J2EE stuff I’m doing these days has tests of a sort, while the J2ME stuff doesn’t.

    Like so much, the real answer to “to unit test, or not to unit test” is “how long is a piece of string”. It’s fun to read the comments following Wil’s blog posts to see just how many people don’t get it.

  3. David Stewart Zink says:

    Bizarre responses. If you want an Oracle Dysdapter that does not actually work, sure have the “engineers” slam something together and ship it. Just make sure they attach a “Developed in the style of MicroSoft” label to it. If you actually need your Sybacle adapted, then you need an Orabase Adapter that actually works, and the only way to know if it actually works is to test its functionality. You _reduce_ development time, not increase it, if you write the tests first because then you find the bugs at the earliest possible moment and fix them before they are confused by other bugs. The tests need to be written eventually, or else you are just shipping garbage.

    As for personal projects, I skip the Unit Tests because my code always works the first time; but for group projects Unit Tests are indispensible.

  4. L'g. says:

    As for personal projects, I skip the Unit Tests because my code always works the first time;

    Yeah and our earth is a disc. Sometimes it’s really fun to read comments.

  5. Jeremy’s Little Corner Of The Web… » On Unit Testing… says:

    […] Looking at the comments, there’s a bunch of people that agree with Wil’s reasoning on face value (“Yeah! Unit testing sucks! It takes up so much time!”), but then you have others who disagree (to an extent). The most enlightening post I’ve seen on the topic is by Bill Bumgarner, who notes that unit testing was used by the team developing Core Data to make sure the code adhered to the specification (or expected behaviour), as well as preventing regressions when bugs were fixed. […]

  6. Rob Rix says:

    Excellent post, Bill, and it definitely highlights the value of unit testing in the approach that you guys took–CoreData is a success, and quite a lot of fun to work with. I’d love to be able to handle Oracle and Sybase and whatever other sorts of DB stores one can think of, certainly, but I consider that secondary to the fact that CoreData works well as is.

    And I’m saying that from the standpoint of someone who just spent two days writing a -copyWithZone: implementation, and thereby learning a lot more about the framework inside-out. It’s beautiful, and the fact that you can do stuff like a generic copy method that takes ownership into consideration is a big deal. (And yeah, I’m kinda plugging my own code there, but I just finished it, I’m kind of on a high here.)

    So yeah, I can verify that experience with unit testing, but from the opposite side– I’d have had a better time of my own code here had I unit tested. Cos infrastructure code tests easily.

    Great job, by the by. I can’t wait to drop Panther so I can use CoreData everywhere.

  7. Jeff Santini says:

    I don’t understand.

    You say Wil’s post is good and you have huge respect for him. You say you unit test and it is extremely valuable to your project. You say Wil may not value unit tests because of his type of software. But Wil simply says Unit Tests suck. He does not say they are good in some situations, he does not say they are only bad sometimes. He simply says they suck. You say they don’t suck but quite the opposite are highly valuable. Then you say you agree with Wil. What is the real story. You can’t agree with his post and say Unit Tests are highly valuable at the same time.

  8. bbum says:

    Wil has never claimed that his weblog is an end-all, be-all, commentary on the state of the world. It focuses on his opinions about the stuff he is doing in his world. Given that context, his statements about unit testing are perfectly valid for the kinds of software he writes and the process he uses.

  9. La Cara Oscura del Desarrollo de Software » Unit Testing says:

    […] Suena muy bonito en teoría, pero ¿Qué son los Unit Test en la práctica?. Según uno de los desarrolladores de Delicious Monster Wil Shipley “unit testing is teh suck” . Leyendo bbum’s el autor nos habla de su experiencia desarrollándo Core Data y de como el uso de Unit Testing lo ayudó inmensamente. ¿Dónde está la verdad?, yo creo que en algún lugar en el medio y sobre todo dependiendo del tipo de aplicación que estemos desarrollándo y de la forma en que lo estemos haciéndo. […]

  10. Rob Rix says:

    Just wanted to note, my -copyWithZone: implementation for NSManagedObject is now on a different site. I’ve had a couple of people e-mail me so I figured I’d let anyone know. Sorry about the post!

  11. Houman says:

    Well Put.

  12. Oscar Morales Vivo says:

    Ah, great to find someone who had put in words what I was thinking about. In the end, testing infrastructure tends to be far less problematic than doing it to GUI programs since the inputs are much better controlled and easier to code into being. For actually breaking a shipping application, however, there’s nothing like users :P.

    Now the hard part will be to convince the other devs in my team to actually use unit testing (for those parts of our work where it makes sense to do so). They still have occasionally a problem with the whole ‘comment your code’ thing so I’m not too optimistic about it :P.

  13. Scott’s Blog » Why Unit Test says:

    […] I came across a great post on unit testing today that provided not just why to unit test, but what to unit test. Differentiating software into “infrastructure” and “end user application” really highlights what sort of code will benefit the most from the approach. Another really useful bit of this article talks about turning bug reports into unit tests. It’s a very smart idea that I’ll try to use on my own code whenever it’s appropriate. You can also bookmark this on del.icio.us or check the cosmos […]

  14. Bill Dudney says:

    Hi,
    How do you debug your tests? I’ve googled around quite a bit for an answer and its not jumping out of the web at me. Hoping you know…

    Thanks

  15. Don Lapre Lover says:

    The post was interesting to read. I have always been a fan of the automated test but there is something to be said about one on one.

    Laura
    Don Lapre Lover
    http://www.lauraglydaband.com
    laura@lauraglydaband.com

  16. The End of the Long Tail » To Unit Test or Not to Unit Test? That is the Question. says:

    […] I said, “rarely a good practice”. Here an Apple developer explains when unit testing is not a waste. Appreciate the differences in […]

  17. Dan Waltin says:

    This is a late comment, I just found this entry (which has many good points)! Now, in response to Bill Dudney “How do you debug your tests?”: Given that the (unit) tests are supposed to reveal bugs in your code, one approach I will test in my current project is to appoint a “saboteur”. This is a person that takes a branch of the code an insert bugs to see if the tests find them!

    Insert a “throw new ApplicationException(“This has not happened”) (*) here and remove an input validation there…

    This is not automatic, I know, but I guess that a good sabouteur can make a difference. Anyone tried this technique?

    /Dan

    (*) Unfortnunately I’m not able to program Objective-C for a living, but C# is really quite nice…

  18. Christopher Humphries says:

    Wow, thanks. Very good post (got from linked off Wil’s blog post on unit testing).
    Bookmarking

  19. Sander’s Weblog » Unit Testing Not Necessarily teh suck says:

    […] course exaggerating for effect and the post is best taken in together with BBum’s excellent follow-up where he argues that unit testing made a huge amount of sense for his project, but may be less […]

  20. Red Sweater Blog - Unit Testing Roadblocks says:

    […] existing code base. I won’t go into much more about the whens and ifs, but Wil Shipley and Bill Bumgarner have delved deeper into this question, if you’re interested in reading […]

  21. Unit Testing: Good/Bad? says:

    […] Bumgarner counters with a good argument of when this case occurs and how unit testing can solve a […]

    [Ed:] Unfortunately, the mojomonkeycoding.com weblog’s comment system is busted. The author of that post missed the point. If he had been paying attention, he would realize that unit testing is critical in a well designed app in that you will have a model/control layer that is a great UT target.

  22. Lukasz Wrobel says:

    While I regard writing unit tests to be controversial, I decided to write out a few most commonly encountered opinions and talk them over: Unit Tests – Love, Ignorance and Getting Real

    Perhaps you will become interested in this article, especially if you have a strong (either positive or negative) opinion on applying unit tests.

  23. Gin margarita via unit testing UIs – Ross Henderson says:

    […] found it by luck, reading this post, about unit testing, which was a response to this post about testing […]

  24. References on Unit Testing & UI Automation for iOS Applications | Jojit Soriano's Blog says:

    […] testing is difficult for application development – Unit Testing is teh suck, Urr. by Wil Shipley + Unit Testing. by bbum – How to Write Your Own Automated Testing Framework by Gus […]

  25. bluelobe says:

    If one is an expert in SQL and has years of experience doing SQL and has mastered SQLite in iOS, would you recommend they still move to CoreData? When I say expert, I mean, all their apps are full-featured db-wise and only use SQLite and do so quickly, bug-free, and development time is super-fast and super-efficient.

  26. Discovering my Unit Testing Philosophy « gerardcondon says:

    […] a response to Will’s article, a follow up post by bbum shows how unit testing was of great benefit on infrastructure code, in this case the Core […]

Leave a Reply

Line and paragraph breaks automatic.
XHTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>