Unit Testing.
Yeah, I can’t avoid this one.
Wil Shipley wrote a great article about his opinions of unit testing. Of course, the community response has been the fairly typical “He’s all wrong” / “He’s totally right” extremes.
Whatever. I would rather relay an anecdote.
As Duncan wrote, we used OCUnit to unit test Core Data. I integrated OCUnit into the Core Data project within a few weeks of the first lines of code being written. The experience was so positive that we decided to integrate OCUnit into Xcode (A huge note of thanks to the Sen:Te folks) and released the result at WWDC ’05 in Xcode 2.1.
Core Data 1.0 is not perfect, but it is a rock solid product that I’m damned proud of. The quality and performance achieved could not have been done without the use of unit testing. Furthermore, we were able to perform highly disruptive operations to the codebase very late in the development cycle. The end result was a vast increase in performance, a much cleaner code base, and rock solid release.
During the development cycle, we saw nearly zero regressions. And, of course, Core Data was developed on top of and along with Tiger. That is, we were frequently running in a highly unstable environment. Time and again, the Core Data unit testing caught problems that we are actually in components below Core Data.
Not only did the unit tests ensure the quality of our code, but it helped other teams to quickly find and fix bugs in other components.
Sure, writing the unit tests was occasionally tedious and maintaining the tests consumed a bunch of engineering time. I can say with complete and total confidence that the return on investment has been huge.
And the ROI continues to be high. When a developer files a bug against Core Data, the bug is generally turned into a unit test to verify that the bug really exists and the exact details. This greatly streamlines the whole bug fixing process and ensures that bugs stay fixed.
The team is also free to pursue some radical and disruptive changes to the framework for the next version with the confidence that it will remain completely backwards compatible.
Why such a different experience than Wil’s?
Software can generally be said to fall somewhere along a line between “infrastructure” and “end user application”. Core Data clearly falls at the “infrastructure” end of the scale while Wil’s application — the rather awesome Delicious Library — clearly falls at the other end of the scale.
And that clearly is the focus of Wil’s testing philosophy. Combined with his “the best software is the software with the fewest lines of clearly written code” attitude, his approach to testing is 100% entirely user driven.
That makes sense for his software.
I spoke with Wil at WWDC and he was very complimentary of the Core Data 1.0 release. I am thoroughly flattered by that as I have a huge amount of respect for Wil and that he was impressed by something I worked on makes me very happy indeed.
Given his “least code possible” philosophy, one could easily imagine that Delicious Library 2.0 or 3.0 will use Core Data as doing so would eliminate vast swaths of code he must have today. I hope he does exactly that at the same time he decides its OK to ship a Tiger+ only product.
And that is the answer. A software developer like Wil — a totally user focused, user experience driven, engineer — may not see the value of using unit testing directly in their development efforts. However, that Delicious Library could use something like Core Data and does use all of the other technologies in Tiger would not be possible without the huge investment in Unit Testing by the various teams at Apple.
Unit Testing is awesome for guaranteeing consistent behavior of infrastructure. UT can be very effectively applied to user interface level code, too, but it is a little harder to do.
But Unit Testing cannot test User Experience and, in the end, a quality UE is what turns a brilliantly engineered piece of software only appreciated by other engineers into a brilliantly designed products used by millions.
September 26th, 2005 at 8:57 am
hi-
Automated testing is always good. You are right about using it to revalidate previous tests for new revs. I groan about having to manually test things on each rev. and the “try not to break things” to coding is very hard :-).
However, coredata does suffer a bit from the lack of unit testing for Oracle adapters :-). Or, to put it another way, it might be better to write a Oracle adapter than to write a unit test to test the Oracle adapter. Or am I missing something or is my info out of date? These are the type of issues I think Wil was thinking of: With limited resources a developer needs to focus on user solutions, with unlimited resources a developer can do whatever.
Plus, with a big team I have found that one developer breaks another developer’s code, unbeknownst to either developer, especially with concurrent development systems. Some developers prefer ownership of big sections of code and lock out other developers from direct access so things don’t break as often and humans can be responsible for not coding wrong thus making unit testing less important and satisfying user needs more important. It seems to be a balancing act.
thanks for coredata!-
-lance
September 26th, 2005 at 3:32 pm
Hey, if Lance can say “Oracle adaptors”, I wanna say “Sybase adaptors”.
I love the stuff Wil writes in his blog, both the lifestyle stuff (“is helicopter an indoor toy”), and the deliberately contentious code-related stuff like “teh suck”. Clearly Wil has devised mechanical aids for seeing the wood for the trees — maybe Delicious Library can handle bark codes.
I’d guess Wil has foresworn unit testing for his stuff not because he thinks it isn’t useful, but because GUI stuff can be, like, really awkward when you’re trying to fit it to a test harness, and any proposed incorporation of unit tests did not survive the first cost/benefit review. Libraries and server stuff — webapps figure big here — are easier to instrument, so all the pass/fail pithy goodness is available at an affordable price in man months. Hey, the J2EE stuff I’m doing these days has tests of a sort, while the J2ME stuff doesn’t.
Like so much, the real answer to “to unit test, or not to unit test” is “how long is a piece of string”. It’s fun to read the comments following Wil’s blog posts to see just how many people don’t get it.
September 26th, 2005 at 5:00 pm
Bizarre responses. If you want an Oracle Dysdapter that does not actually work, sure have the “engineers” slam something together and ship it. Just make sure they attach a “Developed in the style of MicroSoft” label to it. If you actually need your Sybacle adapted, then you need an Orabase Adapter that actually works, and the only way to know if it actually works is to test its functionality. You _reduce_ development time, not increase it, if you write the tests first because then you find the bugs at the earliest possible moment and fix them before they are confused by other bugs. The tests need to be written eventually, or else you are just shipping garbage.
As for personal projects, I skip the Unit Tests because my code always works the first time; but for group projects Unit Tests are indispensible.
September 27th, 2005 at 7:02 am
As for personal projects, I skip the Unit Tests because my code always works the first time;
Yeah and our earth is a disc. Sometimes it’s really fun to read comments.
September 27th, 2005 at 8:56 am
[…] Looking at the comments, there’s a bunch of people that agree with Wil’s reasoning on face value (“Yeah! Unit testing sucks! It takes up so much time!”), but then you have others who disagree (to an extent). The most enlightening post I’ve seen on the topic is by Bill Bumgarner, who notes that unit testing was used by the team developing Core Data to make sure the code adhered to the specification (or expected behaviour), as well as preventing regressions when bugs were fixed. […]
October 9th, 2005 at 1:11 am
Excellent post, Bill, and it definitely highlights the value of unit testing in the approach that you guys took–CoreData is a success, and quite a lot of fun to work with. I’d love to be able to handle Oracle and Sybase and whatever other sorts of DB stores one can think of, certainly, but I consider that secondary to the fact that CoreData works well as is.
And I’m saying that from the standpoint of someone who just spent two days writing a -copyWithZone: implementation, and thereby learning a lot more about the framework inside-out. It’s beautiful, and the fact that you can do stuff like a generic copy method that takes ownership into consideration is a big deal. (And yeah, I’m kinda plugging my own code there, but I just finished it, I’m kind of on a high here.)
So yeah, I can verify that experience with unit testing, but from the opposite side– I’d have had a better time of my own code here had I unit tested. Cos infrastructure code tests easily.
Great job, by the by. I can’t wait to drop Panther so I can use CoreData everywhere.
December 5th, 2005 at 11:58 am
I don’t understand.
You say Wil’s post is good and you have huge respect for him. You say you unit test and it is extremely valuable to your project. You say Wil may not value unit tests because of his type of software. But Wil simply says Unit Tests suck. He does not say they are good in some situations, he does not say they are only bad sometimes. He simply says they suck. You say they don’t suck but quite the opposite are highly valuable. Then you say you agree with Wil. What is the real story. You can’t agree with his post and say Unit Tests are highly valuable at the same time.
December 5th, 2005 at 12:57 pm
Wil has never claimed that his weblog is an end-all, be-all, commentary on the state of the world. It focuses on his opinions about the stuff he is doing in his world. Given that context, his statements about unit testing are perfectly valid for the kinds of software he writes and the process he uses.
December 8th, 2005 at 1:28 pm
[…] Suena muy bonito en teoría, pero ¿Qué son los Unit Test en la práctica?. Según uno de los desarrolladores de Delicious Monster Wil Shipley “unit testing is teh suck” . Leyendo bbum’s el autor nos habla de su experiencia desarrollándo Core Data y de como el uso de Unit Testing lo ayudó inmensamente. ¿Dónde está la verdad?, yo creo que en algún lugar en el medio y sobre todo dependiendo del tipo de aplicación que estemos desarrollándo y de la forma en que lo estemos haciéndo. […]
December 20th, 2005 at 3:28 am
Just wanted to note, my -copyWithZone: implementation for NSManagedObject is now on a different site. I’ve had a couple of people e-mail me so I figured I’d let anyone know. Sorry about the post!
May 22nd, 2006 at 6:02 pm
Well Put.
July 5th, 2006 at 4:07 pm
Ah, great to find someone who had put in words what I was thinking about. In the end, testing infrastructure tends to be far less problematic than doing it to GUI programs since the inputs are much better controlled and easier to code into being. For actually breaking a shipping application, however, there’s nothing like users :P.
Now the hard part will be to convince the other devs in my team to actually use unit testing (for those parts of our work where it makes sense to do so). They still have occasionally a problem with the whole ‘comment your code’ thing so I’m not too optimistic about it :P.
August 29th, 2006 at 11:08 am
[…] I came across a great post on unit testing today that provided not just why to unit test, but what to unit test. Differentiating software into “infrastructure” and “end user application” really highlights what sort of code will benefit the most from the approach. Another really useful bit of this article talks about turning bug reports into unit tests. It’s a very smart idea that I’ll try to use on my own code whenever it’s appropriate. You can also bookmark this on del.icio.us or check the cosmos […]
November 1st, 2006 at 7:22 am
Hi,
How do you debug your tests? I’ve googled around quite a bit for an answer and its not jumping out of the web at me. Hoping you know…
Thanks
November 27th, 2006 at 6:36 pm
The post was interesting to read. I have always been a fan of the automated test but there is something to be said about one on one.
Laura
Don Lapre Lover
http://www.lauraglydaband.com
laura@lauraglydaband.com
March 11th, 2007 at 9:36 am
[…] I said, “rarely a good practice”. Here an Apple developer explains when unit testing is not a waste. Appreciate the differences in […]
July 3rd, 2007 at 4:46 pm
This is a late comment, I just found this entry (which has many good points)! Now, in response to Bill Dudney “How do you debug your tests?”: Given that the (unit) tests are supposed to reveal bugs in your code, one approach I will test in my current project is to appoint a “saboteur”. This is a person that takes a branch of the code an insert bugs to see if the tests find them!
Insert a “throw new ApplicationException(“This has not happened”) (*) here and remove an input validation there…
This is not automatic, I know, but I guess that a good sabouteur can make a difference. Anyone tried this technique?
/Dan
(*) Unfortnunately I’m not able to program Objective-C for a living, but C# is really quite nice…
August 15th, 2007 at 5:49 pm
Wow, thanks. Very good post (got from linked off Wil’s blog post on unit testing).
Bookmarking
December 30th, 2007 at 7:09 pm
[…] course exaggerating for effect and the post is best taken in together with BBum’s excellent follow-up where he argues that unit testing made a huge amount of sense for his project, but may be less […]
August 1st, 2008 at 5:43 pm
[…] existing code base. I won’t go into much more about the whens and ifs, but Wil Shipley and Bill Bumgarner have delved deeper into this question, if you’re interested in reading […]
August 25th, 2009 at 2:44 pm
[…] Bumgarner counters with a good argument of when this case occurs and how unit testing can solve a […]
[Ed:] Unfortunately, the mojomonkeycoding.com weblog’s comment system is busted. The author of that post missed the point. If he had been paying attention, he would realize that unit testing is critical in a well designed app in that you will have a model/control layer that is a great UT target.
February 10th, 2010 at 4:15 pm
While I regard writing unit tests to be controversial, I decided to write out a few most commonly encountered opinions and talk them over: Unit Tests – Love, Ignorance and Getting Real
Perhaps you will become interested in this article, especially if you have a strong (either positive or negative) opinion on applying unit tests.
July 15th, 2010 at 2:38 pm
[…] found it by luck, reading this post, about unit testing, which was a response to this post about testing […]
June 2nd, 2011 at 9:03 pm
[…] testing is difficult for application development – Unit Testing is teh suck, Urr. by Wil Shipley + Unit Testing. by bbum – How to Write Your Own Automated Testing Framework by Gus […]
August 10th, 2011 at 12:52 am
If one is an expert in SQL and has years of experience doing SQL and has mastered SQLite in iOS, would you recommend they still move to CoreData? When I say expert, I mean, all their apps are full-featured db-wise and only use SQLite and do so quickly, bug-free, and development time is super-fast and super-efficient.
February 9th, 2012 at 7:09 pm
[…] a response to Will’s article, a follow up post by bbum shows how unit testing was of great benefit on infrastructure code, in this case the Core […]