Open sourcing is not easy

Recently, John Gruber (DaringFireball) and Tim Bray have been conjecturing on why or why not Apple doesn’t open source its apps.

For many reasons, I’m not going to commit on the political or business side of the decision. I don’t have any knowledge of why Apple — or any other company — has chosen to close or open what they have.

Instead, I’m going to point out a fact that Bray seems to largely ignore and Gruber only touches on in passing.

Open sourcing a large body of source that has always been closed is really damned hard.

It takes a monumental amount of effort to do so and, once open, it takes a whole bunch of ongoing effort to keep it open in a productive fashion (which Gruber does cover, actually).

At the lowest level, pretty much every line of code has to be reviewed to ensure that it isn’t offensive, doesn’t cross any number of lines (licensing, intellectual property, etc..), is formatted reasonably well, and every file must contain the appropriate header.

For any company that has a suite of products where some build on others — especially when the others are developer tools (think Sun’s JDK or IBM’s libraries or Microsoft’s DLLs) — it is quite likely that any given piece of code is going to contain stuff done “the wrong way”. It may have been “right” or “only way” when written and you may have intended to fix it later, but it just hasn’t happened yet.

As a result, you are likely going to have to break the project across potentially unnatural lines to separate the parts to be open sourced from the parts that need to remain closed for various reasons. And the reasons are many and not just because you want to keep something secret. Sometimes the code is so completely counter to the way developers should develop apps that you don’t want to run the risk of other developers doing the same thing (and there are very valid reasons for such a case to arise — bad engineering is rarely the reason).

Then there is the intellectual property. If there are any random patents embodied in the code — of which there likely will be in this day and age — you have to figure out exactly how open sourcing said implementation changes the whole IP landscape.

Example: Before joining Apple, I was involved in various IP related conflicts as they meandered through the court system. In one case, one party used open source to bolster their position. In particular, they used the openly available implementation to strengthen their particular position — had that source remain closed, that particular defense/offense would not have been available. (Suitably vague to protect the, uh, innocent? guilty? wasn’t really clear)

And once you have everything all cleaned up, have the license selected, and are ready to roll it out, you have to expend the engineering effort to actually set up a community portal of some kind. Bug tracking. Source repository. Some kind of channel for receiving patches. It is going to be an ongoing engineering investment.

There is no such thing as “just publish the source and be done with it”. Publishing source — be it under an open or closed license (like the GPL) — changes a product in unavoidably large ways. Sometimes good. Sometimes bad. Always a lot of effort to deal with.

20 Responses to “Open sourcing is not easy”

  1. Stephane says:

    I would add one requirement to the list of requirements at the end of your post. You have to make people realize you are still committed to improve the project after releasing it as Open Source. A bunch (if not the vast majority) of projects released as Open Source seems to be dead-project. Sometimes, I think Source Forge is more a cemetery than a forge.

  2. Johannes Fortmann says:

    Interesting counterpoint: TextEdit (the OS X app) is open source. Nobody even seems to care. All the bad things John Gruber mentioned (people backporting stuff to earlier versions of OS X, people forking the code base) haven’t happened (all cool changes are in Cocoa, not TE, anyway).
    TE does not have a community portal. And it only seems to serve as a warning to others: look at how it’s done in the TE source; if you’re doing it differently, there’s a reasonable chance you’re doing it correctly. It’s a document-based app implemented without using NSDocument, for chrissakes!

    I think releasing stuff like the source to would very much help people to improve on the OS, and show them how it’s supposed to be done. Bad examples with a “don’t do it like this” sign are examples, too.
    No matter how badly designed it would be, it couldn’t be any worse than TE. And I would get rid of those horribly-looking capsule-hover-barf buttons…

  3. leeg says:

    Let’s take an example of a project where only two-thirds of the effort was followed through: Darwin :-). Licensing/IP issues sorted out? Check. Source code inoffensive? [s/EDOOFUS//] Check. Ongoing engineering effort into the community portal? OTOH, we could just give the ISC a G4 PowerMac and The Community will come running into our arms…

  4. bbum says:

    TextEdit? It is just a developer example — with some serious flaws that are noted in the README — that happens to be a useful, though very basic, text editor. Anything interesting about TextEdit is in the frameworks it sits on.

    I just love how people try to make the technical problems with open sourcing sound somehow trivial to address. They aren’t — not by a long shot.

  5. Paul Suh says:

    Amen! I think that you left out the fact that any serious open source project needs documentation. For the bits and pieces that I have released open source on my web site, I ended up spending at least as long writing up documentation as I did writing the code. For most open source projects, releasing the project without adequate documentation is a guarantee that no one can or will use it. Yet another way to get to the cemetary that Stephane describes.

  6. Michael Tsai - Blog - Open Sourcing Is Not Easy says:

    [...] Bill Bumgarner: [...]

  7. Scott Steinman says:

    Going open source does not necessarily mean that benefits will be reaped, and therefore remains a crap shoot. I’ve been in charge of two open source projects. In both cases, a large group of people in the community encouraged us to go open source and promised support. On one, outside of a small group of dedicated programmers who started the project, nobody’s joined in. On the other, I’m still the sole programmer and documentation writer! So all of our code is now public, but the state of coding has not improved.

    I’m not saying that this will happen to all open source projects, and I’ll still probably open source some of my projects just for the sake of the community. I just wouldn’t assume that open sourcing, even with the proper license and good documentation, will benefit the person or organization that releases the code, or even the software project.

  8. Johannes Fortmann says:

    So what? Most of the Apps that would likely be open-sourced are little more than “developer examples”. Nobody wants Apple to open-source iPhoto or Pages. We are talking about little apps like Address Book, Preview, Stickies or Mail. These Apps could easily be open source, enabling people to extend them and learn from them.
    I used TextEdit as an example because it clearly shows that Apple does not particularly care if their source releases show horrible programming styles.

  9. bbum says:

    I kept the original post strictly non-Apple specific for a reason. None of this is Apple specific — the same holds true for Microsoft, Sun, IBM, Maya, Wolfram or any other commercial entity with a closed source base. Please keep it that way.

    Go read Gruber’s post and my post and think very carefully about the monumental effort that would be required to release any of the primary applications on a major [non-Linux/BSD] operating system as open source. Do a patent search and find all of the patents related to said applications. Go and have a look at the size of the binaries and the contents therein, including everything they link against.

    Then ask yourself “What would be the benefit to the company if any given application were open sourced?”

    In utopia, the community would fix the products, supply patches, while the corporate entity would continue to bask in the glow of success — could continue without diluting brand, ending up with a kitchen sink’s worth of mismatched features, and could do releases in a highly organized and timely fashion.

    But this is the real world. More likely than not, a whole bunch of people would check out the source, start trying to do something with it, and give up. Of the handful that persevered, they would come up with half-assed features or bug fixes that huge implications in areas they don’t use, none of which are really very useful to the corporate entity.

  10. Rod Begbie says:

    The all-time classic example of how hard it is to open-source something succesfully.

  11. Jason Smith says:

    Bravo, bbum. As someone who has been down this road before, I can only applaud this article greatly. My dissertation involved a ~50kLOC Python project that I have since had numerous requests for providing as OSS. The only problem is, the work involved getting into a shape ready for publication like that is just immense. (I had enough work just getting the XML Schemas, etc, ready for public consumption – the guts of the tool would just be too much to deal with in addition to my regular workload.)

    Sure, anyone can toss any old code out in the public domain and call it open source… but companies can’t afford to do that. Their business model doesn’t involve tossing crap against the wall to see what sticks (well, most businesses at least.) What they publish reflects on them as surely as products they bring to market deliberately, and there has to be a positive end result for them to put the effort into creating a public version of the code.

    I agree with Gruber that the best approach is to create polished public APIs for extension, and leave the core private. That gets the vast majority of the benefit that the coopt-code-compile folks want, without the pain that comes with it for those not steeped in geek.

    Nice one.

  12. Harald Korneliussen says:

    Stephane, sourceforge is indeed a graveyard-for projects started there. For projects that had a bit of history before going public (either personal, internal or commercial), the survival rate is much better, however.

    Of the expenses needed to launch a project as open source, many, if not most, are sensible, good investements. You should know the IP status of all your code, you should keep it clean and presentable, etc. And it should certainly be well-documented…

    Scott Steinman, I wonder if you could not have profited from taking a small fee for open-sourcing your apps. There’s nothing wrong with making money on open source this way, and perhaps people will follow up more if they make a little economic commitment as well. The poster child of this kind of release is Blender, but even small apps could probably use it… economic tools like makes it less risky for the buyers.

  13. Johannes Fortmann says:

    I think I misunderstood the basic premise of your article: I’ve seen this as a followup to someone’s (Mark Pilgrim-inspired?) idea that Apple should open source their _basic_ applications, an idea I’d wholly support. Of course open-sourcing major apps is completely different from o/s-ing the small stuff.

    But I’d also like to point out that (re John Gruber’s post) few things further basic extensibility more than having (part of) your code out for the whole world to see (especially in the case of the system-delivered utilities).
    As an example, take pre-Tiger Mail. After Tiger came out, many have tried to just keep on using the old Panther, just to recognise that it uses private functionality in Message.framework, which isn’t implemented in Tiger. An open source app (normally) cannot do anything like that; this encourages a clearer design (IMHO). Of course, this has nothing to do with your article anymore.

  14. Pupeno says:

    I think a good way to release proprietary code as free software would be to start, slowly, separating a big project into smaller ones. That is, group parts of the application that can be reusable and separate them in a library, then, release the library as free software.
    The smaller you can make each chunk the better because you can stop at any moment by any reason (not having the resources, change of command of the company and by that of philosophy, going bankrupt, etc) and still some part will be out there and ready to continue its life.
    Eventually, the application may become just a shell for a bunch of free software libraries and you’ll be done (after releasing the shell, off course).
    Another note, you don’t have to set up a comunity portal or anything like that; there are many free software projects developed behind closed doors and in secrecy, it is not the best approach but it is doable.

  15. Eric Foster-Johnson says:

    Great post. I totally agree with you, it is very difficult to take old, proprietary code and turn it into something acceptable for an open-source audience. It may well be worth the effort, though. A few quick reasons and then a longer one follow. Note that I completely agree with your premise–it is hard.

    * Going through the effort, whether you ever release the code under an open-source license, will dramatically improve the quality of your code and help protect your organization from legal challenges (patents and so on). It will require a lot of effort. You may want to perform the work incrementally.

    * If your code ever gets out into the public, having clean code can help preserve your organization’s reputation. Remember when Windows source code got into the wild? Like or dislike Microsoft, it is not good for public relations when newscasters are describing how much profanity lies in your source code.

    As a more important reason, having your software available under an open-source license acts as a form of protection for your customers’ investments in your firm, especially in the long term. Companies come and go. Even companies that remain may decide to no longer support a product or even remain in the same industry segment. It may not be worth the company’s efforts to continue to support the product, but it may well be worth your efforts to keep the product working in your environment.

    If the crucial software for the product is available under an open-source license, then you have a much better chance to keep the product going for your environment.

    I find I am much more likely to purchase a product for which I think there are options for long-term support, even if the company, for whatever reason, loses interest in the product.

    Thinking long term, here are some examples of reasons why going through the difficult exercise of open-sourcing your software may help convince customers to purchase your products:

    * The state government in Massachusetts has been looking into a file format called ODF as a means to be able to read documents created today years from now. The current Microsoft Word does not import files created by really old versions of Word. Without some kind of open-source product, you are unlikely to be able to read files created just a few years ago. Having your files stored in an open, unrestricted format (unrestricted by patents, the US DCMA, and so on), gives you better long-term options for preserving your data.

    * If you look to purchase a music player, your choices are going with Apple’s iPod with its proprietary software and vendor lock-in, or going with the proprietary software and vendor lock-in from one of a number of other companies, most of which are unlikely to even be in the market in a few years. If you work for one of these also-rans, open-sourcing your software could help provide some measure of difference from the dominant product and could help make up for a clunky physical design and crappy software.

    Each company needs to determine what it really is selling and how it makes its money. In a great many cases, the software is not what creates the revenue. The software helps create the revenue, but is likely not the source of the revenue itself. Similarly, chances are your organization has a Web site. For most firms, the Web site itself does not generate revenue. But, if the Web site looked totally unprofessional, chances are quite a few potential customers would have second thoughts of doing business with your firm.

    Anyway, there may well be good commercial reasons for open-sourcing your software, especially when you think long term.

  16. Mark says:

    Id Software doesn’t seem to have been crippled by any of these problems as they’ve GPLed most of their games.

  17. bbum says:

    First, ID has never open sourced any currently shipping game. Secondly, they open sourced otherwise dead projects. ID is clearly taking the approach of open sourcing technologies whose useful revenue generating lifespan has come to an end.

    In other words, ID’s actions have no relevance in this discussion.

  18. Daniel Eran says:

    I’ve written a series of articles on open source, comparing the benefits with the risks and difficulties developers face. One of the clearest points that has jumped out at me in researching them, is the complexity in opening up development, and the significant efforts it takes to support it as an ongoing project. Your posting really highlights that. Many of the open projects out there are open to stave off an otherwise certain death.

    I also wrote a different take on Gruber’s back porting worries, and the fear that Apple would lose significant sales from open source competition: the problem really isn’t about money or lost revenues, it’s the effort required, as you point out.

    CNET’s Charles Cooper Strikes Out in iPod Attack
    There’s a common misconception about what it means to be proprietary. Here’s a disassembly of one of the worst articles yet on the subject, written by CNET’s executive editor, Charles Cooper.

    Open Source Values and the Peanut Gallery
    The value proposition involved in choosing an open source strategy, and a roast of the emerging peanut gallery who are attempting to hijack and betray the free software movement.

    BSD and GPL: Different Sources for Different Horses
    The benefits and the motivations behind two very different styles of open source development: the BSD style license, pioneered by UC Berkeley and MIT; and the GPL invented by Richard Stallman, the founder of the free software movement.

    The Revolution Will be Open Sourced!
    Over the last decade, every player in the software development industry has been dramatically affected by an open source revolution. How will Apple adapt to fit into this new world? Are they leading, following, or falling behind? Do they stand to benefit from an increased adoption of open source practices, or will they simply have to change how they do business?

    Apple and Open Source… Strange Buffaloes?
    Tim Bray’s “Time to Switch?” and John Gruber’s “Why Apple Won’t Open Source Its Apps” both discuss the potential risks and benefits Apple would face in open sourcing their consumer applications. Here’s my take: Apple does not make fierce profits from $130 Mac OS X retail sales, and there isn’t a conspiracy behind new apps not working on an old OS.

    The ‘Mac OS X Closed by Pirates’ Myth
    According to the proponents of this myth, Apple has abandoned their open source initiatives as they move to Intel, because they are afraid that, armed with the Darwin source code, pirate 3lit3 haxx0rs will p0wn them and have Mac OS X running on generic PCs. They’re wrong, here’s why.

  19. A MacMudgeon Lost in Penguin-Land » Open sourcing is not easy says:

    [...] sourcing is not easy Permalink | Jul 18In a response to bbum’s weblog-o-mat » Blog Archive » Open sourcing is not easy, RodBegbie cites the rocky road of to illustrate the difficulty of open-sourcing product. [...]

  20. Memoria de Acceso Aleatorio » El kernel de Mac OS X 10.4.7 para Intel, abierto says:

    [...] weblog-o-mat: Open Sourcing is not easy (vía DaringFireball linked [...]

Leave a Reply

Line and paragraph breaks automatic.
XHTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>