Archive for the 'Mac OS X' Category

Little Snitch 2.0 now available

Thursday, November 8th, 2007

Objective Development relased Little Snitch 2.0 today.

Little Snitch is an outbound firewall, more or less. That is, it monitors all outbound connections and warns if an application makes any kind of connection for which there isn’t a rule permitting said connection.

The warning panel that comes up allows the user to allow or deny the conneciton and to do so one time only, until the app in question quits, or foreever. You can also specific that the app should only connect to a specific server and port, a specific server, a specific port (HTTP, for example), or any connection at all.

LS has an awesome rule editor built in and 2.0 adds a really neat traffic monitor.

All in all, it is a beautifully designed and highly functional piece of software.

I tend to run Little Snitch all the time. It can be a bit noisy when it is first set up, but that is quickly rectified as you refine your rule set.

Little Snitch has busted a handful of applications that “called home” or made connections to surprising places. (I busted an Apple application that was doing so — it was an accident and it was fixed before the software as ever released).

PyObjC 2.0 Source Available

Thursday, November 1st, 2007

Ronald comitted the source for PyObjC 2.0 into the public repository.

It can be found at PyObjC 2.0 trunk (http://svn.red-bean.com/pyobjc/branches/pyobjc2).

The NEWS.txt file is extremely illuminating as there were tons and tons of changes and bug fixes in 2.0.

It cannot be emphasized enough; this is a huge release for PyObjC. As big as when PyObjC first gained subclassing (which wasn’t until about 7 years into its 14 year history). Go read the NEWS file; both major enties on PyObjC 2.0.

Note that libffi on Leopard is fully 64 bit and, thus, PyObjC should be mostly ready for 64 bit, though it was neither built for, nor tested with, 64 bit.

“It just works”: MAME OS X & Totally Cheap GamePad

Wednesday, October 17th, 2007

I recently had a hankering to play a bit of classic video games and downloaded MAME OS X. Awesome port; Dave Dribin has done a completely kick-ass job!

Of course, playing via keyboard is painful. Especially for a 7 year old.

So, I browsed over to amazon and looked up USB game pads. Finally, I settled on the one pictured at the left. Why that one? Cheap and delivered via Prime.

Ordered and received.

Opening the box revealed no drivers. Going to the Logitech website revealed only Windows drivers. Searching around revealed some HID hackery that might work with some god-awful keyboard mapping things.

Wait — maybe MAME OS X has a clue in its documentation. If it has any.

It does! Built in! And it says:

Using Joysticks and Gamepads

USB joysticks and gamepads should be detected when the game starts up. MAME OS X does its best to map joystick buttons and inputs to MAME inputs. The player mappings should be based on which USB port the joystick is plugged into. If you have multiple joysticks, and the player order is incorrect, try swapping the USB ports.

OK… here goes!

It just works.

Dave, dude, you rock. Beers on me the next time we are in the same city. Thank you.

The game pad, btw, is about what you’d expect for $10. It works, but doesn’t compare to real arcade sticks. Maybe I’ll rip it apart when I get home and solder it into my big old honkin’ scrap wood arcade control deck.

What about turning on auto_vacuum on Mail’s envelope index?

Sunday, March 4th, 2007

A couple of days ago, I commented on the vacuum-your-mail-to-make-it-go-faster meme passing through the Mac weblogging community.

In response, Garrett Albright suggested that one could turn on auto_vacuum through a bit of a hack (nice one, too — see the comments).

I wouldn’t do that. It will work. Most of the time. And then it won’t. While there is very little data loss risk — the Envelope Index is just that, an index of all of your Mail — there is a risk and it was clearly left off in Tiger for a reason.

First, auto_vacuum was new to SQLite 3.1.0 and Tiger ships with SQLite 3.1.3, which won’t be updated unless a serious security hole is found. There have been a number of bug fixes to vacuum and vacuum related functionality since 3.1.3. In particular, there have been a handful of fixes that prevent relatively esoteric situations from corrupting a database. Some of these were directly related to auto_vacuum and some were simply catalyzed by auto_vacuum.

Secondly, it isn’t going to buy you that much performance over periodically vacuuming the database. Most of you who have manually vacuumed your database are seeing a performance boost related to the removal of 10s of months of garbage built up in the database. Running vacuum after only a week or two of Mail activity is going to yield very little benefit.

That is, the additional risk of running with auto_vacuum enabled with a 3.1.3 based SQLite database isn’t worth the very slight additional performance you’ll see on a day to day basis. Better to simply manually vacuum the database every six months or so.

Vacuuming Mail’s Envelope Index to make Mail faster (and Aperture, too)

Friday, March 2nd, 2007

Update:

In response, Garrett Albright suggested that one could turn on auto_vacuum through a bit of a hack (nice one, too — see the comments).

I wouldn’t do that and I have explained why in a followup post.


The weblogging world is all abuzz with this simple hack which makes Mail.app faster, more or less depending on how your Mail happens to be configured and your coincidental usage history.

There seem to be quite a few questions about what is going on. I figured I’d throw out a bit of clarity to the situation.

To invoke the command, do the following:

(1) Quit Mail. This will prevent any lock contention issues.

(2) Go to Terminal and copy/paste the following line:

/usr/bin/sqlite3 ~/Library/Mail/Envelope\\ Index vacuum

The details:

/usr/bin/sqlite3 refers to the SQLite command line tool. Nir Soffer suggested that I use the absolute path to avoid any conflict with sqlite3 binaries installed by things like Fink, MacPorts, or the like. He is correct and I updated the command. It is an interactive shell that can be used to interact with a database or with multiple databases.

~/Library/Mail/Envelope\ Index refers to Mail’s envelope index, a SQLite database that Mail uses as a cross reference for all your messages, mailboxes, and accounts. The ~ means “in my home directory” and the \ is shell speak for “no, really, I meant for the next character to be treated as data, not as some special invisible Unix goop”.

vacuum is a SQLite command that will remove all unused pages and generally compact the SQLite database. You don’t need to specify a table name as the original post suggests. It has been ignored since the vacuum command came back in SQLite 2.8.1 (it was originally in 1.x and removed in 2.x, to return in 2.8.1).


So, what is really going on? The Envelope Index file dropped from 10s of megabytes down to only a couple of megabytes??? What data was deleted?

OK — here starts a bit of technical spewage regarding the details of SQLite’s internals. Not in any depth, but enough to understand what is happening.

SQLite is, as its name implies, a SQL based database storage solution. The “lite” part is that it is designed to be embedded into an application and to store data pretty much directly to/from disk, not electing to use the more common client/server two or three-tiered architecture of a traditional relational store.

Unlike most database solutions, there is no server as a part of SQLite. Internally to your application you call — or you tell Core Data to call — some SQLite functions that go off and deal with the database. These are designed to be very fast and the whole thing is designed to have minimal impact on your application. So, just as there is no random external server process, SQLite doesn’t spawn threads — worker bees that can operate independently in an app — to do various tasks.

Now, a database is designed to retrieve data very quickly. As such, the data written to disk is generally written in a highly structured fashion such that scanning and retrieving subsets of the data can be done quickly and can be optimized– or indexed– to make certain kinds of queries very fast.

Whenever new data is written into such a data store or existing data is updated, the structure of the data in the store has to be adjusted to optimally represent the data now contained to maintain query performance.

SQLite happens to use a storage structure derived from B-Trees that uses pages — equally sized chunks of the overall data file — to store the data. As the data in a database is updated, the pages are re-organized and, occasionally, an update of the database will cause a page to be orphaned. That is, a page will no longer be used but, because of the need to optimize the queries, it won’t be reused either.

So, the vacuum command effectively copies each and every in-use page in the database file to a new copy of the database file. As it does so, it omits all unused pages and it optimizes the fresh copy such that query performance will be as fast as possible.

If this is the case, why doesn’t Mail periodically perform a vacuum command?

There are a number of reasons why you would avoid it. Most importantly, it can take a really long time and there is no way to predict how long it might run or what the savings might be. Not exactly the kind of thing you want to subject the user too if the general performance profile is “good enough” (and it mostly is).

And this is a problem that really should be addressed in the database itself. SQLite should be responsible for appropriately recycling unused pages and generally optimizing the database file as it goes. By doing so, it should be able to amortize the potentially large cost of a periodic vacuum across each INSERT or UPDATE of the database.

And that is exactly what SQLite has done in more recent versions. SQLite now has an auto-vacuum feature that will cause the database to effectively stay “vacuumed” of free pages automatically and without need to periodically do a rather intrusive copy-and-fix operation.

It can be turned on or off on a per-database basis for optimization purposes. But only at database creation time. If you have a database into which data will only ever be added — a logging application, for example — you would likely turn off auto-vacuum as it will just incur overhead without ever being able to shrink the database.


There is another useful command you can use at the command line in a similar fashion. During the development of Tiger, I lived by this command as there were a couple of periods of time where the filesystem had embraced chaos when it came to writing data.

/usr/bin/sqlite3 ~/Library/Mail/Envelope\\ Index 'PRAGMA integrity_check'

'PRAGMA integrity_check' causes SQLite to walk the entire internal structure of the database and make sure everything makes sense (as much as possible). The 's at the beginning and end of the SQL stuff tell the shell (the thing in Terminal that interprets what you type, for all intents and purposes) to pass 'PRAGMA integrity_check' as a single argument to sqlite3 command, instead of breaking it into two arguments separated by a space. Now, sqlite3 could be smart enough to deal with this. But it isn’t. Such is the unix way.

SQLite’s integrity checker will look for out-of-order records, missing pages, malformed records, and/or corrupt indices. If there are none, the command emits a single line saying “ok” (sorry about the mistake before — chalk it up to an unfortunate Terminal related issue).

If you happen to execute the above command and it emits any output save for “ok”, recovering from the problem is really quite terribly easy for Mail. Just delete the Envelope Index:

rm ~/Library/Mail/Envelope\\ Index

When deleted, Tiger’s Mail will re-import all of the messages as if you just migrated from Panther. Nothing lost save for a bit of time, potentially quite a bit of time if you have a particularly large set of Mail.

Obviously, if a SQLite database contains unrecoverable data, the above would be a very bad idea. Recovering data in such a situation is very difficult.

Update: Yeah– I hate Unix. When I wrote the post, I had included the appropriate “\” escaping of all the commands and even called out why they are necessary. Apparently, either WordPress or MarsEdit ate the damned slashes. Grr. Fixed. Thanks for pointing out the error.

BoingBoing has picked it up, perpetuating the incorrect command (works, but the table name is not necessary — no big deal). The hawk wing site now has a couple of scripts that nicely automate the process.

Of course, this will work for any SQLite application that uses the built in SQLite on Tiger and does not periodically vacuum it. For example, Steve Weller has tossed together a script to update the sqlite database in Aperture’s library.

A new member of the weblogging community….

Tuesday, February 6th, 2007

Steve Jobs joined the ranks of webloggers today. Mr. Jobs has a rather high profile URL with an RSS feed that spans much more than his personal essay(s?) and an entire company’s worth of contributors.

The first post is quite the mind bomb.

Personally, I hope this is the first of many such posts.

Update: No, I don’t think that Steve’s rather poignant essay is a blog post. But, really, why not? As corporations increasingly rely upon the web to disseminate both formal press releases, success stories, and anecdotal information, where does is the line drawn between “web site” and “weblog”?

There is an URL… an RSS feed… The ‘hotnews‘ section of Apple’s site is certainly a sort of corporate weblog.

This post serves two purposes.

First, go read the essay. It is really is quite interesting.

Secondly, what is the definition of a weblog? Where is the line between corporate news collection and corporate weblog? Apple’s “hotnews” section often has non-press release content. Does that make it a weblog?

iTunes Library Backup, rsync & cover art

Wednesday, January 3rd, 2007

A while ago, I wrote about synchronizing iTunes libraries between a couple of machines using rsync while allowing for multiple locations.

I continue to use that system and it works great. Since I wrote that, iTunes has added significantly improved cover art handling. In fact, the whole cover flow thing has totally changed the way I interact with my library. Much more visual now. Has that feel of flipping through a stack of CDs to discover that hidden gem I haven’t listened to in a while.

With over 16,000 songs across several thousand CDs worth of music, I have a huge number of albums that don’t have cover art and don’t automatically resolve art from the music store. For automatic downloading of cover art, I have been using SonicSwap Boink. For the 30% of the time it mismatches cover art, I do a Google Images search to find the cover art.

Now, I have been using this command (paths modified, obviously) to backup my music library, including all metadata:

rsync -a -v --progress --block-size=220000 mastermachine:/Volumes/Music/ /Volumes/Music/

And I noticed that the synchronization process was taking a really long time recently! Much longer than would be expected given that I haven’t been ripping the rest of the CD collection recently.

It turns out that any track with updated cover art was having at least 220,000 bytes copied between machines. Given that the actual cover art is typically around 20,000 bytes and I couldn’t imagine an additional 200k of ID3 tags associated with the cover art, that order of magnitude difference in bytes changed vs. bytes transferred is a hell of a penalty!

The problem is the block-size. It is both a scanning window size for rsync and appears to also specify the minimum # of bytes transferred to describe a change in a large file.

I cranked it down to 15000…

rsync -a -v --progress --block-size=15000 mastermachine:/Volumes/Music/ /Volumes/Music/

…and the backups now go much, much faster. Of course, it’ll slow down the transfer of newly added music, but not by that much.

Update #1:

Denis suggested that turning on compression would fix the issue. To be specific, he suggested turning on compression in SSH and using rsync over SSH.

It only helps for the transmission of the iTunes database file(s). The media files, including the cover art, doesn’t compress by any noticeable amount when transmitted.

Pushing compression down from rsync into ssh may have the additional advantage of also compressing the rsync adminstrative noise, but larger block sizes are vastly more efficient when pushing media around with rsync!

So, no, compression won’t really help this problem for anything but the iTunes database files. Given their size — many many times the size of the average audio media file — I have been using compression and simply eating the slight CPU overhead related to fruitlessly compressing the already compressed media.

Hex Fiend now open source!!!

Friday, November 24th, 2006

Ridiculous Fish (Peter Ammon) has released Hex Fiend as open source (truly open — BSD style license).

Hex Fiend is an absolutely brilliant hex editor. Beyond being a brilliant hex editor, it is a really really well implemented piece of software. Damned fast. Scales beyond the capabilities of the machine. Works really well and behaves consistently.

Anyone who claims Cocoa is slow needs to use Hex Fiend; it screams.

Even if you have no need of a Hex Editor, I highly recommend checking out the source for the learning experience.

svn co http://ridiculousfish.com/svn/HexFiend/trunk/ hexfiend

There is also an awesome Wiki describing the implementation of HexFiend.

BBEdit vs. TextMate: The Editor Wars Revisited.

Thursday, November 9th, 2006

Update: This is kind of like the recent switch in basketballs used in the NBA. Go read Mark Cuban’s discussion of the new ball. Very interesting. Some players benefit immediately. Many do not.

Erik J. Barzeski asks For those TextMate users in the audience, why do you use TextMate? For those BBEdit users, why have you stuck with BBEdit?

Text editing with any great speed is a tremendously intimate bit of interaction between you and the editing software. It is inherently a keyboard driven task and a very complex task, at that, given the myriad of syntax rules and gestural editing possibilities.

As a result, any user who spends any significant amount of time with any one text editor — I’m talking years, here — will build up a set of usage patterns that employed rapidly and repeatedly throughout an editing session. Often this is called “muscle memory”, but it is really more that your brain builds up a library of “mental macros” that are applied almost subconsciously as you work with the editor.

Because of this, switching text editors is incredibly disruptive to one’s workflow and results in some awesome “religious wars”. Why? Because it is just too damned difficult to actually quantify why one editor is so much better than another.

I have written at length about this before.

So, really, it boils down to how many of the “mental macros” can be transported from one editor to another.

Me? I was an emacs user. Because Cocoa’s text editing subsystem has supported emacs like behaviors since 1990 or so, the move from emacs to TextMate was relatively natural. I lost a lot of the more complex features, but I gained a much more modern and native user interface.

An acceptable trade off.

I have tried BBEdit many times and simply can’t use it. It makes my skin crawl to try and do so. Certainly not because it is a bad piece of software or otherwise inferior, but entirely because my learned mental model of text editing is simply incompatible with BBEdit’s implementation.

As I said in the above article:

Using BBEdit literally makes me grind my teeth with the general discomfort it causes. Now, I know that’ll rub a few people wrong (if anyone reads this) so let me be completely clear: BBEdit is a brilliant product and my inability to deal with it is entirely in my head.

So, in this case, it sounds like Erik’s brain has remapped itself to view text editing within the patterns of use perpetuated by BBEdit. As a result, a move to TextMate or SubEthaEdit is going to be disconcerting. It will feel unnatural and the flaws/bugs are going to be amplified.

Aperture: The Color Tool

Wednesday, October 4th, 2006
Purple Herb Flower
Purple Herb Flower

A couple of days ago, I blogged about the release of Aperture 1.5 and how I really dig the refinements and performance boost. Using the photos at right as a [macro — hey, I’m a happy flower photographer and I’m not afraid to admit it] example, I was able to use the color tools to refine a photo in ways that I couldn’t before.

I’m not the only one digging the new stuff. Duncan — a professional photographer from which I have learned a huge amount — takes on the Aperture color controls and also offers a comparison with similar features in Adobe’s lightroom Adobe Photoshop Lightroom.

Neat stuff. Now I need to go and adjust a few photos given what I just learned…