Archive for December, 2004

I’m ashamed to say that I didn’t figure this out myself. It is, after all, consistent with the behavior across the rest of the system.

I have iTunes configured such that it does not copy songs into my library as they are added. This allows me to split the library across multiple hard drives while also conveniently having a much more frequent backup schedule for the metadata.

When adding new songs to the library via drag-n-drop, I often want the newly added songs to be consolidated into the master music repository.

This turns out to be really easy. Simply opt-drag the songs into iTunes and it will copy the songs into the library instead of just adding ‘em “in place”.

Duh. Just works.

Update

Andy Boyko asked how I have my iTunes library setup and how the above interacts with the setup.

I have two primary listening locations; home and office.

At home, I have an external 160GB drive that is the music library (i.e. iTunes is pointed to that location in its preferences). There are two partitions on it; 140GB primary and 20GB secondary. The primary partition contains all the music. The secondary partition is used for BitTorrents. Specifically, I BT a lot of live music from ETree and use the secondary partition to preprocess the tracks.

For encoding of CDs (I still have about 2/3rds of my collection to encode), the encoded songs are automatically inserted in the library. Fine. Which is good that it is fine, because I don’t think there is another way.

For addition of already encoded songs — DJ mixes and live music — I will add them directly from the BT volume or the desktop. They stay there until I have decided they are worthy. At that point, I simply hit consolidate library and delete the originals.

The office environment is a little bit different. I have a 48GB 2.5″ drive that I don’t trust for historical reasons. I wrote a little AppleScript that copies 40GB of random tunes from my home listening library onto the removable drive. It then becomes the primary source of music in the office environment. iTunes keeps all the library metadata on the machine’s local drive.

When new tracks are added to that machine outside of what is on the external drive, I always keep the new tracks separate so that I can easily keep track of them until they are integrated into the home library.

This includes purchased music.

I wish all of this were easier and that there was a way to share metadata between the two environments. But, frankly, what I have now works well enough for the time being.

The valley fair mall (Santa Clara, CA) now has a Lego Store.

Beyond having all of the random sets and other Lego paraphernalia, they have an entire wall covered in little bins. Each bin contains one type of brick in a single color. Not just your normal bricks, but specialty bricks like hinges, various wheels, doors, windows, flat pieces, bricks with the technic holes, and the like.

You grab either a small or a large plastic cup and proceed to stuff as many pieces of whatever variety you want into the cup. As long as the cap fits on top snugly, you can purchase a small cup of parts for $7 and a large cup for $13.

They also have little plastic bags that are totally full of gears, axels and joints, or the other kind of girders with holes for $12 or $9, depending on the bag.

Excellent. As a young Lego builder, I often remember being irritated that I was missing that one last gear or axel or holed piece, whatever, to complete my current masterpiece. A store like that would have likely consumed most of my allowance.

You can see a picture of the bins on this site (found via Google). 3rd picture down on the left. Cute kids, btw. Another accurate write up here. I like the “bins of doom” description.

TinyP2P claims to be the world’s smallest p2p sharing application. It is implemented as 15 lines of highly obfuscated Python in an attempt to make the point that trying to suppress P2P technologies through lawsuits is futile.

I appreciate their attempt at making a point. Yet, I’m not really that impressed. 15 lines of highly obfuscated Python seems is just another useless geek wank cry for attention.

I would be far more impressed if it were implemented as less than a page of very clearly written Python. Not only would it still be impressively tiny, but it would provide a seed from which many new variants on P2P might arise. Instead of being a finger raised in defiance, it would be an opportunity to teach.

Speaking of futile attempts to squelch P2P technology. The MPAA is now pursuing BitTorrent as the Great Evil That Will Destroy Their Industry. I’m still of the camp that if Hollywood made movies that weren’t utter crap, did not bombard us with 30 minutes of ads prior to showing said crap, and didn’t charge us $9 for the “privilege” of watching the ads, then the crap, they wouldn’t be in such supposed dire straits.

The BitTorrent thing is certainly grabbing headlines.

I find it intensely irritating that BitTorrent is being called a “network” that Hollywood is trying to shut down. It isn’t a damned network already! Most articles actually do try– typically poorly– to describe the whole tracker concept and that the legal efforts are aimed at the tracker admins/hosts.

I have previously written on the topic of BitTorrent and how the law suits might play out. In that missive, I mentioned that it is impossible to prove that any one BT client uploaded an entire copy of any given piece of IP and it is actually quite provable that any one client could not possibly provide a whole copy of a piece of IP to any one other client. That may or may not work.

Regardless, there is likely something much more important at stake.

In particular, ISPs and the like have long been able to effectively defend themselves against the content passing through their networks, mailing lists, NNTP servers, web servers, proxies and the like by invoking a “common carrier” style defense. That is, the ISP actively chooses not to monitor or censor the content passing through its systems.

The trackers could employ a similar defense. Many trackers will accept any random torrents without censorship or filtration. That combined with the fact that trackers generally never host the content being distributed means that the tracker– and the tracker’s owner– never has the IP in their possession nor do they require any active awareness of what might be distributed by their site.

Now, in the email world, a mailing list operator that does not moderate content has significantly less liability than one that does. By choosing to moderate content, the operator is taking responsibility for each post that is broadcast to the list. By choosing not to moderate, the list is effectively a common carrier and the individual poster is the one responsible for the contents of their post. This has been used successfully to prevent ISPs from being liable for the many nasty things that pass through NNTP feeds, among other things.

Could this same defense be applied to trackers? It would appear to be a very similar situation. Obviously, those trackers that called themselves ‘warez depozt’ and made outright claims to have the largest collection of blah-blah-blah-illegal-blah have other problems.

If this kind of defense is successfully applied, then it will be very difficult for the MPAA or anyone else to continue to shut down torrent trackers through legal means.

If it is applied and fails in court, it would set a very ugly precedent. In particular, it would significantly further erode the “common carrier” defense. It could become the foundation for law suits that would effectively require ISPs, NSPs, hosting providers, etc.. to very actively censor the content that passes through and is served from their systems. This would make it prohibitively expensive to publish anything on the Internet and would also cause the cost of simply passing traffic through your system to skyrocket.

Or, to look at it a slightly different way, it would make an ideal environment for corporations with very deep pockets to quite effectively squelch independent content producers while ensuring that their content is widely available on IP in a manner that is thoroughly protected by the courts.

This is exactly the situation that the MPAA, RIAA, and various participating content providers have been gunning for every since they finally realized that the Internet is here to stay. TCP/IP is an incredibly cheap means of content distribution with the distinct advantage that the customer doesn’t, by default, receive a copy of your content that can be consumed repeatedly, forever, without pay-per-play. Yet, the current Internet environment, including the laws that protect it, make it extremely difficult to control the content to the degree said organizations would like.

By no means do I think that the legal pursuit of BitTorrent will be the case that makes or breaks the Internet. It is just one small battle in the ongoing war for control of this relatively new IP distribution mechanism we call the Internet.

I will be watching this one closely for two reasons. First, I think it is a very interesting grey area between P2P style services and services that have been protected so far by fairly strong laws. Secondly, I have played an active, albeit very minor, role in the development of BitTorrent and, as a result, have a personal interest in the technology as well as having used said technology to distribute bits as a part of my professional career.

Last night, we rented a shuttle bus who picked up a bunch of our friends at their houses and drove us all up to the city. The shuttle bus was to ensure that there were no designated drivers in the bunch.

Specifically, we went to Tommy’s Mexican Restaurant. We had an absolutely stellar time thanks to Julio and the hardest working bar staff on the planet.

We ate very well and drank some of the finest Tequila on the planet. Julio gave a great lecture on how Tequila is made, the history, and the qualities inherent to the various different products.

We added about 5 or 6 more people to the blue agave club and all of us received a few punches on our cards. I’m now about 80% of the way to my PHD and, after passing a written test, I’ll be heading to Mexico with Julio and other PHD holders for what will be the ultimate tour of the regions of Mexico from which Tequila is sourced.

Thanks to all of my friends that made the trek. I hope you all had as good of time as I did.

And a huge thank you to Julio and the crew. Couldn’t have been a better evening. It was greatly appreciated.

LineMan.net is running a month of articles about various programming languages. It is a series of short articles that explains the history of the subject language, provides a synopsis of strengths/weaknesses, and provides some sample code.

So far, it includes 6 languages of which 3 are dialects of C. As BASH is an included “language”, it is clear that this language survey will contain a broad range of languages.

Interesting stuff.

We received yet another AOL CD in the mail today. They are no longer shipping out the metal tins (which I actually kinda liked once they were empty of all the garbage).

Instead, AOL is now shipping out relatively nice DVD style cases that contains not one, but two pieces of media. The plastic is a little bit brittle — you get what you pay for.

The first piece of media is the standard AOL installation CD. The second disc is a CDRW. That’s right, a CDRW that likely cost an order of magnitude more than a CDR (4 cents instead of .4 in the quantity AOL ships).

Why a piece of blank media?

Because AOL also includes 5 free iTunes music store songs. That’s pretty cool.

The litle card suggests that you download your free songs, then burn them to the CDRW.

That is almost a really good idea. Problem is, most consumer CD players purchased prior to the last couple of years cannot play CDRW discs. Many current models can’t either.

Oops.

PyObjC now has the ability to build NSBundles that can be dynamically loaded by any Objective-C application while the bundle is entirely implemented in Python. In other words, you can now use Python to implement plugins for any app that supports Objective-C plugins (NSBundle).

The PyObjC Subversion repository contains three examples; a screen saver, a preference pane, and an Interface Builder palette.

Most of the innovation comes from py2app. It is used to build the dynamically laodable bits within the NSBundle that then bootstraps the loading of the Python based class implementations.

py2app is also used within the build script of PyObjC. In particular, you can now execute python setup.py bdist_mpkg –open to build an Installer package containing PyObjC, py2app, the examples, and the Xcode templates. Once installed, invoking bdist_mpkg within any Python project that has a setup.py will build an Installer package containing all the bits necessary to install that package.

If you want to build the latest PyObjC, you will need DocUtils. See below.

Bob Ippolito implemented all of the magic. Thanks, Bob — this stuff is incredible. Awesome stuff. A little over a decade since its inception, PyObjC is still a vibrant project that is growing in exciting new directions while still remaining focused on its core goals.

The pieces involved:

PyObjC bridges Objective-C to Python transparently enough that you can build full featured Cocoa Apps. PyObjC is rapidly approaching a 1.2 release and there are many significant refinements in this release.

py2app builds double-clickable Mac OS X .apps, including all dependencies. Actually, it does considerably more than that. It will also create installer packages from any Python based project that uses distutils (setup.py). py2app can also rewrite mach-o files.

DocUtils is a reStructuredText rendered. reStructuredText is a very simple ASCII format that compiles into nicely formatted HTML, PDF, Open Office, or other kinds of documents.

It will be my birthday in about 20 minutes. Technically, it was my birthday an hour and 40 minutes ago given that I was born in the Central Time Zone and now live in the Pacific Time Zone.

Woot.

Word Documents can be like data motels; data goes in, but you can’t get it out in anything but a printed form.

For years, Christine (my wife) has maintained the family address book in a set of Microsoft Word documents configured to print labels. This worked across multiple operating systems and multiple versions of Word.

It is also a horribly inflexible solution that made it impossible to intelligently manage groups of addresses or tie additional information to people. But it worked back in the days prior to Mac OS X when our computing environment was a hodge podge of PCs and NeXT machines running cobbled together non-production operating systems.

In that period of time, Mac OS X was released and Address Book has matured quite nicely. In particular, Address Book has decent group management capabilities, is nicely integrated with other applications on the system, and can quite effectively print mailing labels.

So, we wanted to move the data out of Microsoft Word and into Address Book. Address Book can import vCard formatted address data. So, the problem is really to export from Word Labels to vCard.

Not so easy. The first problem is that Microsoft Word’s labels are not a contact manager. Labels don’t have any real structure or fields. To compound the problem, there isn’t any kind of a useful export format for Word’s labels. The text only formats have relatively randomized sets of newlines.

I chose to export to HTML because that, at least, yields structured documents that are very easy to parse.

To make matters more complex, there is a lot of noise in the original label data. Things like “The Smith Family”, multiline names, some international addresses, and other weirdness.

The end result is a python script that can parse Word HTML and turn it into vCards. It is terribly specific to the data found in our label documents, containing a number of special cases and the like for cleaning up the data on the fly.

Fortunately, Address Book is really good about consuming poorly deconstructed address data. If you just put the full name into the “FN:” (full name) field and the entire address into the “ADR:” field without decomposing anything, Address Book will happily format labels correctly.

But the result is also something that isn’t really complete. Names don’t sort correctly and you can’t sort by zip or state, for example.

The script goes a bit further and uses a series of regular expressions and some really nasty hardwired logic to try and detect the city/state/zip and break it out into the right vCard fields. The expressions are written such that anything that fails to match exactly will fallback to the less specific rendering of the vCard.

I shoved the whole thing into my random hacques subversion repository and slightly more technical details can be found in a README

Let me re-emphasize the last point from readme: The script is very poorly written. It is a one off and now that the data is freed from the confines of a Word document, I’ll hopefully never have to run it again.

Obviously, backing up tracks purchased from iTunes Music Store on a regular basis is a very good idea (along with anything else you really don’t want to lose).

I use an external hard drive for backups. I also like to burn all of my purchased music to a piece of optical media whenever I purchase new tracks (about once a month).

Now, I don’t really need to backup all of my ripped CDs. The CDs are the backup. Given that Apple has consistent history of releasing better quality audio encoders fairly regularly, I’ll need to re-encode my CD collection every couple of years anyway.

The purchased music is interleaved with non-purchased music. To back it up, I use rsync to copy the purchased music and only the purchased music to the external hard drive. From there, it is trivial to burn the copied files to CD or DVD.

The rsync command to synchronize the purchased music and only the purchased music is a bit tricky because rsync’s include/exclude options are, uh, subtle.

Of note (yes, that is all one single command):

rsync --block-size 32000 -e ssh \
    –progress –partial -a –include ‘*/’ –include ‘*.m4p’ \
    –exclude ‘*’ \
    ‘orange.local.:/Volumes/LargeData/Master\ Music\ Library/’ \
    /Volumes/ExterData/MusicBackup/Purchased\ Music

• It works on any system that can resolve the name ‘orange.local.’ (the name of the machine with the music library) including on that machine itself. rsync is that smart.
• The –block-size option increases the CRC block size to 32,000 bytes. This is several orders of magnitude larger than the default. Larger block sizes are much more efficient for large files.
• –progress shows percentage transferred for each file
• –partial will not delete files on the local end that aren’t fully transferred. Very useful if you happen to be backing up across a slow or intermittent link or you simply want to hit ctrl-c in the middle and finish later.
• -a is a catch-all option that enables a bunch of other options that are fairly optimal for archival purposes
• The order of the –include and –exclude options are critical. The first includes “all directories”, the second grabs all purchased music, and the third excludes everything not selected by the first two. Remove the first –include and rsync excludes the top level directory and does not recurse at all.
• Annoyance: you’ll end up with empty folders for all of the non-purchased content. Doesn’t bother me that much, but I wish rsync had a “don’t create empty folders” option.