Kinda like here for which I posted the same conceptually concrete but technically vague pattern that I have posted for the last decade+.

Which led to this question.

OK — enough is enough. Here is a working example.Specifically, you’ll want to grab the source:

svn co http://svn.red-bean.com/bbum/trunk/PyObjC/CallPythonFromObjectiveC/

You should be able to open the Xcode Project contained within, build it, and then (on Snow Leopard, anyway):

python call-python-from-objc.py /path/to/built/AbstractClassDefinition.bundle

And it’ll spew something like:

creating Bob
returning new instance <Concrete 4346092816 named Bob>
2009-11-20 22:44:49.485 Python[41780:d07] Created instance: <Concrete 4346092816 named Bob>
factor will be 1000000000
2009-11-20 22:44:49.486 Python[41780:d07] Set the fudge factor. Gonna do it up, now.
doing it returning 927.3
Did it! 1000000000 927.299988

The Xcode project produces an NSBundle that contains two unrelated pieces of source; an Abstract Class that is subclassed by python to provide the binding from Objective-C into Python and a class that provides an Objective-C method invoked by the python script to demonstrate calling from Obj-C into Python.

The Abstract Class is the first piece of the puzzle. It is declared as:

@interface AbstractClass : NSObject {
    NSString *name;
}
@property(copy) NSString *name;
@property unsigned long long fudgeFactor;
+ namedInstance: (NSString *) aName;
- (float) doItToIt: (BOOL) fastFlag;
@end

That is, a pretty straightforward (and totally contrived) Objective-C class. The Abstract part is that this class is not designed to be directly instantiated. Instead, it is too be subclassed and the subclass is the bit that is instantiated, as is indicated by the implementation:

void SubclassResponsibility(id classOrInstance, SEL _cmd) {
    NSString *reason = [NSString stringWithFormat: @"Must subclass %s and override the method %s.",
       object_getClassName(classOrInstance), sel_getName(_cmd)];
    @throw [NSException exceptionWithName: @"SubclassResponsibility"
       reason: reason
       userInfo: nil];
}
@implementation AbstractClass
@synthesize name;
- (void) setFudgeFactor:(unsigned long long) aShort
{
    SubclassResponsibility(self, _cmd);
}
- (unsigned long long) fudgeFactor
{
    SubclassResponsibility(self, _cmd);
    return 0;
}
+ namedInstance: (NSString *) aName
{
    SubclassResponsibility(self, _cmd);
    return nil; // not reaached
}
- (float) doItToIt: (BOOL) fastFlag;
{
    SubclassResponsibility(self, _cmd);
    return 0.0; // not reached
}
@end

Note that the Subclass Responsibility pattern used to be quantified in the Pre-OpenStep version of the what-is-now-called-Cocoa APIs. Object — the then NSObject — actually declared a -subclassResponsibility:(SEL) method!

The call-python-from-objc.py does the following:

1. Loads the bundle containing the implementation of AbstractClass:

bundle = Foundation.NSBundle.bundleWithPath_(bundlePath)
if not bundle.principalClass():
    print "%s: failed to load bundle." % sys.argv[0]

2. Creates a subclass of AbstractClass:

AbstractClass = objc.lookUpClass("AbstractClass")
class ConcreteClass(AbstractClass):
    _fudgeFactor = 0
    # factory method
    @classmethod
    def namedInstance_(self, aName):
        print "creating %s" % aName
        newInstance = ConcreteClass.new()
        newInstance.setName_(aName)
        print "returning new instance %s" % newInstance
        return newInstance
    # fudgeFactor property stored in Python subclass
    def fudgeFactor(self):
        return self._fudgeFactor
    def setFudgeFactor_(self, aFactor):
        print "factor will be %s" % aFactor
        self._fudgeFactor = aFactor
    # the method that does it all, so to speak
    def doItToIt_(self, fastFlag):
        if fastFlag:
            returnValue = 516.295
        else:
            returnValue = 927.3
        print "doing it returning %s" % returnValue
        return returnValue
    # Describe instances more pythonically
    def description(self):
        return "" % (id(self), self.name())

3. Invoke a random bit of Python code that calls an Objective-C method that calls back into Python

objectiveCCode = objc.lookUpClass("ObjectiveCCallingPython")
print objectiveCCode.callSomePython()

More specifically, the class ObjectiveCCallingPython exists solely as a demonstration of calling from Objective-C into Python. It happens to be implemented in the same bundle as AbstractClass, but it doesn’t need to be.

The callSomePython method looks like:

+ (NSString *) callSomePython;
{
    Class concreteClass = NSClassFromString(@"ConcreteClass");
    AbstractClass *concreteInstance = [concreteClass namedInstance: @"Bob"];
    NSLog(@"Created instance: %@", concreteInstance);
    concreteInstance.fudgeFactor = 1000000000;
    NSLog(@"Set the fudge factor. Gonna do it up, now.");
    float value = [concreteInstance doItToIt: NO];
    return [NSString stringWithFormat: @"Did it! %llu %f", concreteInstance.fudgeFactor, value];
}

Straightforward; grab the concrete subclass defined in Python, instantiate, configure, then tell it to do something.

The key is use Objective-C as the bridge between the compiled language (C/C++/Objective-C) and the interpreted language (this same pattern will work for other bridges). The easiest way to do that is to define an abstract class in Objective-C that is then subclasses in the interpreted language. From the compiled code, the abstract class acts as the source for all of the compiled type information the compiler needs. From the interpreted code, the abstract class’s stub method implementations provide the runtime metadata necessary to allow the interpreter to correctly convert between compiled and interpreted data types.

Or, a specific example, the fudgeFactor property has stub method implementations that enforce subclass responsibility. Beyond that, the presence of said methods ensure that the Python implemented method implementations contain the correct metadata such that the doesn’t-fit-in-an-id type of the argument to the setter and return value of the getter is correctly handled.

There are about a bazillion different variations possible, including eliminating the Abstract class. Ultimately, the key to success is ensuring that both sides of the bridge have all metadata necessary to work correctly. For the C/ObjC/C++ compiler, it needs the types of the arguments to correctly generate the call sites. For the interpreted language, it needs metadata describing the argumentation of the methods, whether provided automatically (in this case) or dynamically (see @objc.signature).

So, I moved ‘em over.

You can find them in the PyObjC subversion repository.

These aren’t just templates, though. See the README.txt.

As it implies, the project-tool.py script is used to effectively convert between Xcode templates and buildable projects. That is, you can create a project that builds/runs like a regular Xcode project, and then easily turn it into a template with the invocation of a simple command line like this:

project-tool.py -k -v --template Cocoa-Python\ Document-based\ Application/CocoaDocApp.xcodeproj/TemplateInfo.plist \
    Cocoa-Python\ Document-based\ Application/ \
    ~/Library/Application\ Support/Developer/Shared/Xcode/Project\ Templates/AA\ Testing/Cocoa-Python\ Document-based\ Application

Or, specifically:

./project-tool --help
Usage: project-tool.py [options]  

    Copies tree of templates or projects from  to .
    Before copying, it cleans up  by removing various bits of garbage.
    After copying, it transforms  by replacing strings with their Xcode
    template counterparts.

    The reverse flag can be used to reverse this process; turning an Xcode
    template into a working project.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         verbose
  -k, --kill-dest       erase  (no warning)
  -r, --reverse         reverse transformation (template -> editable project)
  -w, --working         try to make destination into a working project
  -n, --nib             rewrite NIB files to 10.5 text-only format
  -t TEMPLATEFILE, --template=TEMPLATEFILE
                        path to TemplateInfo.plist that should be used during
                        conversion

Nothing remotely Python specific about it. And, as some have noticed, the PyObjC Cocoa templates leverage the new Interface Builder file format (xib) such that all kinds of substitutions happen in the templated interfaces, too. As in: No more MyDocument ever again.

You can grab an example of the Python and Ruby bridges working together in a Cocoa application from either this downloadable zip or from this Subversion repository (in case something actually changes).

It works. Sort of. Ironically, this likely would have worked better under the pre-BridgeSupport versions of RubyCocoa and PyObjC.

Specifically, RubyCocoa and PyObjC both assume that nothing else might have loaded the dynamic libraries that are automatically generated by the gen_bridge_metadata script. So, either bridge will quite happily attempt to load /System/Library/Frameworks/Foundation.framework/Resources/BridgeSupport/Foundation.dylib and then barf mightily when the other bridge has already loaded the same dylib.

The example is rife with silliness related to catching the resulting exceptions and ignoring them. Worse, the fallout is such that from Foundation import * doesn’t actually cause the Objective-C classes to be defined within the importing module.

There is a back door — objc.lookUpClass() — but this is yet further evidence that, at this time, mixing these two languages in a single Cocoa application is not anything more than a silly hacque (as the SVN repository subdir indicates).

What does it do?

Not much, really.

• Start with RubyCocoa Application Template (because I can deal with brokeness in Python, I wanted to start with something working in Ruby)
• NSApp Delegate written in Obj-C
• Finish loading hooks used to bootstrap PyObjC/Python (with gross exception ignoring goofiness related to the dylib)
• Bind a table view to an array of dicts where each dict has the key “name” leading to a string value bound through array controller.
• Array controller bound through app delegate method.
• App delegate returns array of dictionaries by calling python based NSObject subclass.
• Python based NSObject subclass composes an array of dictionaries (all python) from a combinatio of Python strings and Ruby strings by calling an instance of a Ruby based subclass of NSObject.

Very little code. Lots of moving parts. Some gears grinding. Maybe even a gear tooth or four missing. Enjoy.

It can be found at PyObjC 2.0 trunk (http://svn.red-bean.com/pyobjc/branches/pyobjc2).

The NEWS.txt file is extremely illuminating as there were tons and tons of changes and bug fixes in 2.0.

It cannot be emphasized enough; this is a huge release for PyObjC. As big as when PyObjC first gained subclassing (which wasn’t until about 7 years into its 14 year history). Go read the NEWS file; both major enties on PyObjC 2.0.

Note that libffi on Leopard is fully 64 bit and, thus, PyObjC should be mostly ready for 64 bit, though it was neither built for, nor tested with, 64 bit.

Both Ronald (who did all the work) and I (sort of) had posted previews of PyObjC 2.0. Now the full magnitude of this release can be publicly discussed.

In particular, Ronald’s post gives an excellent overview of the new features and capabilities of PyObjC.

There is a theme: “greatly tighten the integration with Mac OS X in a more consistent and less special-cased fashion”.

Read on for details:

To that particular end, PyObjC 2.0 — like recent releases of RubyCocoa — embraces the metadata generated by BridgeSupport. BridgeSupport allows a machine-readable description of APIs — be they C or Objective-C — in XML, with support for 32 vs. 64 bit differences. PyObjC and RubyCocoa use the BridgeSupport metadata o make calls from the interpreter into the native ABI on the system. Succinctly; BridgeSupport allows for data driven calls through to APIs that were previously special cased either per call site or per bridge implementation.

BridgeSupport is included with Leopard, including documentation. See the gen_bridge_metadata and BridgeSupport man pages. As well, a full-text search in Xcode’s documentation viewer reveals another chunk of documentation.

At the other end, Leopard also includes a vastly improved version of libffi. Beyond a number of bug fixes and actually having a man page now, libffi also correctly supports both 64 bit ABIs available in Leopard; ppc64 and x86_64.

Specifically, PyObjC and RubyCocoa use the BridgeSupport generated metadata to construct native calls via the FFI API.

Leopard provides metadata for most of the frameworks and CF-or-higher level APIs on the system. Not all; some APIs can’t be expressed via the metadata yet (C++, for example).

The list is quite long:

• libSystem
• AddressBook
• AppKit
• AppleScriptKit
• ApplicationServices
• ApplicationServices/CoreGraphics
• ApplicationServices/CoreText
• ApplicationServices/ImageIO
• ApplicationServices/SpeechSynthesis
• Automator
• CalendarStore
• SpeechRecognition
• Cocoa
• Collaboration
• CoreAudio
• CoreAudioKit
• CoreData
• CoreFoundation
• CarbonCore
• DictionaryServices
• CoreVideo
• DVDPlayback
• DirectoryService
• DiscRecording
• DiscRecordingUI
• ExceptionHandling
• Foundation
• ICADevices
• IOBluetooth
• IOBluetoothUI
• InputMethodKit
• InstallerPlugins
• InstantMessage
• JavaScriptCore
• LatentSemanticMapping
• OSAKit
• OpenGL
• PreferencePanes
• PubSub
• QTKit
• ImageKit
• PDFKit
• QuartzComposer
• Quartz
• QuartzCore
• ScreenSaver
• ScriptingBridge
• SecurityFoundation
• SecurityInterface
• SyncServices
• SystemConfiguration
• WebKit
• XgridFoundation
• WhitePages

Not all of these APIs are available directly in PyObjC. Yet. Some require a touch of glue. If “import FrameWork” doesn’t work, glue will be needed. More on the glue in a later post.

Both the metadata listed above and the tools to generate the metadata are provided without requiring the dev tools to be installed. Dev tools are necessary if metadata generation also creates a dylib that contains any bits — static inline functions, for example — that cannot be expressed purely in XML.

And, of course, it is quite straightforward to generate bridge metadata for your own Objective-C and C APIs. It is just a matter of running the gen_bridge_metadata utility against your APIs. Exceptions in the form of XML that overrides the automatically generated XML can be provided, too.

In essence, Apple is taking a bit of a different approach to embracing and supporting scripting languages to develop native applications.

Instead of trying to crowbar the scripting language’s syntax on top of whatever VM-du-jour happens to be running on the platform, BridgeSupport — and, by integration, Leopard — provides all of the metadata necessary to allow the native scripting language to be used to create native applications through a bridge between the two. Given that BridgeSupport works just fine on Tiger, this isn’t something limited to Leopard.

The metadata is intended to be a resource for use beyond bridging. Most frameworks on the system provide two chunks of XML BridgeSupport metadata; succinct and full. The succinct version contains all of the metadata not provided by the Objective-C runtime (which provides about 80% of what is necessary to do full fidelity calls in / out of Objective-C via libffi). The full version contains just that, the full metadata required to describe the APIs of the framework, including all the bits that could be gleaned at runtime.

Already, the BridgeSupport API metadata is used for more than just bridging languages. For example, Xcode offers Objective-C method name completion in standalone Python and Ruby source files — no project necessary — by leveraging the BridgeSupport metadata!

I am looking forward to seeing the imaginative ways with which developers leverage the BridgeSupport metadata. Certainly, there is more work to be done, but it should provide a very strong foundation for innovation as it is now!

Once PyObjC 2.0 move over to the public repository, I fully expect that development on 2.0 will continue beyond what ships in Leopard. As a consumer of PyObjC, it will be up to you to decide whether you want to stick with what is in Leopard or track PyObjC development.

For most folks, what is in Leopard will be just fine — it works great for building Cocoa applications.

Ronald posted a very nice overview of the tasty goodness in PyObjC 2.0. He has been very very busy.

Yes, it fully leverages the metadata generated by the BridgeSupport project. Via BridgeSupport, PyObjC — and RubyCocoa, and anything else leveraging BridgeSupport — has enough metadata to bridge to just about any C or Objective-C API. The goal of BridgeSupport is to provide a machine readable description of system APIs with full fidelity; something that the Objective-C runtime metadata does not provide (including, for example, information about straight C functions).

Ronald: Thank you. 12 years is a long time for any open source project to remain active and the last year, largely through Ronald’s contributions, has proven to be one of the most vibrant years of development in the project’s history!

Hopefully, that won’t be the pattern this time around.

Ronald Oussoren — of PyObjC and Universal Python fame (amongst other things) — is in town and will be joining us for Korean food and Bowling this evening. Korean food starts at 7:30pm at Fusion Korea next to Homestead Lanes (though we don’t bowl there — long story).

Map of Fusion Korea:
20956 Homestead Rd Ste A1
Cupertino, CA 95014-0310

They have done that and more. A lot more. In the process, many other aspects of Python on Mac OS X have been fixed or optimized. Furthermore, the Python build now uses SDKs as a part of the build. This lays the framework for being able to do development on one version of the OS while targeting older versions with Python. Very nice.

This is one of the areas of the world of universal binaries that few people really understand.

Building a correct universal binary of most open source projects is exceedingly hard.

Simply passing CFLAGS=-arch i386 -arch ppc will generally not yield a correct binary. Worse, if the project uses autoconf, you can rest assured that the result of ./configure ; sudo make install will only yield a correct result for the architecture the build is created on. While autoconf can support cross-compilation, very few developers using autoconf will jump through the hoops necessary to do so.

And that is really what this boils down to. Cross compilation is hard to get right. Xcode makes building universal binaries easy because the entire high level Mac OS X development model and all of the predecessor technology has been focused on cross-compilation for well over a decade.

If you are planning on developing or maintaining a body of source targeted to Mac OS X that cannot use the Xcode native build system — and there are certainly many reasons to do so — I would highly recommend that you go have a close look at Ronald’s and Bob’s work.

In these situations, a debugger does little good because there is little clue as to where things went awry. What I really needed was a way to trace execution.

It turns out that Python has excellent hooks for doing exactly that.

Add the following to any random hunk o’ code:

import sys
import linecache
import inspect

def traceit(frame, event, arg):
    if event == 'line':
        lineno = frame.f_lineno
        if '__file__' in frame.f_globals:
            filename = frame.f_globals['__file__']
            if (filename.endswith('.pyc') or
                filename.endswith('.pyo')):
                filename = filename[:-1]
            name = frame.f_globals['__name__']
            line = linecache.getline(filename, lineno)
        else:
            name = '[unknown]'
            try:
                src = inspect.getsourcelines(frame)
                line = src[lineno]
            except IOError:
                line = 'Unknown code named [%s].  VM instruction #%d' % \
                    (frame.f_code.co_name, frame.f_lasti)
        print '%s:%s: %s' % (name, lineno, line.rstrip())
    return traceit

Then, to turn it on, call sys.settrace(). You can easily toggle the trace functionality, thus reducing the tracing to a very limited portion of the overall execution. This prevents one from drowning in data.

sys.settrace(traceit)

Note that the function has a frame object as an argument. That frame object allows for extremely deep introspection of the execution state, including variable state and just about anything else you would like to find out about the underlying virtual machine.

The original code was cribbed from the dalke scientific site. That article has an awesome explanation of the details of tracing. I just added some error checking and fallback code in the case where the original file/source cannot be determined. This enables the tracing functionality to work seamlessly within the context of Twisted.

It has already saved me a ton of time and I’m about to throw this permanently into my local Python library.

The demo concluded with a “show all [most, really] the tricks at once” sequence.

• How py2app, a part of PyObjC, will automatically create an installer package from a standard Python package
• Injecting a live Python interpreter into a running application
• Introspecting the application
• Interacting with Core Data, including dealing with ** style parameters
• Modifying a managed object and having KVO automatically fire on assignment

First, grab the top-of-tree source from the pyobjc repository:

svn co http://svn.red-bean.com/pyobjc/trunk/pyobjc/

Alternatively, you can mount that URL in the finder (click cancel on all the bloody attempts to write .DS_Store files) and cp -r the source to somewhere from the Terminal.

Then:

cd pyobjc
python setup.py bdist_mpkg --open

That will build PyObjC and open an Installer package that contains the runtime, documentation, examples, and Xcode templates.

Next, grab the OutlineEdit examples from /Developer/Examples/CoreData/ and run it.

Now, from the PyObjC source directory:

cd Examples/Inject/InjectInterpreter
ps aux | grep OutlineEdit
... note the PID ...
python test.py PID

The InjectInterpreter will be built and, shortly, you will see a Window pop up in OutlineEdit containing a Python interpreter.

Click on OutlineEdit and add some new Notes in the Untitled document window. Quantity and structure don’t matter.

Now, click on the interpreter window and do:

from AppKit import *
from objc import *

At this point, we now have the PyObjC bridge imported into the Python interpreter that is running within the Main Event Loop of OutlineEdit. With that in place, we can now interact with the Cocoa infrastructure to interrogate and manipulate the application. Since this is a Core Data application, we can — of course — interact with all the Core Data goodness within the document.

First, we need to get a hold of all that Core Data goodness:

d = NSDocumentController.sharedDocumentController().documents()[0]
mom = d.managedObjectModel()
moc = d.managedObjectContext()
noteEntity = mom.entitiesByName()['Note']
fr = NSFetchRequest.new()
fr.setEntity_(noteEntity)

Now, grab one of the Note objedcts:

notes, error = moc.executeFetchRequest_error_(fr)
note = notes[0]

And manipulate:

note.contents = u'Hello, world!'

Note that as soon as you hit return, the UI updates. This happens because PyObjC is smart enough to trigger the KVO/KVC notifications upon assignment to an attribute of an NSObject subclass.

And, of course, if you screw up the state of the document too badly:

moc.undo()