Mac Daddy World

in which we make the world’s first useful video phone

---

Update: Our demo won first place for Iron Coder Live this afternoon! Thanks for your votes everyone!

Update 2: We’ve added a link to source code at the bottom of the post. We also made a quick video demo in Starbucks today.

---

This weekend’s C-4 developer conference features “Iron Coder Live”, a contest in the same vein as MacHack. The event encourages conference attendees to develop creative “hacks”, written in within a short timeframe. This year’s theme, of course, is iPhone.

This was a great excuse to buy another iPhone, install the iPhone toolchain and waste some time! Before we knew it, the iCal told us it was Thursday and we were putting the finishing touches on iPhone video conferencing.

Camera? Oh really?

Obviously we had to do something with the iPhone’s camera. Doing crazy things with cameras is a full time job for us! Our contest entry captures video from the iPhone’s camera, compresses it, and sends it to a web server, where it’s relayed to another iPhone, and vice-versa, resulting in a nice two-way video conference. Need audio too? That’s not our department but simply make a phone call to the other person’s iPhone and put them on speaker phone. Then fire up our program and you’re in business. (Yes, the iPhone makes phone calls apparently.)

Wait for the Clever Bit

Now you’re probably thinking, how do you do a video conference when the iPhone’s camera is pointed in the wrong direction? The iPhone, like every other smartphone that I’ve seen, has a camera mounted on the back of the device, causing most people to dismiss the possibility of video conferencing right out of the gate.

Phuckleberry

For those of you familiar with the Ecamm product line, you may remember that we sell the Huckleberry mirror, a periscope for your MacBook or MacBook Pro. While the Huckleberry II for MacBook Pro doesn’t quite fit properly on an iPhone, that’s nothing some wire cutters and imagination couldn’t fix. Here are some pictures of our homemade iPhone-Huckleberry-stand running a two-way video conference over wifi. As you can see, the camera image is reflected off two acrylic mirrors and re-oriented in software.

(The orientation changes automatically of course.)

Darn

Ok, whoops, we meant to create something useless and fun. Surprisingly, this actually works very well, and might be considered downright useful in some scenarios. I hope that doesn’t disqualify us from the hack contest…

The iPhone Toolchain

The first step after jailbreaking our iPhones (which sounds dangerous but is actually pretty benign), was downloading and building the iPhone toolchain. The toolchain is clearly the result of hundreds of hours of work by a devoted group of iPhone dev devotees. What have they built? In a Nutshell, devs can now write native iPhone apps in the same integrated development environment we use to write Mac apps, in the same programming lanaguage, using almost the same set of APIs. It’s pretty overwhelming to think of the possibilites this opens up. It’s like a tiny little Mac/phone just like we were all promised! Thanks to the folks at iPhone Dev Wiki, and the binutils project, and these two pages for getting us up and running within hours.

Source Code

We’ve decided to post the source code for our contest entry here on the blog. Please keep in mind, this is most likely not going to be useful to you for anything other than illustrating the mechanisms that we used to accomplish our hack. The current app does not support more than two users, and will not just compile and magically work, as you will also need a server and a relay script to send the imagery between iPhones. (We used Perl for that.)

Download here: squidge_source.zip (84K)

Sample App

Next on our list, we’ll be creating an app that will actually let ambitious iPhone users try out the proof of concept. This primarily involves adding a user interface. Check back for this.

Following a continuing trend of adding new things without telling anyone, the recent QuickTime 7.2 update includes more than just bug fixes.
We’ve confirmed that the QTKit Capture functionality previously billed as a feature of Leopard is included with the update. (See the section on QuickTime Improvements.)

The fun new functionality is all there after updating to 7.2, but there are no new header files (no public API yet). Also, the functionality is now used by QuickTime Player Pro. However, you probably won’t notice much of a change except a little progress indicator in the Recording section of the prefs when it’s scanning for cameras, and hopefully a performance improvement.

Since it’s in the wild, I can now talk freely about this cool new feature of QuickTime. All of this I learned simply by hunting around and looking at sample traces of a running video stream in the new version of QuickTime Player.

QTKit Capture looks to be a total ground-up replacement of the Sequence Grabber. The Sequence Grabber is a very ancient part of QuickTime, designed long ago as the way to capture streaming audio/video. It doesn’t appear to be going away, but QTKit Capture doesn’t use it internally except for reverse compatibility* with old vdig drivers. Where the Sequence Grabber uses QuickDraw, QTKit Capture makes heavy use of Core Video and OpenGL. Also, the top layer is Objective-C so it can be integrated easily into a Cocoa app.

I’ve done some looking around and found some interesting things:

It makes use of some new private frameworks. New to the club are:
CoreMediaAuthoringPrivate.framework
CoreMediaIOServicesPrivate.framework
CoreMediaPrivate.framework

CoreMediaIOServicesPrivate’s framework Resources folder contains a number of plugin modules. There’s one for AVC (DV cams), IIDC (external iSight), and VDC (the built-in iSight). In a nutshell, CoreMedia appears to implement a Device Abstraction Layer (A way to allow access to all video devices without having to know the details of the device.)

Other notes:

• * QTKit Capture still works with existing QuickTime component vdig modules (macam still works). In this case, it is actually running the Sequence Grabber below CoreMedia. For the built-in iSight however, the Sequence Grabber is not being used. Hopefully, Apple will provide a new way to write plugins for third party devices so the SequenceGrabber doesn’t have to be used. However, they do support 3 major video camera standards (IIDC, AVC, and UVC), so device developers going forward would be wise to implement one of these standards, eliminating the need for writing their own driver.
• You can build and link against the new functionality if you have the proper header files.

We just finished some marathon coding and are happy to announce iPhoneDrive. In a nutshell, it lets you use your iPhone for file storage by providing an easy file browser interface. Version 1.0 is very functional considering that it was written in about one week. The demo version works just like the paid version but only for a 7 day period.

It supports drag and drop to and from the iPhone. It also supports transfer of entire folders. This makes it easy to move large directories back and forth between your Mac and your iPhone. Since the iPhone uses USB 2.0, the transfer speed is very fast. Now you can put those spare gigabytes to good use. Backup your important files! Your hard disk could fail at any moment! Do it now!

Here’s a screenshot:

Reinventing the Wheel

You can see we used an NSBrowser for the file browser. If you’ve ever tried to use one of these you’d know that they’re not very useful out of the box. Drag and drop has to be implemented manually, and the updating of cell data is very flaky, requiring many work-arounds. Also, they don’t do many of the things that you’d come to expect from the Finder’s column view mode.

All said and done, the interface came out pretty good. I just wish Apple would update NSBrowser to support more of the things found in the Finder column view. Obviously the Finder isn’t implemented using NSBrowser (it’s not even Cocoa for that matter), but it would be nice to have things like elipsification, marquee selecting, and drag and drop in there without having to reinvent the wheel.

We welcome any feedback about iPhoneDrive. Leave a comment here or use the feedback form on the main Ecamm website.

The iApps like iChat and Photo Booth only request a 640×480 or smaller image from the iSight. Therefore it can be troublesome to find a way to test the full capabilities of the new 1.3 MP iSight. That’s why we wrote our own little test program.

Today we cleaned it up and are posting it here so new-MacBook-Pro-having visitors can try it out see the new and improved image for themselves.

The app requests and displays a video stream from the camera at 1280×1024. If you’re on an older iSight it will still stretch the VGA image out to be this size but it won’t look very good. The app will report the frame actual size from the ImageDescription at the top.

There’s also a snapshot button so you can take your own picture and make your friends jealous of your fancy new laptop. Sorry: no Photo Booth effects yet!

If you get to try it, leave a comment and let me know how it works for you.
The Test App (68 k)

Tech Talk Follows:

The app does Hi-Lite™ a potential reason why Apple might not be yelling from the rooftops about the new camera: You can see right away how slow it is to stream the full sized video. Why? It’s more than four times more data than VGA. The Sequence Grabber was designed decades ago and wasn’t designed to handle more than NTSC/PAL sizes. Until the Sequence Grabber is revamped, it’s a little too slow to run such a large stream.

1,280 x 1,024 = 1,310,720 pixels
It’s 4:2:2 compressed YUV data, so that’s 2 bytes per pixel or 2,621,440, or 2.5 megabytes per raw frame.
So for full 30 fps video, this translates to about 75 megabytes per second or 600 Mbps.
Now keep in mind the camera controller allows for a compressed MJPEG mode which cuts the required USB bandwidth signifcantly. (You’d never fit that over USB 2.0 without compression.)

But even if it is being compressed on-camera, it’s then being decompressed in the driver and passed to QuickTime in YUV. That’s a lot of data to push through the poor Sequence Grabber pipes. Technically, you should be able to request MJPEG data from the sequence grabber, but the default seems to be YUV. I haven’t tried requesting other compression types as I don’t actually have one of these new machines in front of me.*

More details about the new camera:

The original iSight didn’t advertise itself as a UVC device (It was Vendor Specific).
The new one does advertise as UVC, so you can easily use USB Prober to decode the device descriptor:
According to the descriptor the camera supports the following frame formats:

MJPEG 640×480 @ up to 60fps
MJPEG 720×480 @ up to 60fps
MJPEG 800×600 @ up to 30fps
MJPEG 1024×576 @ up to 30fps
MJPEG 1024×768 @ up to 30fps
MJPEG 1280×960 @ up to 30fps
MJPEG 1280×1024 @ up to 30fps
Uncompressed 640×480 @ up to 30 fps
Uncompressed 352×288 @ up to 30 fps
Uncompressed 160×120 @ up to 30 fps
Uncompressed 704×576 @ up to 25 fps
Uncompressed 720×480 @ up to 25 fps

Maybe another SequenceGrabber savvy engineer out there with the new MBP could experiment and see what you get back from VDGetCompressionTypes. Can we request MJPEG data through the pipes?

*Another thanks to Dave for running some tests for me on his MBP. Dave is a web designer and has a website that I’ll plug.

Safari 3 on Windows is old news by now, but I still have this nagging feeling.

Apple has had iTunes on Windows for years, but iTunes is a Carbon app, and doesn’t contain any Cocoa, AppKit, or Objective-C. (A quick analysis with otool will show this.)
I’m not saying that getting iTunes ported to the PC was easy, but it’s not exciting because we don’t write Carbon apps anymore.

Now enter Safari for Windows… Safari is written from the ground up in Cocoa. A quick analysis of Safari will show you that it contains over 200 Obj-C classes and about 4000 Obj-C methods. Its controllers (logic) and views (the UI) are built entirely in Objective-C using AppKit.

Why is this an important fact?

I’m a huge fan of Cocoa and coding using AppKit: I find it makes coding fast and efficient. The only downside? Without a huge nasty porting effort, your code is permanently stuck on the Mac. All of your NSMutableDictionary, all of your NSWorkspace, all of your NSLock, all of your NS-anything. Today, it’s just not leaving your Mac. It’s stuck.

I’d really like to port CardRaider to Windows, but it makes such heavy use of Objective-C and AppKit, that it just seems like a nasty task.

Now enter Safari on Windows. Cursory examinations of the Safari 3 Windows binaries shows no evidence of an Objective-C runtime or any AppKit.dll. So let’s assume for a moment that they didn’t just port the Cocoa and AppKit libraries…
So did Apple really rewrite all 200 Obj-C classes in Safari in C++ or C# and change them to not use Cocoa? Are they somehow going to maintain two codebases, one written entirely in Obj-C based on AppKit, and one in another language and based some other library? Maybe they did… they seem to have done that with WebKit and WebCore….

Or, more likely, do they have some kind of secret button (some kind of tool, script, or compiler modification) that takes Safari’s 200 .m files and makes Windows code.

I’m not trying to trivialize their porting effort, because I’m sure it wasn’t an easy task. I’m just looking at the implications of maintaining two mostly identical versions of Safari 3, one written in Objective-C using Cocoa, and the other not allowed to use those things.

Did they really port this manually? I don’t think so.
otool -ov /Applications/Safari.app/Contents/MacOS/Safari | grep -c Module
202
otool -ov /Applications/Safari.app/Contents/MacOS/Safari | grep -c method_name
4097
otool -tV /Applications/Safari.app/Contents/MacOS/Safari | grep -c _objc_msgSend
20319

I hope for their sakes that it’s somehow automated, and if it is…

Apple: Share the technology!
Give developers the secret button to easily make their Cocoa apps into Windows apps!

Can you think of anything that could do more to encourage Mac development than allowing that code to also be useful for Windows?

But maybe they really did port it line by line, class by class.
If you look at the Windows branch of WebKit/WebCore, all the .m files have become .cpp files and the AppKit calls have become CoreFoundation calls. And it shows no sign of having been automated.
And a lot of the code has clearly been manually ported to Windows code. For example looking at a randomly selected class/routine:

The once graceful:
-[WebIconDatabase iconForURL:withSize:cache:]

has become:
HRESULT WebIconDatabase::iconForURL(BSTR url, LPSIZE size, BOOL cache, OLE_HANDLE* bitmap)


with the underlying NSWorkspace, NSImage, NSMutableDictionary, etc. calls now turned into dirty Windows CreateDIBSection code.