Nerd Fever

(Updated July 11 – it’s finally done.  Notes on regular expressions to search on and resetting to fix sync problems added.)

I’ve finally given in and joined the Apple collective.  On my phone, anyway.

For years my Mac-using friends have raved about how great the Apple experience is… “it just works!”.

If only.

I’ve been migrating my stuff from my old Palm Treo 650 to an iPhone 3GS for well over a week now.  I’m nearly there.  This post is partly to document things for myself, if I ever have to do this again.

Here are my tips if you’re going to do something similar:

1 – Calendar

This was the easiest, since I’d already moved onto Google Calendar some time ago (using GooSync with the Treo).  I just set the iPhone to sync with Google Calendar, and it works fine.

2 – RPN calculator

I’ve been using Russ Webb’s RPN on the Palm for many years.  It’s a solid, well-designed RPN calculator, with full programmability.  It worked well enough that I sold my old HP-42S on eBay (at a huge profit).  But it doesn’t run on the iPhone.  And there’s no Palm emulator for the iPhone due to one of Apple’s many stupid rules.  (“No emulators.”  Another stupid rule that galls me is “No scripting”, which means no Python.)

After trying lots of calculators in the App Store, I settled on 42S, a port of the Free42 re-implementation of the HP-42S (not an emulation; it doesn’t use HP’s ROMs).  It’s powerful, has a good UI, and is fully programmable.  It’s even open-source (but not free on the iPhone; it’s $5). I’m happy.

3 – Memos

I kept a lot of memos on the Palm, mostly shopping lists, to-do, etc.  I want these to be accessible from both my desktop and the iPhone.

Apple’s “Notes” app syncs to Outlook in iPhone 3.0, but I don’t (and won’t) use Outlook.

The solution I found was to put my memos in the “notes” field of Contact entries, then sync them with Google Contacts.  I put the memo title in the Company field, with “a.a ” in front of it – this forces the memos to the top of the list.

That solution was found after looking into a LOT of other possibilities, including the popular “Evernote” app and one called “gNotes” that syncs to Google Notebook.

I’d never have guessed it, but both of these apps (and who knows how many others) sync IN THE CLEAR!  If you’re using an open WiFi connection (say, at Starbucks), anyone on the network can see all your private notes being synced, using nothing more than a freeware packet sniffer like Wireshark.  I tried it myself – it’s true.  God help you if your bank account password is in there.

With all the stupid rules Apple has for the App Store, I can’t see why they allow apps to transmit private data in the clear (without even a warning); preventing that would be far more useful than “No emulators” or “No scripting”!

4 – Contacts

Now the sob story.

I spent days figuring out how to export my 800+ contacts from the Palm and get them into the iPhone and a desktop app.  (I used to use Palm Desktop.)

I don’t (and won’t) use Outlook.

I settled on Google Contacts as the best desktop solution – it has a simple, clean UI, and is accessible from anywhere on the web.  I can still get at my contacts on the iPhone if my Internet connection goes down.  And it’s free.

If you’re running older versions of Palm Desktop, there are export utilities on the Web that can help.  But I didn’t find one that supported Palm Desktop v6.5’s newer format.

However, Palm Desktop will export to vCard format.  Which is fine, except that everyone’s implementation of vCard has some non-standard fields that need to be translated.

So I spent a day or so reverse-engineering the custom vCard formats used by Palm and Google. (Create funny-looking contacts; export; view with text editor, hypothesize, edit, import, compare results.  Repeat…)

Then I exported all my contacts (one Palm “category” at a time, otherwise Google puts all the imports into the same “group”) into .VCF files and made the necessary corrections with a text editor; mostly via simple search-and-replace (but some use of regular expressions was needed).

Here are the steps I used:

A – Translate fields from Palm to Google format

The following vCard fields don’t import to Google; they’d need translation but I didn’t bother since I wasn’t using any of them:

X-PALM-ANNIV (anniversary date) >  no Google equivalent, need custom field

IMPP:IM (IM address) > X-GTALK or X-YAHOO, etc.

IMPP:IM2 (another IM address) > X-GTALK or X-YAHOO, etc.

X-PALM-ASSISTANT (name of secre… um, personal assistant) >  no Google equivalent

X-PALM-ASST-TEL (telephone number of same) > no Google equivalent

Then I translated a couple of custom fields in the Palm that I’d started using back before Palm introduced standard fields for the same thing:

“X-PALM-CUSTOM;1” > “EMAIL;INTERNET”

“X-PALM-CUSTOM;2” > “URL”

Next, the Palm marked a firm’s “main” number (switchboard number) differently than Google did:

“TEL;X-Palm-Main” > “TEL;TYPE=MAIN”

B – Translate extensions into commas

The iPhone address book refuses to dial any number that has characters other than +, 0-9, #, * and , (comma) in it.

This causes more than one kind of problem.

First, if you store telephone extension numbers like I did, you have a lot of numbers that look like this:

+1 800 555 1212 x1234

…indicating the use of extension 1234.  The only way I found to preserve this information is to change the ‘x’ (or “ext.”; whatever you use) into a comma (‘,’).  The comma is allowed, and causes the iPhone to pause (rather a long time) and then dial the extension (rather slowly).  At least it’s still there.

These I changed, after finding them with the regular expression “x[0-9]”.  (I use TextPad as my editor; your regex syntax may differ.)  That expression isn’t perfect, but I looked at each hit and edited by hand.

C – Create Custom Fields

The Palm Desktop had a concept of a”preferred” number for a contact – I think this was supposed to be the number that it would dial by default.  This was marked in the vCard format with “PREF” like this:

TEL;HOME;PREF:+1 800 555 1212

Google (and the iPhone) don’t seem to support this concept, although it is a part of the official vCard 2.1 spec (I read that too).  No problem, I thought – I’ll just move that into a note after the phone number:

+1 800 555 1212 (preferred)

After all, I have lots of other kinds of notes in my Palm address book numbers.  For example:

+1 800 555 1212 (lab)

+41 22 730 5111 (Geneva main)

+1 800 555 1212 (lake house)

etc.

This turned out to be a bigger problem.  As mentioned, the iPhone won’t dial a number with notes in it.  This is a major pain – the Palm happily ignored non-digit characters.

There is a workaround – “custom fields”.  You make a custom field label and put your notes in there.  So:

+1 800 555 1212 (lake house)

becomes

Lake house: +1 800 555 1212

The syntax for this is:

item1.TEL:+41 22 730 5111
item1.X-ABLabel:Geneva main
item2.TEL:+1 800 555 1212
item2.X-ABLabel:Lake house

Again, I edited in these changes by hand, after finding them with the regex “^TEL.+[0-9]+.+[:alpha:]”.  Remember to increment the item number if there are multiple custom fields in the same contact.

Once you do that, the iPhone offers “Lake house” and “Geneva main” as possible labels on every phone number, but I can live with that.

D – Fix truncated notes

Ah, the joys of incompatible software.  The Palm stores control characters (CR and LF) in a “QUOTED-PRINTABLE” format, such that newlines look like “=0D=0A”.  Which is fine, except that Google’s parser sometimes misinterprets ‘=’ symbols in the note (like “Son=Oliver”) in a way that truncates the note.  This only happened a few times, but I had to fix each one by hand.

I searched in the Palm-exported .VCF file for “^NOTE.*=[^0Q]” (a line starting with “NOTE” contaning an equal sign followed by something other than zero or Q).  Then I checked each imported note this found and fixed it by hand if necessary.

E – Resetting the iPhone/iTunes when out of sync

Somewhere while making all these changes (and test imports/exports) the iPhone/Google synchronization (performed by iTunes via USB)  got messed up, such that new entries in one contact list wouldn’t get synchronized with the other.

I tried the File>Preferences>Devices>Reset Sync History button in iTunes, but that didn’t help.  In the end, the only way I found to fix it was:

1. Uninstall iTunes
2. Delete the iTunes folder in /Users/<username>/Music (this was on Vista, try /Documents and Settings/<username>/My Documents/Music if you’re on XP).
3. Delete the “Apple” or “Apple Computer” folders in /Users/<username>/Local and …/LocalLow and …/Roaming.
4. Re-install iTunes

(I did try several subsets of that, which didn’t work.)

After all of that, I imported the result into Google.  And let it sync with the iPhone.

At first it seemed to work OK.  But when I turned on over-the-air syncing, some phone numbers were missing altogether, and others were very messed up on the iPhone.

F – Don’t use over-the-air syncing (Microsoft Exchange, ActiveSync)

After much hair-pulling and googling, it turns out that the ActiveSync protocol that Google uses to sync with the iPhone doesn’t support custom fields.  Any number labeled with a custom field doesn’t sync.  (In fact, just enabling over-the-air syncing on the iPhone disables the use of custom fields.)

So how do you identify which number is for the main office, and which for the regional office?  Or your friend’s main home vs. vacation home?  Answer: You don’t.

That won’t do.

The workaround is to disable over-the-air syncing for Contacts, but allow USB syncing (to Google Contacts) via the iTunes app.  (The setting is in the iPhone Settings app, under Mail, Contacts, Calendars > (your Gmail account) > Contacts.  Turn it OFF.

That’s it.

“It just works”?  I wish.