Is MS Word 2003 loading slower and slower?

It was for me. Turned out that that my Normal.dot file, usually 38 kBytes, had grown to 1.4 MBytes. That’s what was taking so long – loading it each time.

Simply deleting the file was a good temporary fix back in May (Word automatically re-creates it if it’s missing). On Win7 for me it was in:

C:\Users\Dave\AppData\Roaming\Microsoft\Templates

Lately, it’s been happening again – this time Normal.dot had grown to 1.8 MBytes (since May!).

Investigation revealed that the file grew by about 2 kBytes each time I loaded a file. I had a look at the binary of Normal.dot to see what was taking up so much space, and saw lots of Unicode text reading things like “Sign in to Office Live Workspace beta”.

So I un-installed “Microsoft Office Live Add-in”. Now Normal.dot doesn’t grow anymore.

This post is tagged “Sad”.

FileTimes – Discover your own work patterns

FileTimes.exe is a little program I wrote 10 years ago when I was learning C++. Download it here.

It plots a graph of the write timestamps on all the files in any folder you select (recursing thru all the subfolders). You can look at the time stamps by time of day, week, month or year. You can also control the averaging period with a slider.

Plotted by time of day

It’s fun for looking at when you’re busy – for example you can see here that I’m definitely not a morning person – I don’t really get going much before 10am.

If you move the mouse over the graph, it shows how many files were modified at the time point the mouse is at.

It works on any version of Windows since Win95 (up to Win7, so far). There’s no installer, just run the .EXE wherever you put it (the desktop is fine). It doesn’t mess with the registry, won’t modify your files at all, won’t create any files anywhere, and doesn’t have any viruses or spyware – I promise.

I don't slack off much on weekends, it seems.

It’s free – I hereby put it in the public domain.

This version says “ALPHA TEST VERSION – DO NOT DISTRIBUTE”, but it’s 10 years old. I haven’t Continue reading

Rev3 rocket electronics part 6: isr.c

This is post #6 in a series covering the hardware and software design for my work-in-progress GPS-guided rocket recovery project. The main index to the series of posts is here, and an introduction to the project (a PowerPoint presentation) is here.

This is the last post about the source code, it covers the interrupt service routine (ISR) in isr.c.

On any interrupt, PIC instruction execution goes to memory location 0x0008, where int_vector() causes execution of the interrupt service routine (ISR), isr().

Z-80 SPEEDOMETER

On entry to the ISR, macro PUSH_DEBUG_STATE() (in debug.h) stores the current state of the Z-80 speedometer output SPEEDO in variable pushed_debug_state. This will be restored on exit from the ISR by macro POP_DEBUG_STATE. (BTW, the only thing this has to do with a Z-80 is that’s what I’ve been calling this trick since I first did it on a Z-80 project back in the early ’80s…)

Then the ISR raises the SPEEDO output, because regardless of whether the PIC was busy when the interrupt occurred, it is busy during interrupt servicing.

TIMER1 (RTC) RESET

The ISR then checks PIR1bits.TMR1IF to see if the interrupt was caused by Timer1 overflowing, which generates the RTC tick interrupt. The result is recorded in variable timer1_expired.

If Timer1 did overflow, it is reset by reloading it with the value TMR_START, to trigger the next RTC interrupt at the proper time.

Note that the value of TMR_START (calculated in hardware.h) accounts for the 6 Timer1 tick periods of interrupt latency between Timer1 overflowing and the ISR resetting it. A few NOP instructions are executed before resetting Timer1 to round up this time to a whole number of Timer1 ticks. (The number of NOPs needed was found using Microchip’s PIC18 simulator, included in the MPLAB IDE).

Ideally, the first peripheral to be serviced in the ISR would be the UART receive buffer. At 38,400 bps data bytes from the GPS can arrive as little as 260 microseconds apart, making this interrupt much more time-critical than others. But servicing the UART involves enough conditionals that predicting the exact number of CPU cycles becomes complicated, making it difficult to keep the RTC tick interval accurate.

So instead, the Timer1 interrupt is dealt with first, but just the minimum necessary to reset it. Other tasks that will be performed on the RTC interrupt are postponed until after dealing with GPS data in the UART receive buffer.

RECEIVED DATA

While there is data to be read in the UART receive buffer (PIR1bits.RCIF == TRUE), the data is read out of the UART and stored in NMEAbuffer[] for later parsing by CheckForNewGPSFix() in main().

Before each byte is stored in the buffer, I check:

if (RTC - LastUARTrxRTC > GPS_BURST_PERIOD_MS/RTC_INTERRUPT_INTERVAL_MS)

If so, this means that no data has been received from the GPS for longer than one GPS data burst period. Since the time between data bursts (typically 950 mS or so) is longer than the duration of each burst (typically 50 mS), this indicates that this is the first byte of a new burst, and so the current RTC value is saved in global GPSBurstStartRTC. This is logged with the GPS fix data, to indicate the exact time the fix burst started in terms of the RTC.

Next, the current RTC time is stored in LastUARTrxRTC. This will be used byFlushFlashBufferSafe() and BetweenGPSFixesFor() (both in hardware.c) to detrmine if we are current receiving a GPS data burst, and so if writing to Flash memory needs to be postponed.

If the received byte was a comma, it is converted into a zero. This transforms the NEMA-0183 formatted GPS message (a series of comma separated values) into a series of zero-terminated C strings, to make parsing of the data simpler for CheckForNewGPSFix().

If the overrun flag (RCSTAbits.OERR) was set, this indicates that the UART receive data buffer overflowed (was overrun) and one or more bytes of data were lost. This should never happen, and I don’t think it ever does. But just in case I’m wrong (for example if the CPU clock speed were reduced, the UART baud rate increased, or the ISR changed, this could happen), the flag is reset and the entire line of received data in the buffer is discarded – it’s important to avoid parsing corrupted data. (A wrong GPS fix is much more confusing to the navigation algorithm than a missed one.)

RTC SERVICING

Once the UART has been dealt with, the timer1_expired flag, which was set earlier in the ISR, is checked. If the flag is set, this indicates that the ISR needs to service the Timer1 RTC interrupt, regardless of whether or not there was any received UART data to process.

Macro ServiceBeep() (defined in peripherals.h) is called twice in the RTC handling portion of the ISR – here and again at the end. On each call, this inverts the PIEZO output bit (to the piezo buzzer) if the global variable Beep is set. Since the RTC timer runs at 1 kHz (1 mS intervals), this generates a 1 kHz tone whenever Beep is set – this is used, for example, by BeepOutMaxAltitude() which is called in the FULL state.


Servo pulse generation

Hobby servo pulse code

Servo control pulses. (Image from Seattle Robotics Society, http://www.seattlerobotics.org/guide/servos.html)


Continue reading

Rev3 rocket electronics part 5: stateMachine.c

This is post #5 in a series covering the hardware and software design for my work-in-progress GPS-guided rocket recovery project. The main index to the series of posts is here, and an introduction to the project (a PowerPoint presentation) is here.

In this post I’ll cover stateMachine.c, which implements the finite state machine (FSM) that handles most of the functional logic of the system.

STATE MACHINE

When implementing a state machine, I don’t feel an obligation to observe the constraints of a formal FSM of the kind taught in schools, where all functionality happens in state transitions.

Instead, I treat each state as a system mode. There are state entry actions and state exit actions, but while in each state the system also has a distinct behavior. While I could draw a state diagram, I’ve never bothered – the system is simple enough that it’s unnecessary, and the diagram wouldn’t capture much of the important behavior of the system.

The state machine is run one step by a call to StateMachine() from main(), which happens once each time through the main loop.

Each of the six states, IDLE, ARMED, TIMEDLAUNCH, FLIGHT, FULL, and TEST, (all defined in stateMachine.h) has an associated “state function” that implements the system behavior for that state. Each state also has an associated entry function, which can be called by state functions to cause the system to transition to a new state.

The state functions take and return no parameters and have the same names as the state symbols, but in lowercase:

  • void idle(void);
  • void armed(void);
  • void timedlaunch(void);
  • void flight(void);
  • void full(void);
  • void test(void);

The entry functions also take and return no parameters and are named based on the state they enter:

  • void enterIDLE(void);
  • void enterARMED(void);
  • etc.

The current state and sub-state are stored in global variables State and Substate.

STATE TRANSITIONS

When conditions warrant, a state function will cause a transition to a different state by calling the appropriate entry function, which will change the global variables State and Substate to reflect the new state, as well as performing any entry actions needed to configure the system for the new state.

A flag, stateTransitionInProgress, is reset to FALSE on entry into StateMachine(). Each entry function starts by verifying that this flag is FALSE, and if so, setting it TRUE. If the flag is not FALSE on entry, this indicates that the FSM has already transitioned to a new state, and therefore the call to the entry routine is invalid. In that case, the entry routine is aborted and the state remains unchanged.

Wiring to camera shutter button

This eliminates the possibility of a subtle bug which could occur if, once a state function calls an entry function (to transition to a new state), later logic in the same state function tries to put the system into a different new Continue reading

Rev3 rocket electronics part 4: main.c

This is post #4 in a series covering the hardware and software design for my work-in-progress GPS-guided rocket recovery project. The main index to the series of posts is here, and an introduction to the project (a PowerPoint presentation) is here.

This post covers main.c, which has just one function in it, main(), the program that runs on power-up or reset.

INITIALIZATION

main() starts by calling SetupHardware() in hardware.c, which initializes the PIC internal registers. Many of the values used to initialize things are computed from macros based on tweaks.h. OSCCON is set to control the CPU clock frequency, INTCON and PIE are set to enable the global, peripheral, UART, and Timer1 interrupts. SetupHardware() has comments on how Timer1 is used to generate the servo control pulses, which I won’t repeat here, but see my discussion of this in my posting about the ISR.

Next 16-bit timers Timer1 and Timer3 are initialized to run at a rate of 250 kHz (1/32 of the 8 MHz CPU clock) and started.

Then, the I/O ports are setup (direction and weak pullups on Port B), CVRef is setup to produce a reference voltage and an analog comparator is configured for level-conversion of the incoming received GPS data (see the hardware description for details of that) and an ADC is setup to read AN1 for the pressure sensor.

Finally Timer2 (TMR2) and CCP2 are setup to produce the 47 kHz square wave modulation for the camera control IR pulses (again, see the hardware description). Finally, the magnetic compass is initialized, if the compilation switch COMPASS is defined (it isn’t, now).

main() then turns on all 3 LEDs (to indicate that the system is powered up and running), sets the servo to the middle position, initializes the leaky integrator values used to smooth out pressure sensor readings, calls InitGPS() to send GPS commands to set the desired baud rate and mode (38,400 bps, $GPRMC and $GPGGA messages only), and initializes the state machine.

The last thing main() does before entering the main (endless) software loop is:

nextRTC8 = RTC8 + LOOP_PERIOD;

This deserves some explanation, as I use this technique a lot (mostly in stateMachine.c – see next post).

Global 32-bit variable RTC holds a count of real-time-clock (RTC) interrupts since power-up. Macros RTC24, RTC16, and RTC8 (all in globals.h) return the low-order 24, 16, and 8 bits of this count (respectively – note that the C18 compiler supports 24 bit wide variables).

This statement assigns the modulo-256 (8 bit) sum of the lower 8 bits of this count (RTC8) and a constant called LOOP_PERIOD to an 8-bit variable called nextRTC8.

LOOP_PERIOD is defined in tweaks.h as the number of RTC ticks in 40 milliseconds (25 Hz). So nextRTC8 represents the future value RTC8 will have 40 milliseconds from now.

By comparing nextRTC8 to RTC8, we can easily determine whether the 40 milliseconds have expired. Macros YET8(), YET16(), YET24(), YET32() and NOTYET8(), NOTYET16(), NOTYET24() and NOYET32() (all in globals.h) implement this comparison and allow for the possibility that by the time the comparison is made (between nextRTC8 and RTC8), RTC8 might have already moved on to a later value (if the check isn’t performed between each tick of the RTC interrupt).

So, when later in main() we say:

while (NOTYET8(nextRTC8))

we will continue in the while() loop as long as the current time has not yet reached future time “nextRTC8” (40 milliseconds from the time we set the value of nextRTC8).

Of course, nextRTC8 and RTC8 are only 8 bits wide, so the time represented in nextRTC8 had better not be more than 127 RTC ticks in the future (127 mS). If you need to deal with longer time periods than that, you’ll have to use one of the wider versions – RTC16, RTC24, or RTC32.

MAIN LOOP

The main loop of the program starts with while(1) and runs once every 40 milliseconds (25 Hz). All system functions are driven by this loop, except what the interrupt service routine (ISR) does.

A 32-bit variable CycleCount is incremented each time through.

Graph from logged pressure altitude & GPS data. GPS altitude is useless – data is heavily damped, resulting in a lag of 12 seconds or so. But the GPS position data is not too bad.

GPS

Then CheckForNewGPSFix() is called (from nmeaGps.c).  Any incoming data from the GPS has been received by the ISR and placed in a small text buffer, NMEAbuffer[]. CheckForNewGPSFix() looks at the text buffer to see if a complete message has been received from the GPS. If so, it parses the message. ($GPGGA or $GPRMC; these are the only two messages that are handled.) Continue reading

Rev3 rocket electronics part 3: Source code summary

This is post #3 in a series covering the hardware and software design for my work-in-progress GPS-guided rocket recovery project. The main index to the series of posts is here, and an introduction to the project (a PowerPoint presentation) is here.

This post is an introduction to the source code. (Which can be downloaded from my last post.)

OVERVIEW OF SOURCE FILES

The software consists of the following C source files:

  • main.c – hardware init and main program loop
  • stateMachine.c – finite state machine (most functionality is here)
  • globals.c – global variables
  • isr.c – interrupt service routine
  • hardware.c – hardware abstraction wrappers
  • peripherals.c – higher-level (than hardware.c) peripheral control
  • nmeaGps.c – reads and parses NMEA-0183 standard format GPS data
  • logging.c – logging of flight status data to Flash memory
  • compass.c – drives Honeywell HM55B magnetic compass (no longer used)

and the following header files:

  • tweaks.h – main “tweakable” parameter file and compilation switches
  • stateMachine.h – defines state values and prototypes for stateMachine.c
  • globals.h – prototypes for globals.c, generally useful macros
  • hardware.h – symbol definitions for hardware abstractions, prototypes for hardware.c
  • peripherals.h – peripheral-related macros and constants, prototypes for peripherals.c
  • nmeaGps.h – data structure for GPS fixes, prototypes for nmeaGps.c and related constants
  • logging.h – data structures for flight data logging, prototypes for logging.c
  • compass.h – hardware abstractions for compass, prototypes for compass.c
  • debug.h – macros to support “Z-80 Speedometer”
  • djllib.h – variable type definitions, standard constants, misc. macros

In addition there is the Microchip header file pic18f26k20.h and the linker file 18f2620i.lkr. Neither of these are my work – they are unmodified from Microchip.

Servo arrangement after successful radio-controlled landing. To fit in the rocket, the servo arm stays at an extreme setting until after parachute deployment.

SYMBOL NAMING CONVENTIONS

These are the naming conventions I use for function and variable names:

Function scope Variable scope Convention 1st character
n/a Block (local) lower_case_ABBR_excepted_underscore_delimited Lowercase
File (normal) File mixedCaseDelimitedABBRexcepted Lowercase
Global (exported) Global MixedCaseDelimitedABBRexcepted Uppercase
Macro Macro ALL_CAPS_UNDERSCORE_DELIMITED Uppercase
n/a Typedef ALL_CAPS_UNDERSCORE_DELIMITED Uppercase

Also, a name with a leading underscore, for example _name(), indicates a “hidden” function or variable which is not meant for general use. Usually it’s a component of some other function that is meant to be used on a normal basis. (See __waitloop_internal() in hardware.c and #define wait() in hardware.h for an example.)

WHERE TO START

If you want to have a look at the software with an eye to modifying it, I suggest you start with a quick skim of the following files (in this order): Continue reading

Rev3 rocket electronics part 2: Software overview

This is post #2 in a series covering the hardware and software design for my work-in-progress GPS-guided rocket recovery project. The main index to the series of posts is here, and an introduction to the project (a PowerPoint presentation) is here.

This post will introduce the software at a high level.

I’ll start with a rough outline of the software architecture and functional spec for the Rev3 hardware/software system.

I say rough because, like most one-man projects, things evolve without necessarily being thoroughly documented. That said, the code is heavily commented (and the comments are up-to-date), so it shouldn’t be too difficult to follow what is going on.

This post will give you an idea of what to expect, but you really need to read the source code to fully understand what is going on.

Regarding rights: My attitude toward the software is a little different than for the hardware. I’m posting all the source code and these comments to help people understand it, and I want to encourage people to use and modify it. However, much hard-won experience is represented in the software and it has a lot more general utility, even outside of rocketry.

So here’s the deal – I hereby grant everyone and everything in the universe permission to use and modify this software for any NON-COMMERCIAL purpose whatsoever, PROVIDED that you (a) agree not to sue me about it, (b) credit Nerdfever.com as the original source of the software in any publications, and (c) agree that I make no promises and that if you’re unhappy the most I owe you is what you paid me (zip, zero, nada, nothing). Oh, and you agree to USE THIS AT YOUR OWN RISK, that you’re a responsible adult and know that rockets can be dangerous and hurt people if you’re not careful (regardless of whether or not software is involved) so you’ll be careful and will not blame anyone else if you screw up (especially me).

I’ll be very pleased if you leave a comment or drop me an email if you find it useful, but you don’t have to.

For COMMERCIAL use, ask my permission first.  If you’re going to make lots of money off my work, I’d like a (oh-so small and reasonable) cut.  But I’ve no intention of giving anybody heartache over small amounts – just ask, I think you’ll find me surprisingly easy to deal with.

That said, here is the the code, including the Microchip MPLAB IDE project files, just as it appears in the project folder: Rev3Board.zip

You may find it handy to look at the code when reading this introduction, although I’ll save the gory details for later posts.

SOFTWARE ARCHITECTURE

There is no real-time operating system (RTOS) – no operating system of any kind.

All the code is in C running directly on the “naked” hardware. In my experience RTOSes are very rarely useful in simple-to-moderate complexity embedded systems. Instead, they tend to add a needless layer of complexity and overhead that serves to obscure what is really going on. (RTOSes do have a proper place, I just think they’re over-used.)

I also don’t use any of Microchip’s hardware interface libraries. Instead, I control the hardware directly according to the PIC datasheet. I prefer to know and control what is really going on, and learning how to control the hardware doesn’t take much longer than learning how to use the library functions.

Some will say this makes the code less portable to other MCUs, but I prefer to have a solid understanding of what the code is doing so that I can intelligently port it when needed. In my view this is less risky than relying on the compatibility of “black-box” libraries which, in my experience, often disappoint. I do isolate all the hardware interface functions in a single module (see hardware.h and hardware.c).

As you read the code, you’ll see that I compute many constant values in the C pre-processor. (For example, see #define FLASH_BUFFER_ROWS and #define MS_TO_UTICKS  in hardware.h.)  I find this helps clarify how the constants are calculated, as well as reducing code size.

CONDITIONAL COMPILATION (#ifdef)

I make some use of conditional compilation in the code. The major compilation switches are all set (or not) in file tweaks.h. Most of them relate to the presence or absence of various bits of hardware (magnetic compass, GPS models, particular servos, etc.)

The compilation flags __DEBUG and ICD2 disable the “red” button input (symbol SW_RED) for debug builds. This button shares a pin with the ICD2 reset line, so including __DEBUG will include the compilation switch ICD2, which prevents the “red” button from being used (and therefore resetting the CPU). When these switches are defined, the compiler will generate a couple of warnings about code that doesn’t do anything – this is intentional. Continue reading

Things the iPhone could learn from the Treo

As per the topic of my last post, I recently switched from a Palm Treo 650 to an iPhone 3GS.

In most ways the iPhone is far more advanced, but as a PDA the iPhone still falls short of the 7 year old Treo design.

1 – Getting to most-used apps fast

Edit 2014: The Android app “Cover” does a lot of this, altho it’s half-baked. Unfortunately they got bought by Twitter, who seem to have ended development on it.

The iPhone’s UI is beautiful, but it is needlessly slow to get at often-used apps like Phone and Contacts.

On the Treo, a single button press gets you to the phone keypad.  A different single button gets you to Contacts, or any of 2 apps of your choice.  Admittedly, the Treo has many more buttons, but the iPhone could do far better.

On the iPhone, you:

  1. Press Home
  2. “slide to unlock”
  3. Press Home again (if you were previously in an app)
  4. Slide the app menu left or right a few times (if the app you want isn’t on the first menu page or the dock)
  5. Press the app you want

In the best case it’s 3 steps to your app, in the worst 5 steps.  That’s a lot of work just to start your favorite app.

But this is completely unnecessary.  Apple could easily do something like this:

iPhoneMockup2

Apple, you can do better!

(forgive the crude Photoshop work; but you get the idea).

This way you get to your favorite apps much quicker – just Home and one swipe.

Apple, if you want to do this, you have my permission – I won’t sue you.  Just ask if you want it in writing (see “About me“).

2 – Named app pages

The Treo let you name each page of apps, so you could categorize them.  And you could walk thru each page with the Home button.  I don’t see why Apple can’t do that.

3 – Contact searching

The Treo was much quicker at searching for contact entries.  It had a clever system where if you entered “db” it would search not just for names containing “db” but also for names with the initials “D.B.”.  This worked really well – just 2 or 3 letters was usually enough to identify a contact this way.

The only reasons I can think of why Apple doesn’t do this are (1) they didn’t know about it, or (2) patent issues.  But I’d think Apple and Palm are both infringing on enough of each others patents to make that moot – they’re already well into the realm of mutually-assured destruction.

4 – Telephone number selection

Again, Treo wins.

On the Treo all the phone numbers for each contact (office, home, fax, mobile, etc.) are visible on the screen.  You can directly click any one of them and dial it.

On the iPhone, you first find your contact, then select it, and only then can you choose a number to dial.  Three steps vs. one on the Treo.

5 – No casual notes in phone numbers

The Treo would happily ignore everything after the first alphabetic character in a stored phone number, so you could include casual notes like this:

+1 800 555 1212 (lake house)

+1 800 555 1213 (girlfriend’s place)

That’s a no-no on the iPhone – it will simply refuse to dial numbers that contain “invalid” characters.

There is no good substitute way to store this kind of info, which I find pretty important.

I’m very impressed with the iPhone’s capabilities, but I’m surprised how little Apple learned from what was already in the market.

In retrospect, I think I might have been better off buying the Palm Pre.  But I did want to try the “Apple experience”, and having spent two weeks getting the iPhone setup, I think I’ll let the Concorde Effect do it’s dirty work and stick with the iPhone for a while.

At least until my contract with AT&T is up.

Painful migration from Palm to iPhone

(Updated July 11 – it’s finally done.  Notes on regular expressions to search on and resetting to fix sync problems added.)

I’ve finally given in and joined the Apple collective.  On my phone, anyway.

For years my Mac-using friends have raved about how great the Apple experience is… “it just works!”.

If only.

I’ve been migrating my stuff from my old Palm Treo 650 to an iPhone 3GS for well over a week now.  I’m nearly there.  This post is partly to document things for myself, if I ever have to do this again.

Here are my tips if you’re going to do something similar:

1 – Calendar

This was the easiest, since I’d already moved onto Google Calendar some time ago (using GooSync with the Treo).  I just set the iPhone to sync with Google Calendar, and it works fine.

2 – RPN calculator

I’ve been using Russ Webb’s RPN on the Palm for many years.  It’s a solid, well-designed RPN calculator, with full programmability.  It worked well enough that I sold my old HP-42S on eBay (at a huge profit).  But it doesn’t run on the iPhone.  And there’s no Palm emulator for the iPhone due to one of Apple’s many stupid rules.  (“No emulators.”  Another stupid rule that galls me is “No scripting”, which means no Python.)42s on the iPhone

After trying lots of calculators in the App Store, I settled on 42S, a port of the Free42 re-implementation of the HP-42S (not an emulation; it doesn’t use HP’s ROMs).  It’s powerful, has a good UI, and is fully programmable.  It’s even open-source (but not free on the iPhone; it’s $5). I’m happy.

3 – Memos

I kept a lot of memos on the Palm, mostly shopping lists, to-do, etc.  I want these to be accessible from both my desktop and the iPhone.

Apple’s “Notes” app syncs to Outlook in iPhone 3.0, but I don’t (and won’t) use Outlook.

The solution I found was to put my memos in the “notes” field of Contact entries, then sync them with Google Contacts.  I put the memo title in the Company field, with “a.a ” in front of it – this forces the memos to the top of the list.

That solution was found after looking into a LOT of other possibilities, including the popular “Evernote” app and one called “gNotes” that syncs to Google Notebook.

I’d never have guessed it, but both of these apps (and who knows how many others) sync IN THE CLEAR!  If you’re using an open WiFi connection (say, at Starbucks), anyone on the network can see all your private notes being synced, using nothing more than a freeware packet sniffer like Wireshark.  I tried it myself – it’s true.  God help you if your bank account password is in there.

With all the stupid rules Apple has for the App Store, I can’t see why they allow apps to transmit private data in the clear (without even a warning); preventing that would be far more useful than “No emulators” or “No scripting”!

4 – Contacts

Now the sob story.

I spent days figuring out how to export my 800+ contacts from the Palm and get them into the iPhone and a desktop app.  (I used to use Palm Desktop.)

I don’t (and won’t) use Outlook.

I settled on Google Contacts as the best desktop solution – it has a simple, clean UI, and is accessible from anywhere on the web.  I can still get at my contacts on the iPhone if my Internet connection goes down.  And it’s free.

If you’re running older versions of Palm Desktop, there are export utilities on the Web that can help.  But I didn’t find one that supported Palm Desktop v6.5’s newer format.

However, Palm Desktop will export to vCard format.  Which is fine, except that everyone’s implementation of vCard has some non-standard fields that need to be translated.

So I spent a day or so reverse-engineering the custom vCard formats used by Palm and Google. (Create funny-looking contacts; export; view with text editor, hypothesize, edit, import, compare results.  Repeat…)

Then I exported all my contacts (one Palm “category” at a time, otherwise Google puts all the imports into the same “group”) into .VCF files and made the necessary corrections with a text editor; mostly via simple search-and-replace (but some use of regular expressions was needed).

Here are the steps I used:

A – Translate fields from Palm to Google format

The following vCard fields don’t import to Google; they’d need translation but I didn’t bother since I wasn’t using any of them:

X-PALM-ANNIV (anniversary date) >  no Google equivalent, need custom field

IMPP:IM (IM address) > X-GTALK or X-YAHOO, etc.

IMPP:IM2 (another IM address) > X-GTALK or X-YAHOO, etc.

X-PALM-ASSISTANT (name of secre… um, personal assistant) >  no Google equivalent

X-PALM-ASST-TEL (telephone number of same) > no Google equivalent

Then I translated a couple of custom fields in the Palm that I’d started using back before Palm introduced standard fields for the same thing:

“X-PALM-CUSTOM;1” > “EMAIL;INTERNET”

“X-PALM-CUSTOM;2” > “URL”

Next, the Palm marked a firm’s “main” number (switchboard number) differently than Google did:

“TEL;X-Palm-Main” > “TEL;TYPE=MAIN”

B – Translate extensions into commas

The iPhone address book refuses to dial any number that has characters other than +, 0-9, #, * and , (comma) in it.

This causes more than one kind of problem.

First, if you store telephone extension numbers like I did, you have a lot of numbers that look like this:

+1 800 555 1212 x1234

…indicating the use of extension 1234.  The only way I found to preserve this information is to change the ‘x’ (or “ext.”; whatever you use) into a comma (‘,’).  The comma is allowed, and causes the iPhone to pause (rather a long time) and then dial the extension (rather slowly).  At least it’s still there.

These I changed, after finding them with the regular expression “x[0-9]”.  (I use TextPad as my editor; your regex syntax may differ.)  That expression isn’t perfect, but I looked at each hit and edited by hand.

C – Create Custom Fields

The Palm Desktop had a concept of a”preferred” number for a contact – I think this was supposed to be the number that it would dial by default.  This was marked in the vCard format with “PREF” like this:

TEL;HOME;PREF:+1 800 555 1212

Google (and the iPhone) don’t seem to support this concept, although it is a part of the official vCard 2.1 spec (I read that too).  No problem, I thought – I’ll just move that into a note after the phone number:

+1 800 555 1212 (preferred)

After all, I have lots of other kinds of notes in my Palm address book numbers.  For example:

+1 800 555 1212 (lab)

+41 22 730 5111 (Geneva main)

+1 800 555 1212 (lake house)

etc.

This turned out to be a bigger problem.  As mentioned, the iPhone won’t dial a number with notes in it.  This is a major pain – the Palm happily ignored non-digit characters.

There is a workaround – “custom fields”.  You make a custom field label and put your notes in there.  So:

+1 800 555 1212 (lake house)

becomes

Lake house: +1 800 555 1212

The syntax for this is:

item1.TEL:+41 22 730 5111
item1.X-ABLabel:Geneva main
item2.TEL:+1 800 555 1212
item2.X-ABLabel:Lake house

Again, I edited in these changes by hand, after finding them with the regex “^TEL.+[0-9]+.+[:alpha:]”.  Remember to increment the item number if there are multiple custom fields in the same contact.

Once you do that, the iPhone offers “Lake house” and “Geneva main” as possible labels on every phone number, but I can live with that.

D – Fix truncated notes

Ah, the joys of incompatible software.  The Palm stores control characters (CR and LF) in a “QUOTED-PRINTABLE” format, such that newlines look like “=0D=0A”.  Which is fine, except that Google’s parser sometimes misinterprets ‘=’ symbols in the note (like “Son=Oliver”) in a way that truncates the note.  This only happened a few times, but I had to fix each one by hand.

I searched in the Palm-exported .VCF file for “^NOTE.*=[^0Q]” (a line starting with “NOTE” contaning an equal sign followed by something other than zero or Q).  Then I checked each imported note this found and fixed it by hand if necessary.

E – Resetting the iPhone/iTunes when out of sync

Somewhere while making all these changes (and test imports/exports) the iPhone/Google synchronization (performed by iTunes via USB)  got messed up, such that new entries in one contact list wouldn’t get synchronized with the other.

I tried the File>Preferences>Devices>Reset Sync History button in iTunes, but that didn’t help.  In the end, the only way I found to fix it was:

  1. Uninstall iTunes
  2. Delete the iTunes folder in /Users/<username>/Music (this was on Vista, try /Documents and Settings/<username>/My Documents/Music if you’re on XP).
  3. Delete the “Apple” or “Apple Computer” folders in /Users/<username>/Local and …/LocalLow and …/Roaming.
  4. Re-install iTunes

(I did try several subsets of that, which didn’t work.)

After all of that, I imported the result into Google.  And let it sync with the iPhone.

At first it seemed to work OK.  But when I turned on over-the-air syncing, some phone numbers were missing altogether, and others were very messed up on the iPhone.

F – Don’t use over-the-air syncing (Microsoft Exchange, ActiveSync)

After much hair-pulling and googling, it turns out that the ActiveSync protocol that Google uses to sync with the iPhone doesn’t support custom fields.  Any number labeled with a custom field doesn’t sync.  (In fact, just enabling over-the-air syncing on the iPhone disables the use of custom fields.)

So how do you identify which number is for the main office, and which for the regional office?  Or your friend’s main home vs. vacation home?  Answer: You don’t.

That won’t do.

The workaround is to disable over-the-air syncing for Contacts, but allow USB syncing (to Google Contacts) via the iTunes app.  (The setting is in the iPhone Settings app, under Mail, Contacts, Calendars > (your Gmail account) > Contacts.  Turn it OFF.

That’s it.

“It just works”?  I wish.

Detecting Windows software RAID failure

A RAID 1 or RAID 5 array will avoid losing data when one of the N drives in the array fails.

But if a second drive fails, you lose your data.  So it’s important to know when one of your RAID drives has a problem.

Windows NT, 2000, and XP (if you do this), plus server versions of Windows, support software RAID – no special RAID hardware needed, just drives visible to the OS.  (It might work with Vista, too; I haven’t tried.)

But there’s no way for Windows to notify you when there’s a problem with one of the drives.  (You really want to replace a failing drive before a second one goes bad!)

So I put together CheckRAID to solve this.  (Click to download; it’s a 133 kB ZIP file).  It’s freeware (public domain).

Unzip it into a folder (anywhere you please) and run CheckRAID.bat every so often to check for a failed (or unsynchronized) RAID drive.   See the README.TXT inside for more details.

Enjoy.  If you find this useful, I’d appreciate a comment.