Author Archives: Dave

Questions are more valuable than answers

Posted on by
1

…at least if you’re Google.

The interesting site Terms of Service; Didn’t Read gives Google a thumbs-down because “Google can use your content for all their existing and future services”.

I don’t think a thumbs-down is really fair here – I mean, that’s the whole point of Google.

Google is a service that gives out free answers in exchange for valuable questions.

Answers are worthless to Google (though not to you) because Google already knows those answers. But it doesn’t know your questions. So the questions are valuable (to Google, not to you). Because Google learns something from every question.

When you start typing a search into Google and it suggests searches based on what other people have searched for, that’s using your private information (your search history) to help other people. They’re not giving away any of your personal information (nobody but Google knows what you searched for or when), but they are using your information.

Google gets lots of useful information from the questions that people ask it. It uses that information to offer valuable services (like search suggestions) to other people (and to you), that they make money from (mostly by selling advertising).

That’s not a bad thing. It’s the only way to do many of the amazing, useful, and free things that Google does. I’m perfectly fine with it, but you have to more-or-less trust Google to stick to their promise to keep your private info private.

I think Google does a lot more of this than most people suspect.

When you’re driving and using Google Map to navigate, you’re getting free maps and directions. But Google is getting real-time data from you about how much traffic is on that road, and how fast it’s moving.

When you search for information on flu symptoms, Google learns something about flu trends in your area.

Sometimes I ask Google a question using voice recognition and it doesn’t understand. After a couple of tries, I type in the query.  I’ve just taught Google what I was saying – next time it’s much more likely to understand.

When you use GMail, Google learns about patterns of world commerce and communication, who is connected to who, who is awake at what time of day, etc. Even if it doesn’t read the contents of the mail.

When you search for a product, Google learns about demand in that market, by location and time of day and demographics (it knows a lot about you and your other interests).

Google learns from our questions – answers are the price Google pays for them.

Email migration from Eudora (or Thunderbird) to GMail

Posted on by
30

Update November 17 2012: 

I started this post back on July 20 – four months ago – and have only finished the migration now. That’s because I found a lot of unexpected problems along the way.

The rest of this post is my new, improved, and corrected step-by-step guide on how to migrate email from Eudora (up to version 7) or Thunderbird to GMail, based on hard-won experience.

BACKGROUND

I’ve been using Eudora since 1995 (17 years). Before that I used Unix mail (Sun; early 1990s), CompuServe (back to 1981), and something called “The Source” (1979; I was ‘TCA818’). Disk space was expensive back then so I didn’t save a lot of email.

But since 1995 I’ve kept everything – disk drives got bigger faster than my mail archive. So I had about 14 GBytes of email in Eudora.

Eudora has been abandoned by Qualcomm since 2006 and is getting old; there’s a new open source version based on Thunderbird (“Eudora OSE”), which sucks.

As our company grows the job of administering Eudora users (plus myself) was getting too big for me, so I decided to outsource it to Google Apps for Business. Which means GMail, and migrating the old email.

ABBREVIATIONS

In the following, I’ll use these abbreviations:

  • E7 – Eudora 7 (or any earlier version of Eudora)
  • OSE – Eudora Open Source Edition (used only for migration)
  • Tbird – Mozilla Thunderbird
  • Outlook – Microsoft Outlook 2003 or later
  • OE – Outlook Express
THINGS THAT WON’T WORK

To save you time, here are some things that won’t work:

  1. Syncing E7 to GMail using IMAP
  2. Converting E7 to E-OSE, then sync E-OSE with GMail using IMAP
  3. Converting E7 to E-OSE, then sync Tbird with GMail using IMAP
  4. Converting E7 to Outlook Express, that to Outlook, then Outlook to GMail
  5. IMAPSize
  6. Importing E7 data with Outlook, then sycning to GMail
  7. Importing Tbird data with Outlook, then sycning to GMail
  8. Importing E7 data with Thunderbird
  9. Syncing Outlook with GMail via IMAP (use this instead; it works)

Trust me, don’t bother.

If you really care or don’t believe me, in the Appendix at the very end of this posting I’ve put some of my notes on why some of those don’t work.

(I haven’t the patience to describe all the problems…comments in the Python files give some more details.)

STUFF YOU WILL NEED

To do the migration, you’ll need the following:

These are only for the migration. Once it’s done you can throw away all of them.
Continue reading →

Slow-motion rocket videos shot with Nikon J1

Posted on by
Reply

On July 7 I shot some video with the Nikon J1 of the joint CMASS/MMMSC launch at the Tuckahoe Turf Farm in South Berwick, ME.

After my generally scathing review of the camera (more for missed opportunity than anything else), I figured I’d give it a chance to show what it can do with high-speed photography – specifically, I wanted to try the 10 Mpixel 60 frames/second mode as well as the 400 and 1200 fps high-speed video modes.

Here is the result:

I put it together in Sony Vegas. The blurry clips were shot at 400 frames/second (240×640 pixels). The blurrier ones are at 1200 frames/second (120×320 pixels). The video is at 30 fps, giving 1/13.3x and 1/40x speed. This video shows the full resolution output by the camera.

(To be pedantic, playback is at 29.97003 Hz (that’s 30000/1001); from what I saw in Sony Vegas, the Nikon actually records at 399.6004 and 1198.8012 fps – which makes an odd sort of sense if you know NTSC.)

As you can see, all the video is lousy. It’s poorly exposed (despite some fixing in Vegas), heavily overcompressed (in-camera) and oversharpened (again, in-camera). The 1200 fps mode is worse than the already bad 400 fps mode. You can’t control it. I don’t blame Nikon too much – the high-speed Casio cameras seem to have similar problems. On the plus side, most of the video was shot at a shutter speed of 1/5000 second, which is neat to do. A couple clips were at 1/10,000th (!).

The video is good enough for some technical purposes, but it’s not a joy to look at.

Finally, you’ll note that none of the clips are at that fantastic, promised, 10 Mpixel resolution (60 Hz). It turns out that although the Nikon J1 will record stills that fast (for 1/2 second), you can’t control the shutter speed while it’s doing it. I didn’t know that until I got there and tried it. The shutter speed it picked (on a reasonably bright day) was so slow that each frame had lots of motion blur in it. So I didn’t bother. Just another needless firmware-based disappointment from the Nikon J1.

I’ve put the camera and lenses up for sale on eBay. Such a shame, Nikon. Oh well – I’m getting excited about the rumors of the new Canon mirrorless ILC system; maybe they’ll do better.

A nerdy review of the Nikon J1 camera

Posted on by
11

The Nikon J1 is unique and wonderful and frustrating and stupid. All at once. It’s the bastard child of a brilliant engineer and a retarded UI designer. It’s the best digicam ever built for using classic and exotic lenses, yet is deliberately crippled when you mount them. It’s the only digicam on the market that can do 60 frames/second at 10 MPixel resolution, yet it has only enough buffer to do it for half a second. And the user interface seems designed to deliberately frustrate.

It’s conflicted enough to make my head want to explode.

I’m just going to to talk about the J1 because that’s what I have. It’s sister camera the V1 is (even) more expensive and has an EVF and a higher resolution LCD screen but seems to be otherwise identical. So I assume it’s the same in all the things that delight and enrage.

Nikon J1 in candy apple red (credit: Nikon)

NIKON CX FORMAT HITS THE SPOT

Nikon has taken a lot of heat for their choice of a 1″ sensor for the CX format (Nikon’s name for their 1″ sensor size and mount); most reviewers think it’s too small.

I strongly disagree. I think it was a brilliant choice. Look at this table of sensor sizes:

Sensor Area (mm^2) Stops
Pentax 645 (medium format) 1452 3.6
35mm (“full frame”) 864 2.9
APS-C (Canon) 329 1.5
4/3 225 1.0
Nikon CX 116 0.0
1/1.7″ (Canon G12) 43 -1.4
1/2.3″ (Pentax Q) 28 -2.0

All else being equal, light sensitivity is proportional to the area of the sensor. There’s a big gap between sensor sizes of compact cameras (1/1.7″ and below) and “DSLR” sensors (4/3″ and bigger). The CX sensor size is smack in the middle. It allows lenses and cameras much smaller, lighter, and cheaper than do APS-C or larger formats, but with a vast image quality improvement over compact sensor cameras – a full 2 stops better than the (tiny sensor) Pentax Q.

Camera size is really important – if the camera is too bulky to take with you, you’re not going to get any pictures at all. Yet we want quality images, so we have to compromise.

When image quality is all-important, there are plenty of DSLRs on the market for that. And if size is all that matters, there are plenty of cameras the size of a deck of cards (as well as mobile phones with cameras). But the Nikon 1 offers a unique compromise.

Unfortunately Nikon didn’t exploit this very well when designing the (four) lenses available in CX format – they’re hardly any smaller than Micro 4/3 lenses. But that’s not the fault of the sensor choice. And lens design is far from Nikon’s worst mistake here.

FLANGE FOCAL DISTANCE AND LENSES

The 1″ sensor format is also a good match for the image circle produced by many classic cine and video lenses. Many really interesting lenses meant for 16mm cinema film, or for broadcast television sensors are available on eBay. And these lenses can be easily adapted to the CX mount, while still focusing to infinity, because the CX mount has the smallest flange focal distance of any digicam on the market (excepting the Pentax Q). At just 17.0 mm, any lens designed for a longer flange distance can be easily adapted: C-mount film and video lenses, the Fuji X mount lenses (X-Pro1; 17.7 mm), Sony E mount (NEX; 18.0 mm), Micro 4/3 (19.3 mm), Samsung NX (25.5 mm), RED ONE (27.3 mm), Leica M (27.8 mm), M39 (Leica screwmount; 28.8 mm), Contax G (38.7 mm), Canon FD and FL, Minolta SR, Canon EF-S and EF, Minolta/Sony A mount, M42, and Nikon F mount – just to name some of the more popular ones.

NIKON SHOOTS ITSELF IN THE FOOT

The ability to use all these different lenses is unique to the Nikon 1 (again, the Pentax Q excepted). But did Nikon make the J1/V1 an attractive platform for these lenses? No, just the opposite. The firmware turns off all modes except full manual when any manual lens is mounted. Even the light meter is disabled. Worse, when any manual lens is mounted, the “focus assist” mode – where the central part of the image is magnified to make critical manual focusing possible – is disabled. Yet it is precisely these manual-focus lenses that need this function!

Think about that. Some Nikon firmware engineer wrote code specifically to turn off the focus assist and light meter that were already there.

They’re selling a camera that can mount more classic and exotic lenses than any other in the world, but they deliberately spent effort to make it difficult to use those lenses. And this wasn’t done to “protect” some other, more expensive, camera that has that feature – they don’t offer one. WTF, Nikon? Are you trying to make this camera fail?

A BRILLIANT SENSOR BY APTINA

But let’s go back to the positives. The camera has what I think is the only phase-detect focus system on a mirrorless camera on the market; this lets it focus fast, even during video.

Nikon also made a brilliant choice in their sensor supplier – Aptina. The Aptina sensor has super high speed readout, which makes possible a fast all-electronic shutter with no moving parts. And that enables features no other camera on the market can touch:

  • Absolutely silent shutter operation
  • Shutter speeds as short as 1/16,000 second (freezes motion like no other camera!)
  • Full 10 MPixel output at 60 frames/second (!)
  • Smaller resolutions at up to 1200 fps
  • Almost no “rolling shutter” effect

Continue reading →

DIY conformal coating

Posted on by
3

I’ve been having some problems with nasty black somewhat conductive sooty stuff getting onto my PCBs. I decided the quickest solution was conformal coating to protect the PCB.

But I’ve never done that before and don’t know how. At first I thought of painting on epoxy with a paintbrush. But I was worried that once I coat the board it’ll be difficult to work on if I need to replace a component or something.

So I decided to try it with hot glue. Hot glue is removable, and unlike epoxy, isn’t brittle. And if I get some on the wrong place (like a connector pin or PCB button), I can take it off again.

Board fresh out of oven

I don’t really know if this is a good idea or not, but I tried it and it seems pretty OK so far.

First, I cleaned the sooty stuff off the PCB the best I could with 100% ethanol and a toothbrush. Then I baked it dry in a convection oven at 210 F for 30 minutes. It still worked.

Board in oven, above aluminum foil drip tray

Then I covered all the parts and traces with hot glue with an industrial-type glue gun (a hobby type gun would probably work too, but that’s what I have). I used Bostik type 0130 glue, which I think is pretty much the standard stuff you get in a hardware store.

I did my best to avoid covering connector pins and moving parts (buttons, piezo sounder, etc.).

The result was really ugly. Since the oven had worked well for drying the board, I thought I’d try to reflow the glue in the oven to let excess glue drip off and even out the glue.

It seemed to work pretty well. After experimenting with temperature a little, I decided 260 F for half an hour was about right. I made a drip catcher out of aluminum foil and put that under the board.

Bottom of board after reflow in oven. Note the drips.

Once it cooled off, it was really difficult to get the board off the oven rack (that glue works well).

And I still haven’t found a great way to clean the dripped glue off the oven racks. The best I could do was reheat the oven to about 300 F and then wipe off the glue with paper towels while it’s hot (carefully!). (My wife never reads my blog postings anyway…)

Next time I think I’ll remove all but one oven rack first, and hang the board from that rack so no glue gets on the rack.

Bottom of board after unsticking it from the oven rack

About date formats

Posted on by
3

The general problem with numeric date formats is this – what does “04.07” mean? Is it July 4th or April 7th? Or April 2007? Maybe July 2004?

The answer is it’s ambiguous unless you have some other clues.  In general the separator is the clue; a dot (.) means it’s in Euro order (Day;Month;Year).  A slash (/) means it’s in USA order (Month;Day;Year) and a dash (-) means it’s in international (also Asian) order (Year;Month;Day).

So in general – don’t use MM.DD.YY date format ever.  That is confusing because DD.MM.YY (with dots) is one of the standard date formats in Europe, and MM/DD/YY (with slashes) is the most common format in the USA.  So using dots (European) with the USA order Month-Day-Year is doubly confusing.

If you must use USA order, use slashes (/) to help identify it.  If you’re going to use a Euro format, use the one that spells out the month (20 Jul 1969); at least it’s unambiguous.

Best of all is to use the ISO 8601 international format: YYYY-MM-DD.

For example, use 1776-07-04.

The dashes and the 4-digit year coming first indicate clearly that it’s in neither the Euro formats (04.07.76 or 4 July 1776) nor the USA formats (7/4/76 or July 4, 1776).

The ISO international format also has the advantage that it sorts correctly in a computer (dates will get sorted chronologically); neither the Euro nor USA formats do that.

The ISO format also happens to be the same format used in Japan & China, but that’s just a coincidence.

In general writing (when sorting order doesn’t matter) you can avoid the whole problem if you spell out the date instead of using numbers:

4 July 1776
July 4, 1776

These are both completely clear and unambiguous.

The only good patent is an expired patent

Posted on by
Reply

Update – 2024-05-09:

In the 14 years since I posted this, I’ve changed my mind. Without patent protection against low-effort knock-offs, it can be really difficult for small firms to get VC funding. At the very least, I overstated the case.

Somewhere else I’ve said “Everything is more complicated than it seems.”. This is one of those.

…indeed the best thing about patents is that they eventually expire. (And I say this as an inventor with multiple issued patents.)

See, for instance, http://news.cnet.com/8301-10784_3-57374941-87/litigation-lunacy-silicon-valleys-lost-its-collective-mind/?part=rss&subj=news&tag=2547-1_3-0-20.

Or http://en.wikipedia.org/wiki/Nest_Labs#Litigation.

And in general: http://www.techdirt.com/.

Having said that, I guess I have to also say that I’m not against patents in principle, only in practice.

I usually try to stay out of politics on this blog, but this affects us nerds in particular.

If we could somehow have a working patent system that limited patents to truly original and (above all,) non-obvious inventions, ideally ones that involved genuine investment (instead of off-the-cuff ideas), then I’d be in favor of that. But the current system is supposed to do that already, and fails miserably.

The result is worse than no patent system at all. Ironically, the current patent system, which is supposed to encourage innovation, instead stifles it – the risk of company-killing lawsuits over genuinely independent inventions (and therefore, in my book, obvious ones) far outweighs any encouragement.

In my view, to qualify as non-obvious, an applicant should be required to show that her invention solves a long-standing (not recent) problem which other people have had ample opportunity to solve, but have been unable to. Too many modern patents are obvious solutions to new problems which either never existed before (because a new technology raises new problems) or which only recently became solvable because of new technology. For example, sending voice over the Internet is obvious once you have an Internet. Nobody should get a patent on that just because they were the first, as this is obvious. Heatsinking a LED for domestic lighting is obvious – no patent should issue simply on the basis that nobody did it before, as that is because LED use for domestic lighting is a new application, and therefore the problem never came up before.

Apparently somebody was issued a patent on using a rotating mirror to scan a laser in a polymer 3D printer – because it’s a “new application” of the invention of the rotating mirror. There’s at least one guy who therefore is using a rotating prism instead, to work around the patent. In his own words “even though this would have been obvious for any bachelor physics student looking into this topic”. Using a rotating mirror (or prism) this way is not in any sense “invention” – it’s workaday engineering.

Solutions should be considered obvious if they appear very quickly after appearance of the problem, or if multiple independent “inventors” come up with the same solution over a short period of time.

While I’d prefer real reform of the patent system along these lines – which would reject 98%+ of currently issued patents (including most of mine) – political reality seems to make that unlikely. Given the choice between the current system and no patent system at all, I’d choose none.

Simple driver code for Microchip MRF24J40 radio

Posted on by
82

Update February 2017:

User “actondev” on the Microchip Forum has posted a version for the PIC16. I haven’t tested it.

Click this to download it: MRF24J40_actondev_forPIC16.zip.

Update November 2016:

I now have this working with the MRF24J40MD module. No changes at all were needed from the code for the MRF24J40MB.

Back in September 2011 I wrote about the rocket telemetry system I built using the Microchip MRF24J40MB radio module.

As I mentioned back then, I ended up re-writing Microchip’s “MiWi P2P” stack to vastly simplify it for my application. A few people have asked for a copy of my simplified driver code, and today I’m posting it (after having cleaned up a few loose ends).

The radio supports IEEE 802.15.4 on the 2.4 GHz ISM band. The “-MB” is Microchip’s long-range module – it has the radio, antenna, power amplifier (PA), and low noise amplifier (LNA), and it’s good for ranges of 2500 meters or more (line-of-sight, outdoors). It comes pre-approved by the FCC for unlicensed use. You can read my old post for more info about the module.

File package

To get started, download the ZIP file from here.

Update 2012-03: “maxruben” from the Microchip Forum has ported this driver to the PIC24. Click here to download his PIC24 version. I haven’t tested it, but he says it works. (Thanks for permission to post this!)

I’m also told the code has been successfully ported to the PIC18, but I don’t have a copy of that. (If someone sends me one, I’ll post it here.)

This contains a whole buildable project (for MPLAB IDE v8.83) that works on my PIC32MX440-based platform.  I’ve built it with C32 v1.11b and v2.02; it should be trivial to port it to other MCUs (see notes about that below).

The files include a very short demo program in main.c that shows how to use the driver to send and receive simple packets.

The driver itself consists of 3 files:

MRF24J40.c – Driver source code.
MRF24J40.h – Headers and public function declarations.
radioAddress.h – Sets the address for your radio.

Now would be a good time to unzip the files and have a quick look at the source code.

Oh, and:

Software rights: I hereby grant everyone and everything in the universe permission – to the extent I have rights to grant – to use and modify this software for any purpose whatsoever. In exchange, you agree not to sue me about it. I make no promises. By using the design you agree that if you’re unhappy the most I owe you is what you paid me (zip). That seems fair.

Be aware that the original MiWi P2P v3.1.3 code this started from (of which there might not be any code left) was copyrighted by Microchip – they offer it free for use with their hardware (which is all it’s useful for), and I doubt they’d want to sue their own customers over it, but talk to them if you have concerns.

That having been said, if you work for Microchip (this means you, Yifeng) and find this code useful enough to refer it to customers, or if you want to supply it directly, you are very welcome to do so. I’d appreciate (but do not demand) in return (a) credit in the source code to this posting on NerdFever.com, and (b) a small token of your appreciation. A RealICE would be great. If that’s too much, how about a Microchip T-shirt or coffee mug? (I already have a ICD3 and a Microchip bag; but swag is good. You know how to find me.)

Update: A free RealICE arrived about 2 weeks after I posted this.  Thank you, Microchip (and Yifeng)!

General concept – Initialization

Call RadioInit() to initialize the radio.

General concept – Transmission

To transmit a packet, fill out the “Tx” structure to describe the packet, then call RadioTxPacket().

General concept – Reception

Call RadioRXPacket(). If there is at least one received packet the return value is non-zero (it returns the number of un-processed received packets) and the next packet is described by structure “Rx”.

Do what you want with the packet, and then call RadioDiscardPacket() to throw away that packet. Now you can call RadioRXPacket() again to get the next one (if any).

API – Transmitting

void RadioTXRaw(void);

Low-level routine to transmit a single packet described by Tx.

The Tx structure must be completely setup before calling this routine. (Don’t set the lqi or rssi fields; these are used only on received packets.)
Continue reading →

Notes on using Eagle PCB

Posted on by
1

These (link to PDF file) [updated 2012-06] are my notes for using Eagle PCB, an excellent CAD program for printed circuit board design with a free (but still fully functional) “light” version.  Most of the text is my own, but there are places where I’ve cut-and-pasted information from the Eagle documentation or material found online.

To go with the notes, here is a .ZIP file containing my customized EAGLE.SCR file (sets defaults on starting Eagle), default.dru (defaults for routing), and Dave-gerb274x(2layer).cam (for making gerber files; works fine with BatchPCB).

The notes are not meant as a replacement for the Eagle documentation, but rather as commentary and interpretation.

I’ve been using BatchPCB.com to manufacture boards; their design rules influence some of the notes.

I find Eagle’s user interface counter-intuitive in many ways.  It’s powerful, but the assumptions differ a lot from common GUI programs like MS Office and Photoshop.  The fact that the manual is a poor translation from German doesn’t help. So I have to refer to these notes to refresh my memory about how Eagle works after not using it for a while.

I make no promises that this material is accurate; I write what I learn but sometimes I may misunderstand.  Feedback with corrections and improvements are welcome; please post them on NerdFever.com or email to dave@nerdfever.REMOVE_THIS.com.

Mystery of the stainless steel pinhole

Posted on by
4

Here’s a puzzle; maybe some of my readers can figure it out.

My wife showed me a tiny pinhole that appeared in one of our stainless steel pots. How did this happen? What mechanism could have created the hole?

I took one look and said “cosmic ray hit”. But I was joking.

The pot is a few years old, but has never been used for anything unusual – just a little cooking and storage of food in a fridge.

The hole is just above the penny – you can see the light shining thru it (click on the image for a larger version).

It’s fairly thin steel; about half a millimeter.

Here’s a close-up photo of the hole from the inside:

The hole is about 0.2 millimeters in diameter (the crater is larger).

Here it is from the outside:

So it looks like whatever happened started from the inside.

What did happen?