Migrating From Gmail

One of the projects I wanted to undertake over the 2014 Christmas holiday was to migrate the last bits of my email away from Google’s Gmail service. There were several reasons for this: I’ve mostly been using the email service on my own webhost for the past few years, Gmail was simply filling up with Spam, I wanted to actually make a local backup of my emails, and I don’t really care much for having my emails mechanically analyzed for advertising content.

I feel a bit guilty, too. As I’ve been reading through my old, old emails, I keep running across emails saying “So and so accepted your invitation to join Gmail.” (Remember when Gmail was invite-only?)

Anyway, there were two ways of migrating the email that I am documenting here. I’ll call them The Slow Way and The Fast Way.

The Slow Way

The Slow Way is the easier of the two ways, but it probably would take far too long to be practical unless you have an extraordinary amount of patience. It involves simply setting up your Gmail account and your other email account using Mozilla Thunderbird and then using the context menus to copy emails between the two accounts.

You select the emails from Gmail that you want to move, then right-click and select the folder on your other email account where you want the messages to go, which looks something like this:

And Thunderbird will then diligently download the messages from Gmail and then upload them to your other email account. This is fine for normal text messages, which tend to have small amounts of data, but this totally takes too long when moving messages where your non-smartphone-literate friends inconsiderately send raw 8 megapixel images of whatever. (And noticeably, actually, this only becomes a problem around 2012 or so, when everyone finally bought a smartphone.)

If you’re moving an entire mail account from one host to another, it will probably take all night to complete.

So I needed to look for something faster, a way to bulk transfer all of the data out of Gmail to my mail host, and then use Thunderbird to organize the messages locally between IMAP mail folders on the same mail server.

The Fast Way (More Technical)

The Fast Way is the harder of the two ways and requires pretty good knowledge of UNIX / Linux. It involves using a piece of software called OfflineIMAP, which is a program written in Python that can run on a webhost somewhere (and therefore has access to a very fast network connection).

I made a clone of the OfflineIMAP git repository like so:

git clone git://github.com/spaetz/offlineimap.git

Then I set up a migration plan, that looked something like this:

The important bits are the folderfilter and nametrans settings:

• folderfilter determines which Gmail folders you want synchronized to your mail host. In this case, I specified “Commerce” because I wanted to synchronize a single folder containing very little content, before doing the whole INBOX migration.
• nametrans translates the names of the folders you want synchronized, and allows you to prefix them or modify them in any way you’d like before committing them to the target mail host. In this case, all of the folders that are transferred will be prefixed with “Gmail.”

So in this case, after running the OfflineIMAP program, a new folder should appear in my mail host called “Gmail.Commerce” which contains the same emails as the “Commerce” folder on the Gmail mail host.

If you remove the folderfilter setting, OfflineIMAP should synchronize every folder on Gmail to your mail host. (Probably not exactly what you want.)

Then I ran offlineimap.py, which will prompt you for the passwords to the two email accounts before proceeding:

$ python2.7 bin/offlineimap.py
OfflineIMAP 6.5.4
  Licensed under the GNU GPL v2+ (v2 or any later version)
Account sync Migrate:
 *** Processing account Migrate
 Establishing connection to imap.googlemail.com:993
Enter password for account 'Gmail':
 Establishing connection to mail.webfaction.com:993
Enter password for account 'WebFaction':
 Creating folder Gmail.Commerce[WebFaction]
Folder Commerce [acc: Migrate]:
 Syncing Commerce: IMAP -> MappedIMAP
 Copy message 1 (1 of 16) Gmail:Commerce -> WebFaction
 Copy message 2 (2 of 16) Gmail:Commerce -> WebFaction
 Copy message 3 (3 of 16) Gmail:Commerce -> WebFaction
 Copy message 4 (4 of 16) Gmail:Commerce -> WebFaction
 Copy message 5 (5 of 16) Gmail:Commerce -> WebFaction
 Copy message 6 (6 of 16) Gmail:Commerce -> WebFaction
 Copy message 7 (7 of 16) Gmail:Commerce -> WebFaction
 Copy message 8 (8 of 16) Gmail:Commerce -> WebFaction
 Copy message 9 (9 of 16) Gmail:Commerce -> WebFaction
 Copy message 10 (10 of 16) Gmail:Commerce -> WebFaction
 Copy message 11 (11 of 16) Gmail:Commerce -> WebFaction
 Copy message 12 (12 of 16) Gmail:Commerce -> WebFaction
 Copy message 13 (13 of 16) Gmail:Commerce -> WebFaction
 Copy message 14 (14 of 16) Gmail:Commerce -> WebFaction
 Copy message 15 (15 of 16) Gmail:Commerce -> WebFaction
 Copy message 16 (16 of 16) Gmail:Commerce -> WebFaction
Account sync Migrate:
 *** Finished account 'Migrate' in 0:19

Once OfflineIMAP is finished, you can go back to Thunderbird and look for the newly synced folders. However, they will not appear automatically. You have to subscribe to them first, by opening the context menu for your mail host settings and selecting the Subscribe menu option:

Thunderbird will then retrieve the message headers when you open the newly-subscribed folder, and everything will just work.

To synchronize the Gmail INBOX and [Gmail]/Sent Mail folders to your IMAP mail host, you change the .offlineimaprc file slightly so that the Gmail mail host configuration looks like:

When it is synchronizing the folders, OfflineIMAP will now print out tons of messages that look like this, as it copies the thousands of messages:

[...]
Folder INBOX [acc: Migrate]:
 Copy message 8302 (39 of 5322) Gmail:INBOX -> WebFaction
Folder [Gmail]/Sent Mail [acc: Migrate]:
 Copy message 42 (42 of 4045) Gmail:[Gmail]/Sent Mail -> WebFaction
Folder INBOX [acc: Migrate]:
 Copy message 10941 (40 of 5322) Gmail:INBOX -> WebFaction
Folder [Gmail]/Sent Mail [acc: Migrate]:
 Copy message 43 (43 of 4045) Gmail:[Gmail]/Sent Mail -> WebFaction
[...]

Now you just have to subscribe to these newly-synchronized mail folders in Thunderbird, and you should be good to go:

One the long process of copying the thousands of messages over to your mail host is complete, you can pretty much wipe out the messages on the Gmail side and be done with it.

The other wild thing that OfflineIMAP enables is that if you now delete messages from your local mail host, it will synchronize these changes back up to Gmail. So you can essentially pool all of your various IMAP mail feeds into a single place for management, and OfflineIMAP will make sure that any flag changes or message deletions will make their way back to the source server. But that’s a topic for another day.

Troubleshooting

A bunch of times, when I ran OfflineIMAP, the following error would pop up:

$ offlineimap.py
OfflineIMAP 6.5.4
  Licensed under the GNU GPL v2+ (v2 or any later version)
Account sync Migrate:
 *** Processing account Migrate
 Establishing connection to imap.googlemail.com:993
Enter password for account 'Gmail':
 Establishing connection to mail.webfaction.com:993
Enter password for account 'WebFaction':
 ERROR: While attempting to sync account 'Migrate'
  zero length field name in format
 *** Finished account 'Migrate' in 0:11
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'Migrate'
  zero length field name in format

Turns out that when this “zero length field name in format” pops up, you’re running the script using Python 2.6, and you need to make sure you’re explicitly running the Python 2.7 interpreter, so just run it as python2.7 offlineimap.py.