Month: December 2014

Migrating From Gmail

One of the projects I wanted to undertake over the 2014 Christmas holiday was to migrate the last bits of my email away from Google’s Gmail service. There were several reasons for this: I’ve mostly been using the email service on my own webhost for the past few years, Gmail was simply filling up with Spam, I wanted to actually make a local backup of my emails, and I don’t really care much for having my emails mechanically analyzed for advertising content.

I feel a bit guilty, too. As I’ve been reading through my old, old emails, I keep running across emails saying “So and so accepted your invitation to join Gmail.” (Remember when Gmail was invite-only?)

Anyway, there were two ways of migrating the email that I am documenting here. I’ll call them The Slow Way and The Fast Way.

The Slow Way

The Slow Way is the easier of the two ways, but it probably would take far too long to be practical unless you have an extraordinary amount of patience. It involves simply setting up your Gmail account and your other email account using Mozilla Thunderbird and then using the context menus to copy emails between the two accounts.

You select the emails from Gmail that you want to move, then right-click and select the folder on your other email account where you want the messages to go, which looks something like this:

slow-way-thunderbird-context-menu

And Thunderbird will then diligently download the messages from Gmail and then upload them to your other email account. This is fine for normal text messages, which tend to have small amounts of data, but this totally takes too long when moving messages where your non-smartphone-literate friends inconsiderately send raw 8 megapixel images of whatever. (And noticeably, actually, this only becomes a problem around 2012 or so, when everyone finally bought a smartphone.)

If you’re moving an entire mail account from one host to another, it will probably take all night to complete.

So I needed to look for something faster, a way to bulk transfer all of the data out of Gmail to my mail host, and then use Thunderbird to organize the messages locally between IMAP mail folders on the same mail server.

The Fast Way (More Technical)

The Fast Way is the harder of the two ways and requires pretty good knowledge of UNIX / Linux. It involves using a piece of software called OfflineIMAP, which is a program written in Python that can run on a webhost somewhere (and therefore has access to a very fast network connection).

I made a clone of the OfflineIMAP git repository like so:

git clone git://github.com/spaetz/offlineimap.git

Then I set up a migration plan, that looked something like this:

The important bits are the folderfilter and nametrans settings:

  • folderfilter determines which Gmail folders you want synchronized to your mail host. In this case, I specified “Commerce” because I wanted to synchronize a single folder containing very little content, before doing the whole INBOX migration.
  • nametrans translates the names of the folders you want synchronized, and allows you to prefix them or modify them in any way you’d like before committing them to the target mail host. In this case, all of the folders that are transferred will be prefixed with “Gmail.”

So in this case, after running the OfflineIMAP program, a new folder should appear in my mail host called “Gmail.Commerce” which contains the same emails as the “Commerce” folder on the Gmail mail host.

If you remove the folderfilter setting, OfflineIMAP should synchronize every folder on Gmail to your mail host. (Probably not exactly what you want.)

Then I ran offlineimap.py, which will prompt you for the passwords to the two email accounts before proceeding:

$ python2.7 bin/offlineimap.py
OfflineIMAP 6.5.4
  Licensed under the GNU GPL v2+ (v2 or any later version)
Account sync Migrate:
 *** Processing account Migrate
 Establishing connection to imap.googlemail.com:993
Enter password for account 'Gmail':
 Establishing connection to mail.webfaction.com:993
Enter password for account 'WebFaction':
 Creating folder Gmail.Commerce[WebFaction]
Folder Commerce [acc: Migrate]:
 Syncing Commerce: IMAP -> MappedIMAP
 Copy message 1 (1 of 16) Gmail:Commerce -> WebFaction
 Copy message 2 (2 of 16) Gmail:Commerce -> WebFaction
 Copy message 3 (3 of 16) Gmail:Commerce -> WebFaction
 Copy message 4 (4 of 16) Gmail:Commerce -> WebFaction
 Copy message 5 (5 of 16) Gmail:Commerce -> WebFaction
 Copy message 6 (6 of 16) Gmail:Commerce -> WebFaction
 Copy message 7 (7 of 16) Gmail:Commerce -> WebFaction
 Copy message 8 (8 of 16) Gmail:Commerce -> WebFaction
 Copy message 9 (9 of 16) Gmail:Commerce -> WebFaction
 Copy message 10 (10 of 16) Gmail:Commerce -> WebFaction
 Copy message 11 (11 of 16) Gmail:Commerce -> WebFaction
 Copy message 12 (12 of 16) Gmail:Commerce -> WebFaction
 Copy message 13 (13 of 16) Gmail:Commerce -> WebFaction
 Copy message 14 (14 of 16) Gmail:Commerce -> WebFaction
 Copy message 15 (15 of 16) Gmail:Commerce -> WebFaction
 Copy message 16 (16 of 16) Gmail:Commerce -> WebFaction
Account sync Migrate:
 *** Finished account 'Migrate' in 0:19

Once OfflineIMAP is finished, you can go back to Thunderbird and look for the newly synced folders. However, they will not appear automatically. You have to subscribe to them first, by opening the context menu for your mail host settings and selecting the Subscribe menu option:

fast-way-subscribe-1

fast-way-subscribe-2

Thunderbird will then retrieve the message headers when you open the newly-subscribed folder, and everything will just work.

To synchronize the Gmail INBOX and [Gmail]/Sent Mail folders to your IMAP mail host, you change the .offlineimaprc file slightly so that the Gmail mail host configuration looks like:

When it is synchronizing the folders, OfflineIMAP will now print out tons of messages that look like this, as it copies the thousands of messages:

[...]
Folder INBOX [acc: Migrate]:
 Copy message 8302 (39 of 5322) Gmail:INBOX -> WebFaction
Folder [Gmail]/Sent Mail [acc: Migrate]:
 Copy message 42 (42 of 4045) Gmail:[Gmail]/Sent Mail -> WebFaction
Folder INBOX [acc: Migrate]:
 Copy message 10941 (40 of 5322) Gmail:INBOX -> WebFaction
Folder [Gmail]/Sent Mail [acc: Migrate]:
 Copy message 43 (43 of 4045) Gmail:[Gmail]/Sent Mail -> WebFaction
[...]

Now you just have to subscribe to these newly-synchronized mail folders in Thunderbird, and you should be good to go:

fast-way-subscribe-3

One the long process of copying the thousands of messages over to your mail host is complete, you can pretty much wipe out the messages on the Gmail side and be done with it.

The other wild thing that OfflineIMAP enables is that if you now delete messages from your local mail host, it will synchronize these changes back up to Gmail. So you can essentially pool all of your various IMAP mail feeds into a single place for management, and OfflineIMAP will make sure that any flag changes or message deletions will make their way back to the source server. But that’s a topic for another day.

Troubleshooting

A bunch of times, when I ran OfflineIMAP, the following error would pop up:

$ offlineimap.py
OfflineIMAP 6.5.4
  Licensed under the GNU GPL v2+ (v2 or any later version)
Account sync Migrate:
 *** Processing account Migrate
 Establishing connection to imap.googlemail.com:993
Enter password for account 'Gmail':
 Establishing connection to mail.webfaction.com:993
Enter password for account 'WebFaction':
 ERROR: While attempting to sync account 'Migrate'
  zero length field name in format
 *** Finished account 'Migrate' in 0:11
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'Migrate'
  zero length field name in format

Turns out that when this “zero length field name in format” pops up, you’re running the script using Python 2.6, and you need to make sure you’re explicitly running the Python 2.7 interpreter, so just run it as python2.7 offlineimap.py.

Disassembling Guinness Keg

So disassembling a Guinness keg is not the most common thing to do in the universe, but I had to do it this weekend. Guinness kegs use a different tap style than the standard European taps. It can be a bit daunting at first, but the steps are:

  1. Lay the keg on its side, take a screwdriver and press the valve stem center to release any leftover pressure in the keg.
  2. Drill holes where you can use a countersink punch and a hammer to smack the crap out of the retention ring, smacking the ring in the counterclockwise direction. (This may not strictly be necessary, but I needed the punch to get some good hits in.)
  3. As you are smacking the ring, you should start seeing it move slightly. In my case, the ring moved a few millimeters at a time, before loosening completely.
  4. Once the retention ring is out, I used the pliers to grab the valve stem in the center, and pulled the whole valve upwards.
  5. The whole valve assembly will get caught, you have to rotate it while pulling, until the metal nub in the valve assembly lines up with the notch in the neck of the keg. Once aligned, the whole valve assembly will slide right out.

Here’s what it looks like:

guinness-keg-disassembly-01

Drill holes where you can use a countersink punch and a hammer to smack the crap out of the retention ring, smacking the ring in the counterclockwise direction.

As you are smacking the ring, you should start seeing it move slightly. In my case, the ring moved a few millimeters at a time, before loosening completely.

guinness-keg-disassembly-05

Nevermind the wire I ran through the holes I drilled (you don’t have to drill them all the way through, either). I was trying to run a stick through the wire and then unscrew the retention ring that way (kind of like a tourniquet). But it didn’t work, so I reverted to smacking it with a hammer some more, which did work.

guinness-keg-disassembly-06

guinness-keg-disassembly-07

Once the retention ring is out, I used the pliers to grab the valve stem in the center, and pulled the whole valve upwards.

The whole valve assembly will get caught, you have to rotate it while pulling, until the metal nub in the valve assembly...

...lines up with the notch in the neck of the keg.

Once aligned, the whole valve assembly will slide right out.

Berlin, Bullshit Startup Capital

For the second time in a year, a friend of mine has been unceremoniously fired from a job at a startup in Berlin. Without the two weeks of notice that was in her contract. No, in fact, it was a firing from one day to the next, which is highly unusual in Germany. The fact that her firing occurred one day before the end of her 6-month probationary period and several weeks after the founders had successfully attracted fresh investment was sketchy as hell, but not out of the ordinary for Berlin (at least not anymore).

When she asked one of the founders why the notice had been so late and so abrupt, his “excuse” was that they’d been meaning to fire her sooner, but had gotten too busy with the investor search. Gee, thanks. It takes a particularly clueless person to both admit that they were exploiting you for their benefit and to think that it was a reason you would find acceptable.

It also takes a particular breed of asshole to say that my friend wasn’t sufficiently devoted to the success of the startup, after she spent a year earning a 450 EUR/month pittance wage working a full-time “internship” for them. (And didn’t leave them after getting the full-time “permanent” position, with heaps of mandatory unpaid overtime, because she was tied to the startup due to visa restrictions.)

It’s an open secret in Berlin that companies will talk up the chance of a full-time position, or a working visa sponsorship, or some promotion in the nondescript future when funding arrives, or the “valuable” experience you will gain, as a way to keep you on the hook, right until they pull the hook out and let you go. Nevermind equity, you will never see it offered, and no one here thinks to ask.

So what was the reason given for my friend’s termination? None, except, uhh, well actually one of the founders thought he would be a better lead designer for the firm, even though he has no design experience besides having attended a single Google design workshop after the new Material Design guidelines release. (Someone’s been drinking the flat-design, everything-looks-like-a-Google-App Kool Aid, without thinking about the importance of design or consistency in unique branding and positioning.)

No one’s reaction in the company was positive, upon hearing this news.

If I were an investor, and I knew that your pitch deck was full of mostly bullshit anyway about your conversion rate, retention rate, monthly active users, ARPU, CAC, burn rate, KPIs, and whatever, and that you just fired the only competent designer in your company, and that you were about to load up one of the cofounders with lead design duties (not delegated), I sure as hell would not invest a cent in your dipshit startup.

But this is Berlin, a city full of moron investors with more money than sense, and a negative interest rate policy in the Eurozone that all but assures misallocation of capital from zero yield to any idea you can write down on a napkin in the next coffee shop at the (one of the five) corners of Rosenthaler Platz. As long as it’s Other People’s Money you’re losing, who cares? Let’s go all-in on another sentiment analysis company, and build it with other people’s unpaid overtime! Let’s build another company with an entirely assailable business model in a saturated market! Let’s close the books on this investment fund and rollover into another, newer, fresher, better, investment fund with fresh suckers from Southern Germany!

This reminds me a bit of my brief stint in Startupland here (which also ended with me being cheated out of a fair sum of money by two very unscrupulous people). One of the pitch decks the cofounders I worked with forwarded to potential investors made it all sound so good. A solid team, solid technology, a real chance at capturing a market niche. Nevermind that the entire technology stack had been built by an endless stream of inexperienced undergraduate interns. No one ever came to talk to us, to do any due diligence, to figure out if we knew what we were talking about. I started questioning who was running the show at these various investment funds. Did they have people who knew technology? Did they have people who knew the right questions to ask? It certainly didn’t seem so.

I remember once looking over a webpage full of startup mentor biographies, at one of the startup incubators in town, with some other founders: “Useless, useless, useless, good, sketchy, useless, loudmouth, this guy almost cheated us out of a huge amount of equity,” and on it went as we looked at the biographies of mostly people with zero technical background, who were offering their services to technical startups. Of course all of this backstory information is hidden, you can’t read it in the bio statements, you can’t read it on the mentors’ faces, it’s information stored in people, and people aren’t talking enough.

If there’s one thing I’ve learned (or should I say, been taught) this year, it is this: The line between a decent person and a scoundrel is very thin and easily crossed.

So this is the situation Berlin finds itself in. The boom times are good, as long as interest rates don’t rise. And as long as the boom times are good, there will be asshole founders firing the few people in their companies who have clue.

Good luck with that.

Notes

To Google:

Hangouts, on an Android smartphone, should not keep trying to replace the SMS functionality by default. This is really pissing me off. I didn’t buy a phone to route all of my messages through Google, and I didn’t buy a phone to have Google misdeliver messages to others or misdeliver them to me.

Case in point: A friend of mine sent me a Hangouts messages today, which I received several hours later when I reconnected to WiFi. But I guess I must have appeared Online to them for whatever reason, even though the message never made it through. Thanks for making it look like I was ignoring them.

To Ebay:

Fix your damn service so it has two-factor authentication already.

Fix your Android App so that it actually has useful offline alarm notifications. I’ve missed so many ending auctions because the App doesn’t notify me that an auction is ending in 10 minutes. But, again, when I connect to WiFi again, it tells me I missed the end of the auction. Thanks a lot guys. And well done on not engaging your users.