Migrating from offlineimap to mbsync for mu4e

I have just switched over from offlineimap to mbsync (part of isync) for synchronising my local email with Gmail for use with mu4e. I found offlineimap worked well enough but it would hang frequently if I had connection problems (e.g. on my laptop) and mbsync is much faster. I couldn’t find a guide for migrating, and I encountered a few issues along the way, so I thought I’d document the process here.

The isync mailing list has some discussion of how to convert the maildir folders and related files from offlineimap to mbsync format, but I didn’t see any examples of this being done successfully so I decided to play it safe and sync a new maildir directory with mbsync, downloading a new copy of all of my emails from Gmail’s servers.

I’m using a Mac, and built the newest version of isync from source without problems (the version on Mac Ports is older). I based my mbsyncrc file on the example on this page, and set up the structure of the maildir to match the one created by offlineimap (not really needed but makes things simpler). Here is my mbsyncrc

# mbsyncrc based on
# http://www.ict4g.net/adolfo/notes/2014/12/27/EmacsIMAP.html
# ACCOUNT INFORMATION
IMAPAccount gmail
# Address to connect to
Host imap.gmail.com
User name@address.com
Pass ************
AuthMechs LOGIN
SSLType IMAPS
SSLVersions SSLv3
CertificateFile /opt/local/share/curl/curl-ca-bundle.crt

# THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE
# - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE
#   SPECIFICATION OF AN IMAP ACCOUNT)
# - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER

# REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE)
IMAPStore gmail-remote
Account gmail

# LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail)
MaildirStore gmail-local
Path ~/mbsync/
Inbox ~/mbsync/INBOX

# CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS
#
# CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl
# FOLDERS. SOME COMMONLY USED PATTERS INCLUDE:
#
# 1 "*" TO MATCH EVERYTHING
# 2 "!DIR" TO EXCLUDE "DIR"
# 3 "DIR" TO MATCH DIR

Channel gmail-inbox
Master :gmail-remote:
Slave :gmail-local:
Patterns "INBOX"
Create Both
Expunge Both
SyncState *

Channel gmail-trash
Master :gmail-remote:"[Gmail]/Bin"
Slave :gmail-local:"[Gmail].Bin"
Create Both
Expunge Both
SyncState *

Channel gmail-sent
Master :gmail-remote:"[Gmail]/Sent Mail"
Slave :gmail-local:"[Gmail].Sent Mail"
Create Both
Expunge Both
SyncState *

Channel gmail-all
Master :gmail-remote:"[Gmail]/All Mail"
Slave :gmail-local:"[Gmail].All Mail"
Create Both
Expunge Both
SyncState *

Channel gmail-starred
Master :gmail-remote:"[Gmail]/Starred"
Slave :gmail-local:"[Gmail].Starred"
Create Both
Expunge Both
SyncState *

# GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE
# MBSYNC ON A GROUP TO SYNC ALL CHANNELS
#
# FOR INSTANCE: "mbsync gmail" GETS MAIL FROM
# "gmail-inbox", "gmail-sent", and "gmail-trash"
#
Group gmail
Channel gmail-inbox
Channel gmail-sent
Channel gmail-trash
Channel gmail-all
Channel gmail-starred

Now I synced with Gmail using

mbsync -V gmail

which worked fine for a while but then kept returning errors from Gmail about throttling and my quota being exceeded. My email is about 6GB in size, so large but not ridiculous, but it seems google didn’t like me trying to download it all in one go (possibly a limitation of my employer’s corporate google service). Anyway, I checked with the isync mailing list and there should be no problems with syncs being interrupted and resumed so it would be possible to keep repeating the sync until it completed. Instead of that I used Apple’s network link conditioner (part of Hardware IO Tools for Xcode) to limit my bandwidth to 750 kbps and left the sync to run overnight, which completed without problems.

Once this was done, I had my offlineimap copy of my maildirs in ~/offlineimap and my mbsync maildirs in ~/mbsync. To make it easy to switch between them if needed, I made a symbolic link from ~/Maildir to the mbsync maildir.

Next I deleted my mu index and re-indexed

rm -rf ~/.mu
mu index

I then set some mu4e variables to point to my new maildir

;;location of my maildir
(setq mu4e-maildir (expand-file-name "~/Maildir"))

;;command used to get mail
;; use this for testing
(setq mu4e-get-mail-command "true")
;; use this to sync with mbsync
;;(setq mu4e-get-mail-command "mbsync gmail")

;;rename files when moving
;;NEEDED FOR MBSYNC
(setq mu4e-change-filenames-when-moving t)

;;set up queue for offline email
;;use mu mkdir  ~/Maildir/queue to set up first
(setq smtpmail-queue-mail nil  ;; start in normal mode
      smtpmail-queue-dir   "~/Maildir/queue/cur")

Now, after restarting mu4e, I was using the mbsync maildir.

Note that I initially used a dummy command to sync the mail in mu4e so that I could run the sync manually on the command line at first. I really only needed this because I kept hitting problems with duplicate UIDs, which are solved by getting mu4e to rename files when moving them, as in the code above.

The switch over has been very smooth. In fact I was surprised to find that the message IDs were preserved so that the org-links I stored to emails when I was using offlineimap still take me to the correct email!