Migrating from offlineimap to mbsync for mu4e

I have just switched over from offlineimap to mbsync (part of isync) for synchronising my local email with Gmail for use with mu4e. I found offlineimap worked well enough but it would hang frequently if I had connection problems (e.g. on my laptop) and mbsync is much faster. I couldn’t find a guide for migrating, and I encountered a few issues along the way, so I thought I’d document the process here.

The isync mailing list has some discussion of how to convert the maildir folders and related files from offlineimap to mbsync format, but I didn’t see any examples of this being done successfully so I decided to play it safe and sync a new maildir directory with mbsync, downloading a new copy of all of my emails from Gmail’s servers.

I’m using a Mac, and built the newest version of isync from source without problems (the version on Mac Ports is older). I based my mbsyncrc file on the example on this page, and set up the structure of the maildir to match the one created by offlineimap (not really needed but makes things simpler). Here is my mbsyncrc

# mbsyncrc based on
# http://www.ict4g.net/adolfo/notes/2014/12/27/EmacsIMAP.html
# ACCOUNT INFORMATION
IMAPAccount gmail
# Address to connect to
Host imap.gmail.com
User name@address.com
Pass ************
AuthMechs LOGIN
SSLType IMAPS
SSLVersions SSLv3
CertificateFile /opt/local/share/curl/curl-ca-bundle.crt

# THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE
# - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE
#   SPECIFICATION OF AN IMAP ACCOUNT)
# - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER

# REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE)
IMAPStore gmail-remote
Account gmail

# LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail)
MaildirStore gmail-local
Path ~/mbsync/
Inbox ~/mbsync/INBOX

# CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS
#
# CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl
# FOLDERS. SOME COMMONLY USED PATTERS INCLUDE:
#
# 1 "*" TO MATCH EVERYTHING
# 2 "!DIR" TO EXCLUDE "DIR"
# 3 "DIR" TO MATCH DIR

Channel gmail-inbox
Master :gmail-remote:
Slave :gmail-local:
Patterns "INBOX"
Create Both
Expunge Both
SyncState *

Channel gmail-trash
Master :gmail-remote:"[Gmail]/Bin"
Slave :gmail-local:"[Gmail].Bin"
Create Both
Expunge Both
SyncState *

Channel gmail-sent
Master :gmail-remote:"[Gmail]/Sent Mail"
Slave :gmail-local:"[Gmail].Sent Mail"
Create Both
Expunge Both
SyncState *

Channel gmail-all
Master :gmail-remote:"[Gmail]/All Mail"
Slave :gmail-local:"[Gmail].All Mail"
Create Both
Expunge Both
SyncState *

Channel gmail-starred
Master :gmail-remote:"[Gmail]/Starred"
Slave :gmail-local:"[Gmail].Starred"
Create Both
Expunge Both
SyncState *

# GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE
# MBSYNC ON A GROUP TO SYNC ALL CHANNELS
#
# FOR INSTANCE: "mbsync gmail" GETS MAIL FROM
# "gmail-inbox", "gmail-sent", and "gmail-trash"
#
Group gmail
Channel gmail-inbox
Channel gmail-sent
Channel gmail-trash
Channel gmail-all
Channel gmail-starred

Now I synced with Gmail using

mbsync -V gmail

which worked fine for a while but then kept returning errors from Gmail about throttling and my quota being exceeded. My email is about 6GB in size, so large but not ridiculous, but it seems google didn’t like me trying to download it all in one go (possibly a limitation of my employer’s corporate google service). Anyway, I checked with the isync mailing list and there should be no problems with syncs being interrupted and resumed so it would be possible to keep repeating the sync until it completed. Instead of that I used Apple’s network link conditioner (part of Hardware IO Tools for Xcode) to limit my bandwidth to 750 kbps and left the sync to run overnight, which completed without problems.

Once this was done, I had my offlineimap copy of my maildirs in ~/offlineimap and my mbsync maildirs in ~/mbsync. To make it easy to switch between them if needed, I made a symbolic link from ~/Maildir to the mbsync maildir.

Next I deleted my mu index and re-indexed

rm -rf ~/.mu
mu index

I then set some mu4e variables to point to my new maildir

;;location of my maildir
(setq mu4e-maildir (expand-file-name "~/Maildir"))

;;command used to get mail
;; use this for testing
(setq mu4e-get-mail-command "true")
;; use this to sync with mbsync
;;(setq mu4e-get-mail-command "mbsync gmail")

;;rename files when moving
;;NEEDED FOR MBSYNC
(setq mu4e-change-filenames-when-moving t)

;;set up queue for offline email
;;use mu mkdir  ~/Maildir/queue to set up first
(setq smtpmail-queue-mail nil  ;; start in normal mode
      smtpmail-queue-dir   "~/Maildir/queue/cur")

Now, after restarting mu4e, I was using the mbsync maildir.

Note that I initially used a dummy command to sync the mail in mu4e so that I could run the sync manually on the command line at first. I really only needed this because I kept hitting problems with duplicate UIDs, which are solved by getting mu4e to rename files when moving them, as in the code above.

The switch over has been very smooth. In fact I was surprised to find that the message IDs were preserved so that the org-links I stored to emails when I was using offlineimap still take me to the correct email!

  • JohnKitchin

    Thanks! I made the switch this afternoon. No problems at all!

    • eduardo mercovich

      Hi John.

      Did you re-downloaded the mailboxes, or just used your previous ones?

      Best… ๐Ÿ™‚

      • JohnKitchin

        I reloaded them.

        • eduardo mercovich

          It seems the norm, from what I could read.

          Thanks a lot… ๐Ÿ™‚

          • JohnKitchin

            It makes it easy to go back if needed.

  • eduardo mercovich

    Hi Ben.

    This is something I’been wanting to try, but since offlineimap works ok for me and in the background, it always had low priority. How much faster/lighter is mbsync?

    OTOH, I have almost 30 Gb of maildir and a few accounts, so just re-downloading is not the ideal option. I’ll have to see if my current maildir is used…

    As always, thank you very much for your very useful articles. ๐Ÿ™‚


    e

    • For me, syncing with offlineimap averages 40 seconds, taking 25% CPU while mbsync takes about 8 seconds using 5% CPU.

      • eduardo mercovich

        That is some serious difference… only 20% of the time and CPU!

        I’ll look at how to migrate without having to re-download my whole maildir.

        Thanks! ๐Ÿ™‚

  • Pingback: 2016-03-28 Emacs News - sacha chua :: living an awesome life()

  • Michael D Jones

    I followed all of this and it all worked out great. Really impressed at how fast mbsync downloaded all my thousands of emails in the initial sync and I appreciate how fast it is on every other sync too.

    But I went back to offlineimap. Emails loaded up slowly in Mu4e with the mbsync maildir. I don’t know how the two programs differ, but there would be a noticeable second or two lag when I loaded up a list of email headers after i started using mbsync. Can this be fixed? Anyone had this issue?

  • Gastove

    This got me so close! I had to make two small changes:

    1. I’m using the brew-installed cert.pem in `/usr/local/etc/openssl/cert.pem` as the value for CertificateFile.
    2. Based on https://sourceforge.net/p/isync/bugs/23/, I removed the line `SSLVersions` entirely.

    Blam! Thanks very much.

    • Thanks for sharing your changes. I also had to update my SSLVersions line but I switched mine to
      SSLVersions TLSv1.2

  • Im trying to sync my gamail all mail into the INBOX folder, but I keep getting the [gmail]/all mail label in mu4e. Any hints on how to set this up?

  • Yaroslav Danko

    Hi, thanks for your article.

    I followed the instructions and switched from offlineimap to mbsync. I was quite happy with new setup, until i realised that messages deleted from my local “All Mail” folder remain in remote All Mail folder. Now the remote All Mail folder contains approximately twice more mails than local folder. All other folders but “All Mail” are synced OK.

    I have in my .mbsyncrc

    Channel gmail-all
    Master :gmail-remote:”[Gmail]/All Mail”
    Slave :gmail-local:”[Gmail].All Mail”
    Create Both
    Expunge Both
    SyncState *

    Group gmail
    Channel gmail-all

    What could be the reason?
    Thanks in advance.

    • This looks okay to me so I’m not sure why. Maybe ask on the mbsync help forums…

      • Yaroslav Danko

        Hi, thanks for the advice.