Gmail, IMAP, and iPhones – Part 2 – The Tutorial

July 23, 2007 at 2:18 pm by in code
 

UPDATE [09/18/2009]: While this technique still has it’s uses, I no longer use it myself. For various reasons I have switched to using gmail’s normal imap, which was put in place several months after I created this system. So don’t be looking for this page to be updated. Also, I have closed comments on this post as it receives more spam than real comments these days.

As I posted previously, I’ve been experimenting with finding a my perfect iPhone email set up. Since then I’ve made a few changes and I’m really happy with the results. As such, I thought I’d post a bit better of a tutorial since it’s a bit convoluted from the back-end—though still simple to use and the complexity is invisible during daily use.
This post has been modified since it was first published to reflect alternate approaches and improved techniques. I will continue to improve this post in the future to fix any bugs that emerge.
Tutorial Revision (8/21/07): Because of a discovered issue with MS Exchange, I have revised how outgoing emails are handled.

Benefits to this system

  • Spam trapped via gmail’s awesome spam filter
  • Use your personal domain name address as your public-facing address (i.e. from and reply)
  • Read and send email using iPhone and Mail.app with
  • Email marked as read, deleted, or moved is automatically synced between iPhone and Mail.app via IMAP
  • Gmail archives all email both received and sent

How it works from the front-end

  • My catcubed.com email is my main public email address, but I also still receive messages sent to my Gmail.
  • All incoming messages (including my catcubed.com account) are spam filtered via gmail’s fabulous spam filter
  • I use my iPhone and/or Apple Mail for normal day to day email reading/sending
  • Personally, I read email, reply if needed, and then delete; thus, keeping my inbox clutter-free (in keeping with a GTD approach). If I need to review an old email, I go to Gmail where all my emails both incoming and outgoing are automatically archived.
  • Alternately, if deleting is not your style, you can set up your Mail.app to automatically move mail to an archive folder on your computer.
  • Additionally to reduce digital clutter, I’ve manually set it up so that all mailing lists are filter to remain in gmail instead of forwarding my iPhone. I go to gmail to check mailing lists.

Email account cast of characters:

  • catcubed.com account – public email addy (also used as your outgoing mail server)
  • Gmail.com account – main email reservoir, spam filter, and archiver
  • private iPhone IMAP account – secondary email reservoir receives mail after it has been filtered via gmail. This account name is not used publically. .Mac IMAP Issues:Numerous people with .Mac accounts have had problems. I have not fully tested this yet, so I don’t know what the problem is; however using .Mac for your IMAP is not recommended at this time.

Variant Setup – Gmail as public facing: You can use your gmail addy as your public facing. In which case, you only need the gmail and the IMAP account

Behind the scenes: What happens to incoming messages

  1. Emails sent to catcubed.com account are set on the server to auto-forward to gmail
  2. Gmail spam filter does it’s thing
  3. Manually created gmail filter filters out all mailing lists and then forwards the remaining emails to the iPhone IMAP account; in gmail the emails are also archived (skip inbox) and labeled “iphone” (this labeling makes for easier searching later)
  4. iPhone and Apple Mail check for mail on this IMAP account
  5. Emails read on iPhone are labeled read in Mail and vice versa thanks to IMAP (deleted mail, saved drafts, are also synced)

Behind the scenes: What happens to outgoing messages (Ver. 2.2)

  1. Email sent from iPhone/Mail goes out through your public facing email account STMP (this email is also synced to the IMAP in my sent folder)
  2. All email is auto BCCed back to yourself so that sent mail can be archived in gmail (mail sent back this way appears as sent mail in gmail and is connected to conversation threads like normal — gmail’s magic that way)

Varient Setup – Using Gmail as public facing: If you are using gmail as your public facing it will be your SMTP.
Note on 8/21/07 Revision:Until today, I had been recommending using Gmail as your SMTP even for use with a different public facing email, however MS Exchange handles emails routed in this manner bizarrely (see comments for details), so this is not recommended.

Setup Tutorial

  1. Create 3 email accounts:
    • Your gmail address
    • Your public facing address (a normal POP is fine) [this account is not required if you are using gmail as your public facing]
    • your private IMAP account used only for retrieval on messages for your iPhone/Mail.app
  2. Set your public facing account to auto-forward all emails to your gmail account (this should be done server-side; you can set it to delete the emails too since there is no need to store emails here)
  3. Set up a manual filter on gmail to forward on all mail — except mail sent from yourself — to your IMAP account, skip inbox, and label “iphone”
  4. You can also set this same filter to filter out mailing lists like I did (i.e. does not include [mailinglist email] or [other mailing list email] or [Uncle George who sends all those bad email jokes]) For info on advanced gmail filtering techniques read Lifehacker’s advanced gmail filter article
  5. Set up your Mail/iPhone programs. You will only need to set up one and the other will copy the settings. However it only copies the settings initially so any later changes after this will have to be replicated on both.
    1. In the address field use your public facing email addy
    2. For incoming use your private iPhone IMAP account configuration (read your email server’s instructions for proper settings).
      While it depends on your particular email server, you will most likely need to set IMAP Path Prefix to “INBOX”; this is found in your account settings under Advanced. Also select “auto synchronize changed mailboxes” is selected in Mail.app
    3. Use your public facing email as your outgoing SMTP account
      You may need to set the port number. On an iPhone the port number is appended after the server url (i.e., mail.domainname.com:26) YPNMV
    4. Set iPhone/Mail.app mail settings to “Always BCC Myself” (this makes sure it gets into gmail) Sent messages are also saved to your iPhone sent folder and synced magically to your Mail.app (and vice versa) via the IMAP server. Yes this works even though you are not sending out through this server.
  6. Optionally Step: you can go back to gmail and set it up to allow you to send emails from it as your public facing account. This way you can also use it as your complete webmail client. Instructions on how to do this are here
  7. Optional Step 2: Setup rules on your Mail.app for incoming mail. Note, these rules will only take effect when you return to you machine and start Mail.app. You can these rules this to move mail off the server folder and into local folders for archiving.

That’s it! Not too bad really! Granted it’s a bit confusing to setup with all those accounts, but it’s invisible during daily use. You have the benefits of using the iPhone for email, but also have Gmail as your main email repository, spam filter, and optional webmail client.

Did this improve your iPhone experience?
By request of one of my visitors, I’ve now added a PayPal Tip Jar to my blog over in the right hand sidebar under my photo. It did take me a bit of effort to figure all this out and write it up. And I’ve been busier than I expected answering everyone’s questions, so if you find this useful toss a couple bucks in so I can buy myself a drink or something! Or even better pay some of my hosting fees. As a bonus, people who tip will receive email updates if any fixes or major improvements made to this tutorial.

Thanks!
Colin

I will be away from CatCubed for a bit as I pack and head out to Burning Man, and I may need a bit of decompression time after too. So my replies to your questions may not be as quick as they have been.

Tags: , , , , ,

80 Comments

  1. I manage three email accounts from my gmail. With this setup, is it possible to reply using the address that the message comes in on? For example, if i get an email @school.edu that comes through to Apple Mail or the iPhone via IMAP, will my response always be from @gmail.com?

    Thanks for this great work!

  2. your email will be sent using whatever main address set up originally. If you want to send with a different account you’d need to add a separate account profile on you’re iPhone.

  3. Rebecca S.H.F. says:

    hey guys,- thanks for everyone that put their effort/opinion on this! I am a VERY new iPhone user and am having trouble understanding some of the lingo, but after searching the internet for hours, what I’ve understood here is as close as i’m going to get! What im trying to do is this:
    My gmail account is my main account but i want my mail pushed from yahoo! to my iphone. I’ve set up a yahoo! account just for this purpose, which I hope i can just keep a secret and use it as IMAP push. With that being said I want all mail sent from my iphone to be from my gmail account, or have it display the name. I believe this is exactly what you guys are talking about here but I’m getting lost in all the gmail/yahoo! settings!?! My iphone now has two accounts open (yahoo and gmail) and im getting the same mail messages sent to both! Im confused and Im sorry but is there another step-by-step instructional anywhere? This has been VERY helpful and I am impressed with the effort put in, as I have said; i’m just a slow and very detailed kind of learner :( Any help would be appreciated! Thank you!

  4. holy shit.

    all my public facing accounts are hosted through gmail apps. do you have any idea how frustrated i was facing thousands of emails that needed to be deleted anytime i felt like restoring my iphone?

    beer is on me — shortly.

    thanks, stud.

  5. Colin,

    Thank you so much. I was a bit confused, but muddled through and everything works amazingly, ridiculously well. You should be given a medal or something. Thanks again.

  6. After a long, muddling process, I adopted both your workaround (brilliant) and Jared’s idea of using the public email as the sending email in gmail/making gmail the SMTP.

    At that point, I was lost as to why this was such a smart workaround ‚Äî Gmail had suddenly been turned into a big dumb box of archives. But then I reopened Mail.app, which I hadn’t used in years, and realized the glory of this system. Fast, sleek, interconnected access to my email in three spots (home, work, on the road), and a searchable archive from anywhere (including on my iPhone).

    Nice work. All respect due.

  7. Pingback: closet full of weasels - blah, blah, blah. Are you still talking? » Nerdvana

  8. I am finding that my sent mail is not showing up on my iphone. It is in google and all that seems to be working fine, but sent mail is empty on my iphone even though I’ve sent a couple of test messages to my work account.

    Ideas?

  9. I’m having some trouble setting up the manual filter correctly in gmail. Mail is forwarding to the private IMAP account correctly but the BCC’s are showing up in the gmail inbox. Any more detail on the filter set-up would be much appreciated!

  10. Pingback: Awards link scam; Truth about book-writing; Better iPhone mail; Old-timey photo maps; How I really found your site; ViaTalk? Dead to me. at aoortic! dot com

  11. so i haven’t found the ‘auto synchronize changed mailboxes’ setting you mentioned.. u mind giving me some direction? thanks, -Trent

  12. I setup an aol account and managed to get email coming in, but when I setup my iphone to send through smtp.gmail.com:465 it keeps failing. I tried to also send it via :587, still a no go. Anyone have any thoughts on what I am doing incorrectly?

  13. I still think this is a brilliant system, but so far my setup has one flaw: when I’m part of a long BCC list, the message is delivered to Gmail, but won’t then forward on to the iPhone account. I’m sure it’s that I haven’t done the filter correctly in Gmail. Anyone know how to make BCC and CC part of the “to” filter?

  14. Slight tweak to my question. With the following two filters set up, emails directly TO me and CC’ed to me are forwarded on to my iPhone as desired:

    from: {-myvanity@vanity.com -mygmail@gmail.com}
    to: {myvanity@vanity.com mygmail@gmail.com}

    and

    from: {-myvanity@vanity.com -mygmail@gmail.com}
    Has the Words: {bcc: myvanity@vanity.com bcc:mygmail@gmail.com cc: myvanity@vanity.com cc:mygmail@gmail.com}

    BUT those emails on which I’m BCC’ed do NOT show up. Note how odd it is that the CC works, even though it comes SECOND in the Has the Words string.

    Whoo. I’m out of my depth.

  15. Thanks for the excellent tutorial. This setup is working out great, so far. I do have one small issue though. The time stamp on my incoming email on my iPhone doesn’t match the archived email in my Gmail archive. The email has the correct time stamp when it arrives in Gmail, but when it arrives on my iPhone the time is 6 hours later… Anyone have any idea where to fix this?

  16. Late to the party here, but thank you! It works wonderfully. Again, many thanks!

  17. ping imap.gmail.com

  18. Again, late to the party – but works great too… almost!

    Anyone have any idea why I can send e-mail fine logged into GMail, but when I try and send it via my iPhone I receive a bounce back message saying “relaying denied”.

    It’s also worth mentioning my setup involves the GMail account, the IMAP account and an Exchange Server from which messages are forwarded and messages are being sent on to (the recipients are all on the same Exchange Server).

    Cheers!

  19. Now that Gmail supports imap, which I was understandably excited about, I’m suddenly a little skittish about backing out of your system and going full-time with Gmail and my vanity domain. Anyone attempted it? Any pitfalls to be aware of?

  20. Answered my own question. It works, technically speaking, but when I reply from my iPhone, it sends from my gmail.com account, which defeats the whole purpose of putting my vanity domain through gmail. In short, your system, while tougher to implement, is way more effective, in the end. Thanks again for doing what Google cannot. Someone out there owes you big money.

  21. G-Mail now do IMAP !

  22. Jake -
    I had the same concern about the public facing address on mail sent from the iPhone… but it’s actually addressed in the tutorial. If you set up your outgoing smtp on the iPhone from the address you to be public facing, that’s what will show. I set mine up that way, and it works.

    Great tutorial….
    thanks,

  23. Okay, I’m a little late to the game, but I just got an iPhone and am trying to figure all this out. My problem is that I have TWO public-facing emails. One business and one personal. AND now that Gmail does IMAP I feel like it’s all up in the air for me. What do I do?

  24. ANSWER – Put an asterisk (*) in the “to” box when setting up your Gmail filter. It will forward ALL email, including those that you are BCC’ed and CC’ed on. I was having this trouble for the longest time, and now I get all of my email, including things that I’m CCe’d or BCC’ed on :)

  25. Pingback: Spam-free e-mail ontvangen op je iPhone - iPhone Forum - alles over de apple iPhone (in Nederland)

  26. I’m wondering if you’ve gone for the iPhone 2.0 software, and whether it streamlines your system at all. I’m not touching my settings until I hear from smarter folks than myself, and yours was the smartest workaround I’d ever heard of last time.

  27. we should talk more about this. I think I have a pretty nice system going atm. What do you use for synced to do lists?

  28. i recreated my gmail account on my iphone and macbook as imap and it all syncs itself quite beautifully, minimum fuss and i have access to all my emails whether i’m on the phone or laptop connected to the web or not (laptop only mail.app)

  29. Mike Stevens says:

    Hi and thank you to everyone, particularly Colin who started this. I’ve really tried to follow the story but to me, as a newbie and an old one at that, I’m overwhelmed with information.

    I have an iPhone 3G and would like to reduce spam. I see that using Gmail will help.

    I have a main email account which is a POP3 or is it IMAP (****@*********.freeserve.co.uk)
    I also have a Gmail account, which is on my pC and on my iPhone 3G.

    Could someone simply list the settings that each device should have?

    Thank you in advance.

    Mike

  30. Pingback: andyman online: The Perfect iPhone Email Setup?