Gmail, IMAP, and iPhones – Part 2 – The Tutorial

July 23, 2007 at 2:18 pm by in code
 

UPDATE [09/18/2009]: While this technique still has it’s uses, I no longer use it myself. For various reasons I have switched to using gmail’s normal imap, which was put in place several months after I created this system. So don’t be looking for this page to be updated. Also, I have closed comments on this post as it receives more spam than real comments these days.

As I posted previously, I’ve been experimenting with finding a my perfect iPhone email set up. Since then I’ve made a few changes and I’m really happy with the results. As such, I thought I’d post a bit better of a tutorial since it’s a bit convoluted from the back-end—though still simple to use and the complexity is invisible during daily use.
This post has been modified since it was first published to reflect alternate approaches and improved techniques. I will continue to improve this post in the future to fix any bugs that emerge.
Tutorial Revision (8/21/07): Because of a discovered issue with MS Exchange, I have revised how outgoing emails are handled.

Benefits to this system

  • Spam trapped via gmail’s awesome spam filter
  • Use your personal domain name address as your public-facing address (i.e. from and reply)
  • Read and send email using iPhone and Mail.app with
  • Email marked as read, deleted, or moved is automatically synced between iPhone and Mail.app via IMAP
  • Gmail archives all email both received and sent

How it works from the front-end

  • My catcubed.com email is my main public email address, but I also still receive messages sent to my Gmail.
  • All incoming messages (including my catcubed.com account) are spam filtered via gmail’s fabulous spam filter
  • I use my iPhone and/or Apple Mail for normal day to day email reading/sending
  • Personally, I read email, reply if needed, and then delete; thus, keeping my inbox clutter-free (in keeping with a GTD approach). If I need to review an old email, I go to Gmail where all my emails both incoming and outgoing are automatically archived.
  • Alternately, if deleting is not your style, you can set up your Mail.app to automatically move mail to an archive folder on your computer.
  • Additionally to reduce digital clutter, I’ve manually set it up so that all mailing lists are filter to remain in gmail instead of forwarding my iPhone. I go to gmail to check mailing lists.

Email account cast of characters:

  • catcubed.com account – public email addy (also used as your outgoing mail server)
  • Gmail.com account – main email reservoir, spam filter, and archiver
  • private iPhone IMAP account – secondary email reservoir receives mail after it has been filtered via gmail. This account name is not used publically. .Mac IMAP Issues:Numerous people with .Mac accounts have had problems. I have not fully tested this yet, so I don’t know what the problem is; however using .Mac for your IMAP is not recommended at this time.

Variant Setup – Gmail as public facing: You can use your gmail addy as your public facing. In which case, you only need the gmail and the IMAP account

Behind the scenes: What happens to incoming messages

  1. Emails sent to catcubed.com account are set on the server to auto-forward to gmail
  2. Gmail spam filter does it’s thing
  3. Manually created gmail filter filters out all mailing lists and then forwards the remaining emails to the iPhone IMAP account; in gmail the emails are also archived (skip inbox) and labeled “iphone” (this labeling makes for easier searching later)
  4. iPhone and Apple Mail check for mail on this IMAP account
  5. Emails read on iPhone are labeled read in Mail and vice versa thanks to IMAP (deleted mail, saved drafts, are also synced)

Behind the scenes: What happens to outgoing messages (Ver. 2.2)

  1. Email sent from iPhone/Mail goes out through your public facing email account STMP (this email is also synced to the IMAP in my sent folder)
  2. All email is auto BCCed back to yourself so that sent mail can be archived in gmail (mail sent back this way appears as sent mail in gmail and is connected to conversation threads like normal — gmail’s magic that way)

Varient Setup – Using Gmail as public facing: If you are using gmail as your public facing it will be your SMTP.
Note on 8/21/07 Revision:Until today, I had been recommending using Gmail as your SMTP even for use with a different public facing email, however MS Exchange handles emails routed in this manner bizarrely (see comments for details), so this is not recommended.

Setup Tutorial

  1. Create 3 email accounts:
    • Your gmail address
    • Your public facing address (a normal POP is fine) [this account is not required if you are using gmail as your public facing]
    • your private IMAP account used only for retrieval on messages for your iPhone/Mail.app
  2. Set your public facing account to auto-forward all emails to your gmail account (this should be done server-side; you can set it to delete the emails too since there is no need to store emails here)
  3. Set up a manual filter on gmail to forward on all mail — except mail sent from yourself — to your IMAP account, skip inbox, and label “iphone”
  4. You can also set this same filter to filter out mailing lists like I did (i.e. does not include [mailinglist email] or [other mailing list email] or [Uncle George who sends all those bad email jokes]) For info on advanced gmail filtering techniques read Lifehacker’s advanced gmail filter article
  5. Set up your Mail/iPhone programs. You will only need to set up one and the other will copy the settings. However it only copies the settings initially so any later changes after this will have to be replicated on both.
    1. In the address field use your public facing email addy
    2. For incoming use your private iPhone IMAP account configuration (read your email server’s instructions for proper settings).
      While it depends on your particular email server, you will most likely need to set IMAP Path Prefix to “INBOX”; this is found in your account settings under Advanced. Also select “auto synchronize changed mailboxes” is selected in Mail.app
    3. Use your public facing email as your outgoing SMTP account
      You may need to set the port number. On an iPhone the port number is appended after the server url (i.e., mail.domainname.com:26) YPNMV
    4. Set iPhone/Mail.app mail settings to “Always BCC Myself” (this makes sure it gets into gmail) Sent messages are also saved to your iPhone sent folder and synced magically to your Mail.app (and vice versa) via the IMAP server. Yes this works even though you are not sending out through this server.
  6. Optionally Step: you can go back to gmail and set it up to allow you to send emails from it as your public facing account. This way you can also use it as your complete webmail client. Instructions on how to do this are here
  7. Optional Step 2: Setup rules on your Mail.app for incoming mail. Note, these rules will only take effect when you return to you machine and start Mail.app. You can these rules this to move mail off the server folder and into local folders for archiving.

That’s it! Not too bad really! Granted it’s a bit confusing to setup with all those accounts, but it’s invisible during daily use. You have the benefits of using the iPhone for email, but also have Gmail as your main email repository, spam filter, and optional webmail client.

Did this improve your iPhone experience?
By request of one of my visitors, I’ve now added a PayPal Tip Jar to my blog over in the right hand sidebar under my photo. It did take me a bit of effort to figure all this out and write it up. And I’ve been busier than I expected answering everyone’s questions, so if you find this useful toss a couple bucks in so I can buy myself a drink or something! Or even better pay some of my hosting fees. As a bonus, people who tip will receive email updates if any fixes or major improvements made to this tutorial.

Thanks!
Colin

I will be away from CatCubed for a bit as I pack and head out to Burning Man, and I may need a bit of decompression time after too. So my replies to your questions may not be as quick as they have been.

Tags: , , , , ,

80 Comments

  1. What are you using for the private IMAP?
    I am having trouble getting the smtp server on the private IMAP account to send email from my gmail account.
    I can set the outgoing server as smtp.gmail.com, but then I lose the ability to view sent messages on my iPhone.

  2. Hey Seth,

    I’m just using an extra IMAP account that I get from my hosting ISP.

    It should be sending from private IMAP straight out to the interwebtubes. How it gets to gmail is by setting your iPhone to automatically CC yourself. It will CC your public facing address and these CCed emails basically circle back and are received by gmail.

    Also as I mentioned, you’ll need to filter out emails that come from yourself or these CCs will come to your iPhone inbox.

  3. Got it. My IMAP is giving me errors because I’m trying to send “from” the front-facing address using the private IMAP smtp server.
    I guess this just varies from ISP to ISP.

  4. I did a similar setup for my mail… but I wanted to make a suggestion…

    Instead of CC yourself, you can do as you suggest in #10 and set your public email as the sending email in gmail. Then when you setup your mail on your iPhone, just make the outgoing smtp.gmail.com and send your mail with gmail. This way, you will actually be sending from gmail, you won’t have to CC everything, and it will appear to your recipient as your public email account.

    Nice tutorial by the way… I’m finally happy with my email, gmail and iPhone :)

  5. Thanks Jared,

    You’re correct. You can just have gmail as the outgoing SMTP server. I had set it up to use the IMAP so that the sent messages would be in my IMAP for when I check it in Apple Mail.

    However, I actually find myself using the iPhone more and more and my Apple Mail less and less, so really there is no need store the sent on the IMAP. I might just switch over to using Gmail as my SMTP.

  6. Seth,

    I guess it does vary. In any case, you could just use gmail as the SMTP as Jared suggests.

  7. Finally, a solution that works! I was having trouble with my gmail alone…getting 25 old messages downloaded and not being able to clear the inbox. It just would retrieve more and more old messages. The IMAP intermediate works great and I get gmail filtering.

  8. dan claire says:

    thanks for the setup tutorial. I needed it. Works like a charm!

  9. John Minard says:

    Excellent work-around! Nicely explained.

    However, I use Gmail’s tagging feature extensively. I suppose this can only be done via a web browser. I usually tag an item before archiving it, then I go back and pull out action items based on the tags. I will also star an email that I need to act on right away, and leave it in the inbox as a reminder.

    As a concept, I think email really needs to be integrated with a To-Do kind of function that let’s us categorize and prioritize items. Gmail is as close as I’ve found in a Web-based environment.

    I’m ready to buy an iPhone, when I can fully use Gmail, gCalendar, and contacts on it. I think the Plaxo products may be my answer, but not sure.

    Thanks for all the effort you put in to this tutorial. It’s very helpful.

    - John

  10. Yeah I was also using gmail in this way too, but I found that it quickly grew out of control. When your email box is also your to do list it makes you dread visiting it when stuff piles up.

    Also all the labeling and archiving just took too long. This is my same problem with some aspects of the GTD approach: all that adding context and projects and whatnot just takes too long. It’s often better to just write down the things you may forget and then tackle whatever’s in front of you.

    This is what I like most about using the iPhone email. I just read, reply, delete. It’s quick and painless. It highly contrasts with my work setup where I have to use the slow and painful Outlook 2003 — I can’t stand Outlook. Plus I only get 45MB of online email storage — what’s up with that?

    Relatedly I was asked by a colleague, how I get it to work with the exchange servers. I said, “I don’t which is fine because i’m almost always near my laptop at work and frankly I don’t want work emails on my phone.”

  11. Pingback: Ultimate iPhone email setup at Σπιτάκι

  12. Any thoughts on how to make this work with .Mac mail, if that’s all the IMAP you got?

  13. Hey Stretch, I don’t have a .Mac account to test it on, but from reading the .mac configuration page you should be able to just manually set up using the following settings…
    incoming server: mail.mac.com
    login: (your name before the @)
    password: (your .mac password)

    And then outgoing is your gmail SMTP

  14. Colin,
    do you have a similar set-up for people who don’t have IMAP accounts? my hosting company (xo.com) does not provide IMAP accounts… or maybe you know where one can be had. thanks.
    B

  15. Just found out the AOL offers free IMAP email accounts which can work with a personal domain. I hope the SPAM filter is as good as gmail. The size limit is unlimited.

    https://domains.aol.com/personaldomain/app/domainMainSearch
    Just click on the bring your own domain.

    Plan to try this out with the iPhone. Any one tried this or have experience of AOL email?

  16. Colin,

    Excellent tutorial. I’m not the sharpest nail in the toolbox, but even I got it to work. Everything functions as advertised except for the automagically syncing sent folder on the Mac and the iPhone. I never did get that added bonus.

    Regardless, great work!

  17. Oliver,
    While I didn’t look to much into the AOL email you linked to. I’d guess that you should be able to use the AOL IMAP email with my setup so that you are using gmail’s spam filtering.

  18. Brock,
    Did you make sure to add the correct IMAP Path Prefix? I’m not sure about your server, but I had to add “INBOX” for this setting in both my Apple Mail and iPhone to get it to sync.

  19. Bogdan,
    You might want to check out the free AOL email accounts that Oliver pointed out.

  20. So all the email on your iphone/imail show as forwards “Fw: …” isn’t that annoying?

  21. Matt,
    No gmail filter forwarding happens before all that and there is no “FW:” or other sign that it has been forwarded. On the frontend it’s completely transparent that any of this kung fu filtering is happening.

  22. Pingback: Awards link scam; Truth about book-writing; Better iPhone mail; Old-timey photo maps; How I really found your site; ViaTalk? Dead to me. | 43 Folders

  23. Colin,

    You are absolutely right and hit the nail on the head. In my excitement I forgot to do just that. Thank you – my email setup is now perfect.

  24. Colin,

    Thanks for the great tip…however I’m having problems since it doesn’t seem to be forwarding the mail to the IMAP account–in this case my .Mac account.

    I’m using my spam-filled sbcglobal addy as my main and that’s being forwarded to gmail and that seems to be working. The gmail filter was applied properly (I think), but I don’t see anything coming into my .Mac email.

    Here’s what my gmail filter looks like:
    Matches: from:(xxxxxxxxxxxxxxxx@sbcglobal.‚Äãnet)
    Do this: Forward to xxxxxxxxxxxxxxxxxxxxx@mac.com, Skip Inbox, Apply label “iPhone”

    BTW, can you do more than one email in this filter? Separated by commas perhaps?

    Regarding my sbcglobal account…I have a main account with two sub accounts–one of which is my main account (hence my previous question). Would that be screwing it up? When I check my email through a web client, the mail keeps coming and I don’t see it hitting gmail. Am I screwing things up by checking the sbc account with the web client–could it be preventing it from being forwarded?

    Sorry for all of the questions, but AT&T’s pitiful spam filters give me a daily job that I don’t want–ESPECIALLY on my iPhone. Thanks for listening.

  25. Hey Bob,
    Technically, the mail forwarded from your sbcglobal.‚Äãnet addy is not coming “from” it (automated forwards are different). So you need remove it out of the “from” field. It should then work.

    Also If you want, you can add your sbcglobal.‚Äãnet addy (and your gmail addy) to the “to” field. This will make sure mail addressed to you is forwarded, but mail were you are just BCCed (like some spam and mailing lists) is not forwarded.

  26. Thank you SO much. This is the holy grail i’ve been wanting for EVER. Server side spam filtering sucks on the mail server I use so using IMAP on a mobile device was far from practical due to the amount of spam I get. This setup solves all of my problems. If there was a donate link, i’d hit it =]

  27. Okay, thanks for that clarification. Added the sbc addy to the “to” field, but it still doesn’t seem to be forwarding on to the .Mac account.

    Another question is on the iPhone settings…I set up a test account and used my sbc addy in the “address” field. Here’s the settings…
    Incoming:
    Host name: mail.mac.com
    User name: xxxxxxx
    Password: xxxxxxxxxxxxxx
    Outgoing (SMTP):
    Host name: smtp.gmail.com:465
    User name: xxxxxxxxxxxxxx@gmail.com
    Password: xxxxxxxxx

    Advanced:
    Incoming uses SSL: ON
    Outgoing uses SSL: ON
    Authentication: Password
    IMAP path prefix: INBOX.
    Is this correct?

    According to the Apple pages on .Mac mail, the incoming ports will switch between secure and non-secured ports. I’m wondering if turning off SSL on the iPhone would fix this issue? Here’s part of that .Mac help page:

    “The Incoming mail (IMAP) port number automatically changes from 143 to 993, but you will need to manually change the Outgoing server (SMTP) from 25 to 587. If you are configuring the account as POP, the Incoming mail port instead changes from 110 to 995.”

    Still scratching my head on this one…

  28. Thanks loren,
    Glad it’s working out for you! And I guess I can’t pass up a request like that, so I’ve now added a PayPal Tip Jar in the right sidebar under my photo area.

  29. Aha, works very nicely, thank you again! gmail on the iphone doesn’t suck anymore :)

  30. So one issue I’m seeing after setting this up ‚Äì how do you search the archived emails that you’ve deleted from your IMAP account? Sure, they’re archived in gmail, but I can’t search for sent or old mail messages from within mail.app? Any advice on this?

  31. Jimbo,
    Your sent mail should be being synced already. When your deleting the mail off the iPhone the IMAP syncs the deletes with your Apple Mail.

    Personally this is the system I want as I like gmail’s search functionality. If you’d rather keep the messages locally, you can set up filters in Apple Mail that moves it from your inbox to other folders (either local-only or folders that sync on your server). These filters will only be triggered when you open Apple Mail so you’d just read and NOT delete on the iPhone (maybe move it another folder manually) and let the Apple Mail take care of it later. This moves will be synced to your iPhone: if you moved to a local-only it’ll be treated like it was deleted; if it’s a synced folder it’ll show up in that folder on your iPhone.

    Otherwise, if what you want is a system that you can actualy delete from your iPhone but not sync these deletes, then what you really want is a POP account not an IMAP.

  32. I was able to follow your instructions but only with partial success: when I delete emails on iphone, they remain on my IMAP server. Any ideas why?

  33. I’ve ran into the same issue, GMail doesn’t forward to .Mac account, so I don’t think that can be used for IMAP account.

    Actually my guess is .Mac throws away email that is forwarded to it. I can send directly to my .mac account and I get it fine.

  34. Marcus,
    Have you tested the filter forwarding to a different account just to make sure that it is actually forwarding properly? Have you checked any .Mac spam filters?

    Has anyone else got this working with a .Mac account? I thought I read that someone had it working with .Mac — unfortunately, I don’t have a .Mac account so I can’t test it myself.

  35. K all,
    I did some testing using a trial .Mac account and I’ve had some trouble myself. Not sure if it’s a fault of mine or .Mac, but I’m not seeing the emails. But it’s late and my brain is tired so I could have missed some step.

    This forum seems to have people that have gotten it to work. I’ll troubleshoot it a bit more later. Also, if anyone else has a more definitive answer to the .Mac question let me know.

  36. Colin, I think that link is incorrect. ;-)

  37. Thanks, for the heads up Bob! Link in my comment above now works!

  38. Hey… great work.
    One thing I notice… when I send email from the imap using gmail as the outbound and after setting up my “facing” email in gmail to send out…

    When people with exchange server get an email from my … that went from my imap via gmail that I want to look like it came from my facing email…
    It shows that it was $$$$$$@gmail.com on behalf of facingemail@whatever.com

    so basicly people see my gmail account and on my of my friends work computer my email sent this way gets spam filtered out.

    Anyone else have this issue?

    thanks.
    T

  39. This seems brilliant but when Gmail forwards my mail the reply-to is set as my gmail (forwarding) account instead of the actual sender. The From box shows the actual sender but the Reply-to naturally overrides it. I am using .Mac/Mail as my IMAP account. I have checked Gmail and nowhere is it set (that I can see) to force a Reply-to address. Any ideas?

  40. Huh, you’re right Terry.
    Exchange does display that rather odd message. I had tested it earlier with Exchange when I was using my old system (out through IMAP and BCCed back to myself) and that didn’t happen. So I didn’t think to test it again.

    It still replies to your header email fortunately, but it’s annoying. If you want to avoid this you can possibly set it up the way I used to do it:
    - Set your outgoing as your IMAP Or better your public facing
    - Select the iphone option to BCC yourself on all outgoing emails. This way it routes back to your gmail (it’ll still thread with the conversations like normal).
    - You’ll need to gmail filter out any messages that come from yourself so that they don’t get sent back to your iphone

    Unfortunately, some ISPs don’t allow you to use a different header for outgoing emails, so this my not work for you. It worked on my Bluehost account though.

  41. To Cary and all others having trouble with using .Mac,

    .Mac seems to be a problem. It doesn’t work at all for some people and for others it works halfway. I haven’t had time to fully test it myself.

    As I wrote in my lLatest Post, I will be away from CatCubed for the next couple weeks as I pack and head to Burning Man. After I return, I will do my best to find time to tackle these issues. Also I’m planning on writing a bit more detailed instructions for people less apt at geekdom than I.

  42. Thanks Colin.
    I have Blluehost as well.
    But I was using a different imap account.
    I will try it with my bluehost.
    My only point I guess.. would be the possibility of people finding out my gmail account. (an account which I treasure)

    Thanks again for your work on this.

    T

  43. I’ve recently revised the tutorial to reflect issues that were brought up in the comments.

  44. Thanks for the help and heads up Colin.

    I have just forced myself into the realization that .Mac is sluggish and will not work using your technique. I have it set up with a test account on my iPhone, but now it won’t even display the email body for some bizarre reason–but it sends from Gmail just fine (go figure).

    I started playing with Yahoo mail as a possible IMAP replacement, but as that works well with forwarded mail, I can’t set it up manually for your method since the iPhone’s screens won’t allow it when setting up Yahoo accounts (this is how Yahoo will give iPhone customers free push IMAP by ID’ing them this way).

    Since I’ve created a new Yahoo account for this test, I think I’ll just enjoy Gmail’s spam filtering and forward it to the Yahoo account or play with it some more…I’ve been playing with all of these accounts for so long now that it’s making my head spin.

    Thanks again and enjoy Burning Man.

  45. Hi Cat, I’m not sure if u know about Google Apps [ http://www.google.com/a ]. It offers to host your domain’s emails in real. You don’t need to set FROM: and REPLY-TO: addresses in a standard Gmail account to send & receive your own domain’s email.

    Point ur domain’s MX record to Google and create a free standard account on Google Apps and u can have unlimited email addresses on ur domain each with 2GB and growing storage space.

    In case u think 2GB is not enough, u can always upgrade to a $50/yr/email account Premium service with 10GB storage!

  46. Well um “Google Packistan”, if that’s your real name

    Google Apps is great and all but they only offer POP access, so if I were to redirect my catcubed.com MX record I’d no longer have IMAP; thus couldn’t use my system.

  47. Cat,

    Your tutorial is wonderful, thanks. I set up GMAIL to filter all my listserv messages with Skip the Input, Apply the Label, so these do not come to my iPhone.

    The filter is working, but these messages are still being downloaded to my iPhone from G-mail. Did I miss a step somewhere?

    I haven’t implemented forwarding the G-mail back to DOT MAC yet, still just POP’ing from G-mail.

    Dave

  48. For those of you who are trying to use your .Mac account, I figured out an easy way to make it work. On your iPhone, get rid of the existing .Mac email account setup. Then add a new account, and choose the other option. You then set up the new account with your public email, you use your .Mac info as your incoming mail server and your gmail account as your outgoing server. It works perfectly.

    Also, I was having trouble creating really long filters in gmail. Once I got too many email domains within the brackets, I kept getting errors whenever I tried to save the filter. I switched over to basic HTML for gmail, and the errors disappeared.

  49. Bram Posted:

    The only problem with manually configuring .Mac email on your iPhone is that you can no longer upload a photo taken with your iPhone to your .Mac web gallery. Took me a day to figure that out. Which sucks.

    I have followed the steps above to forward my email from two gmail accounts to my .Mac email account. One of my gmail accounts is my unique personal email tied to my personal domain name. I’ve also had a Yahoo email forever and an AOL AIM email too.

    On the iPhone I’m able to manually configure my AIM email account to send out as my personal domain email. I have configured both my Yahoo and .Mac email accounts normally to get PUSH email from Yahoo and photo upload to .Mac.

    I’m also forwarding my .Mac email to my Yahoo account to get the PUSH feature for all my mail. I have iPhone set to Auto Check manually.

    So when I get email to any of my accounts I get an indication immediately with Yahoo. I delete the forwarded email that came from my .Mac and then know to manually check that account on the phone.

    A pain in the butt but I couldn’t figure out any other way to get PUSH.