User Tools

Site Tools


software:dovecot:archiveserver

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

software:dovecot:archiveserver [2018/09/17 19:06] (current)
Line 1: Line 1:
 +====== Archive Mail Server using Dovecot ======
  
 +We have run across this a few times, and thought it might be good to document. A client uses some service which severely limits the amount of e-mail which can be retained. I have seen anywhere from 2G to 10G recently (2016-2018). For some clients, it is a requirement to save e-mail for years, perhaps decades. One common example is the Legal field, where something you did 10 years ago can end up in court.
 +
 +Many e-mail clients allow archival of e-mail, but they store the information locally, on your workstation,​ where it is subject to hardware failure, theft, or natural disaster. Additionally,​ Microsoft Outlook stores all of its e-mail in one huge file, so backups require copying one huge file each time instead of just looking for new/changed files and copying them. Simply opening Outlook can cause the file to be modified, even if you do not do anything with it.
 +
 +A much better solution in many cases is to create an IMAP store specifically for archival purposes. In the last two cases we had, the clients were connecting to an Exchange server and already had an internal Unix file server which had an automated/​monitored off site backup set up, so it was straight forward to set up an IMAP store for archival purposes.
 +
 +This article covers building a Dovecot IMAP server on Linux, manually setting up the users (and space for them). If you want a pretty GUI (actually WebUI) you might look at installing [[https://​www.ispconfig.org/​|ISPConfig]] on a new machine or virtual, but we'll cover doing everything manually here. It is mainly taken from the article at [[https://​wiki2.dovecot.org/​HowTo/​SimpleVirtualInstall]].
 +
 +**NOTE**: ISPconfig has a handy thing in their webui that allows you to disable POP and SMTP for an account. I did not look into how this worked, but it is very handy to ensure you don't accidentally send from your archive account. I'm guessing it will work with the below procedure since we are A) only installing IMAP and B) using a separate credentials file for Dovecot than what an SMTP server would use.
 +
 +===== Setup and Install Dovecot Server =====
 +
 +This is pretty straight forward; allow the operating system to install Dovecot, then override the default configuration file.
 +
 +Install Dovecot
 +<code bash>
 +   ​apt-get install dovecot # debian
 +   yum install dovecot # CentOS
 +</​code>​
 +
 +===== Create a user and store for the e-mail =====
 +
 +We should use a different user/group for this and all mail will be owned by that user/group. Additionally,​ we don't want a login, so we'll set the shell to /dev/false. We'll also tell the adduser script to not create the home directory (we'll create it ourselves),
 +
 +Message store (ie, home directory) can be anyplace. I'm going to set it up in /srv/vmail. This will be the head of a tree of subdirectories for individual users.
 +
 +<code bash>
 +adduser --home-dir /srv/vmail --gid vmail --no-create-home --shell /dev/false --user-group
 +mkdir -p /srv/vmail
 +chmod 755 /srv/vmail
 +chown vmail:vmail /vmail
 +</​code>​
 +
 +===== Modifying dovecot =====
 +
 +Create a configuration file for dovecot. I generally back up the original, then create a completely new file, so:
 +
 +<code bash>
 +mv /​etc/​dovecot/​dovecot.conf /​etc/​dovecot/​dovecot.conf.original
 +</​code>​
 +
 +<code bash dovecot.conf>​
 +protocols = imap
 +
 +# It's nice to have separate log files for Dovecot. You could do this
 +# by changing syslog configuration also, but this is easier.
 +log_path = /​var/​log/​dovecot.log
 +info_log_path = /​var/​log/​dovecot-info.log
 +
 +# Disable SSL for now.
 +ssl = no
 +disable_plaintext_auth = no
 +
 +# We're using Maildir format
 +mail_location = maildir:​~/​Maildir
 +
 +# Authentication configuration:​
 +auth_verbose = yes
 +auth_mechanisms = plain
 +passdb {
 +  driver = passwd-file
 +  args = /​srv/​vmail/​passwd
 +}
 +userdb {
 +  driver = static
 +  args = uid=vmail gid=vmail home=/​srv/​vmail/​%u
 +}
 +</​code>​
 +
 +Save this in /​etc/​dovecot,​ then set it with the correct ownership
 +
 +<code bash>
 +chown root:root /​etc/​dovecot/​dovecot.conf
 +chmod 644 /​etc/​dovecot/​dovecot.conf
 +</​code>​
 +
 +===== Add a user =====
 +
 +First, let's create the file /​srv/​vmail/​passwd and set its permissions:​
 +
 +<code bash>
 +touch /​srv/​vmail/​passwd
 +chown vmail:vmail /​srv/​vmail/​passwd
 +chmod 644 /​srv/​vmail/​passwd
 +</​code>​
 +
 +To add a user, simply make an entry as follows. The extra colons are to conform to the dovecot format, but they can be ignored if you like; the format can be simply username:​password. The difference is, if you want to add additional flags later (like quotas), you'll want the extra colons.
 +
 +<code bash>
 +test:​{PLAIN}test::::::​
 +</​code>​
 +is the same as
 +<code bash>
 +test:​{PLAIN}test
 +</​code>​
 +
 +The text inside the curly braces tells what encryption is used on the password (which immediately follows, no space or anything). Obviously, you don't want plaintext in most situations. ''​doveadm''​ has a pw function which will calculate the hash for you and it gives you the correct format for the entire passwd file format.
 +
 +Here's an example from the dovecot article, where '> ' shows you are typing something at the prompt:
 +
 +<code bash>
 +> doveadm pw -s ssha256
 +Enter new password: foo
 +Retype new password: foo
 +{SSHA256}ZpgszeowIcHdoxe3BNqvUTtPxFd6fMsyQxEWyY0Qlobaacjk
 +
 +</​code>​
 +
 +We could change the entry for the test user above using the line emitted by doveadm
 +
 +<​code>​
 +test:​{SSHA256}ZpgszeowIcHdoxe3BNqvUTtPxFd6fMsyQxEWyY0Qlobaacjk::::::​
 +</​code>​
 +
 +which would give us greater security.
 +
 +===== Setting up mail client =====
 +
 +All of the email clients I know do not allow a stand alone IMAP server; they want an SMTP server to be associated with it. I just use one of my other SMTP servers for this purpose; you won't be sending mail from this account, but if you accidentally do, you'd at least send from your main mail account.
 +
 +Note, after this is done, you can move mail into the archive account manually, or through rules.
 +
 +Following are some of the clients we have run into, and the fixes we made to get them to work.
 +
 +==== OS X Mac Mail ====
 +
 +Actually, Mac Mail (aka '​mail'​) does allow you to say "we have no SMTP server for this" but, you have to modify it after the creation of the account.
 +
 +Do the standard create new account, and it will fail with a verification error. At that point, click the Continue button below. The account will be created, but disabled.
 +
 +Now, follow these instructions to set up the account correctly.
 +
 +  - Go to Mail then Preferences
 +  - Select Server Settings from the popup window
 +  - Select the Archive account on the left panel
 +  - Add the username to the account (it "​failed"​ so Mail removed it)
 +  - Click the Save button which appears at the bottom and wait for verification to complete
 +  - On the Outgoing server, either select None or select a different account
 +    - If you choose //none//, you will get an error message if you accidently try to send from that account
 +    - If you choose a different SMTP server, it will automagically use that account anytime you have the archive account selected and try to send a message
 +  - Click on the SMTP dropdown again and choose Edit SMTP Server List
 +  - Locate and highlight the SMTP server the wizard created. It should have nothing in the "In use by this account"​ column
 +  - Delete the server definition by clicking the minus sign (-)
 +  - Close all windows to return to main screen.
 +
 +==== Thunderbird ====
 +
 +Thunderbird changes fairly regularly, so these instructions cover v52.9.1, but they should at least get you close. Thunderbird requires an SMTP server be associated so, unlike OSX Mail, you have to point the SMTP server to some pre-existing account.
 +
 +  - File | New Existing Mail Account
 +  - fill in information and click Continue
 +  - Select Manual Config
 +  - Click Advanced Config
 +  - Click on the account name itself
 +  - Click on Outgoing Server (SMTP) (bottom of window) and choose a different SMTP server
 +  - Click '​Manage Identities'​ (optional)
 +    - Click Edit on the default
 +    - fill in the information //as if you were sending from other mail server//
 +    - Click Ok, then close
 +  - Go to bottom of account list and select Outgoing Server (SMTP)
 +  - Locate and select SMTP server that was automatically created
 +  - Click Remove
 +  - Click the Ok button.
 +
 +
 +===== End result =====
 +
 +Once you have added the username/​password,​ the user then sets up their e-mail client to access it. Dovecot will create the correct directory tree on first login, so you don't have to do anything else on that. The user can then create folders and use the archives IMAP store to save e-mail which does not need to be on the main server. Additionally,​ e-mail is stored in a known location, in IMAP format, so it is very easy to back up (it is just text files).
 +
 +===== Tuning =====
 +
 +Several things are possible with tuning. If you are using ZFS, setting dedup and compress on will result in very high efficiency as far as disk space is concerned. Or, you can use one of the many scripts available to strip attachments from e-mail messages and stored them (in binary format) in a separate web accessible location for retrieval. Turning mime encoded attachments into standard binary files will result in a space savings of about 25% by itself, and if you use ZFS dedup and/or ext hard links, you can save even more as the fifteen copies of the cute kitty video your friend sent you will only be stored once.
 +
 +===== Automation =====
 +
 +If I can find the script I wrote a long time ago, I'll include it here in the future. But, basically, you can use Perl's IMAP library to go through one or more active IMAP directories and move files from it into this storage area. I used that script to connect to a standard IMAP server and to a Microsoft Exchange server (via IMAP) to auto-clean older mail from the active storage, while maintaining the directory structure. It simply visited each directory (folder) on the active, looking for e-mail which was older than a certain date, then move (creating directories where required). If you write it yourself, be sure to only download the headers when checking a file! Save tons of processing and network traffic.
 +
 +The script could also be set up to remove MIME attachments and store them. A good place to start on that would be in the article [[http://​www.perlmonks.org/​bare/?​node_id=525036]] where they describe how to pull a MIME attachment out and store it as a file. The script could then replace the MIME attachment code in the e-mail with a link to the extracted file.
software/dovecot/archiveserver.txt ยท Last modified: 2018/09/17 19:06 (external edit)