Differences

This shows you the differences between two versions of the page.

--- software:dovecot:archiveserver [2018/09/17 19:06] – external edit 127.0.0.1
+++ software:dovecot:archiveserver [2023/09/25 15:19] (current) – rodolico
@@ Line 2: / Line 2: @@
 We have run across this a few times, and thought it might be good to document. A client uses some service which severely limits the amount of e-mail which can be retained. I have seen anywhere from 2G to 10G recently (2016-2018). For some clients, it is a requirement to save e-mail for years, perhaps decades. One common example is the Legal field, where something you did 10 years ago can end up in court.
+Additionally, storing a large number of e-mails on a server can severely impact performance, especially when using e-mail clients which continuously index the Inbox (like Microsoft Outlook) and users who do not store mail in separate folders. Generally speaking, 1000 e-mails per folder is the limit for efficiency except in mail servers which index and store internally like Zimbra (https://www.zimbra.com/) or Microsoft Exchange.
 Many e-mail clients allow archival of e-mail, but they store the information locally, on your workstation, where it is subject to hardware failure, theft, or natural disaster. Additionally, Microsoft Outlook stores all of its e-mail in one huge file, so backups require copying one huge file each time instead of just looking for new/changed files and copying them. Simply opening Outlook can cause the file to be modified, even if you do not do anything with it.
-A much better solution in many cases is to create an IMAP store specifically for archival purposes. In the last two cases we had, the clients were connecting to an Exchange server and already had an internal Unix file server which had an automated/monitored off site backup set up, so it was straight forward to set up an IMAP store for archival purposes.
+A much better solution in many cases is to create an IMAP store specifically for archival purposes. In the last cases we had involving actual e-mail limits, the clients were connecting to an Exchange server and already had an internal Unix file server which had an automated/monitored off site backup set up, so it was straight forward to set up an IMAP store for archival purposes.
 This article covers building a Dovecot IMAP server on Linux, manually setting up the users (and space for them). If you want a pretty GUI (actually WebUI) you might look at installing [[https://www.ispconfig.org/|ISPConfig]] on a new machine or virtual, but we'll cover doing everything manually here. It is mainly taken from the article at [[https://wiki2.dovecot.org/HowTo/SimpleVirtualInstall]].
@@ Line 13: / Line 15: @@
 ===== Setup and Install Dovecot Server =====
-This is pretty straight forward; allow the operating system to install Dovecot, then override the default configuration file.
+This is pretty straight forward; allow the operating systems package manager to install Dovecot, then override the default configuration file.
 Install Dovecot
 <code bash>
-   apt-get install dovecot # debian
+apt -y install dovecot-core dovecot-imapd # devuan/debian
-   yum install dovecot # CentOS
+yum install dovecot # CentOS
 </code>
 ===== Create a user and store for the e-mail =====
-We should use a different user/group for this and all mail will be owned by that user/group. Additionally, we don't want a login, so we'll set the shell to /dev/false. We'll also tell the adduser script to not create the home directory (we'll create it ourselves),
+We should use a different user/group for this and all mail will be owned by that user/group. Additionally, we don't want a login, so we'll set the shell to /bin/false. We'll also tell the adduser script to not create the home directory (we'll create it ourselves),
-Message store (ie, home directory) can be anyplace. I'm going to set it up in /srv/vmail. This will be the head of a tree of subdirectories for individual users.
+Message store (ie, home directory) can be anyplace. I'm going to set it up in /srv/vmail. This will be the head of a tree of subdirectories for individual users. Note, I use useradd (vs Debian's adduser) for simplicity.
 <code bash>
-adduser --home-dir /srv/vmail --gid vmail --no-create-home --shell /dev/false --user-group
+useradd --home-dir /srv/vmail --no-create-home --shell /bin/false --user-group --comment 'Used for vmail only' vmail
 mkdir -p /srv/vmail
 chmod 755 /srv/vmail
-chown vmail:vmail /vmail
+chown vmail:vmail /srv/vmail
 </code>
@@ Line 40: / Line 42: @@
 <code bash>
 mv /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.original
+edit /etc/dovecot/dovecot.conf
 </code>
@@ Line 116: / Line 119: @@
 which would give us greater security.
+If you want, here is a little utility written in Perl that will do all of it for you. It takes a username and a password, then either updates or adds that information to the password file. It is NOT very friendly, and could use some cleanup, and the username/password are left in your history file, so it is insecure. Call it with <code bash>./updatePasswd username 'password'</code>
+<code perl updatePasswd>
+#! /usr/bin/env perl
+# WARNING: This is insecure as it will leave the users password in the
+# bash history file
+use strict;
+use warnings;
+my $pwfile = '/srv/vmail/passwd'; # location of the password file
+my $user = shift;
+my $password = shift;
+die "Usage: $0 username password\n" unless $user && $password;
+my $found = 0; # determines if user already exists
+# call doveadm to get the hash
+my $key = `/usr/bin/doveadm pw -s ssha256 -u '$user' -p '$password'`;
+# read the password file
+open PW,"<$pwfile" or die "Could not open the password file: $!\n";
+my @data = <PW>;
+close PW;
+# go through it and see if the user already exists
+my $newLine = "$user:$key";
+for ( my $line = 0; $line < @data; $line++ ) {
+   my ($thisUser,$thisPass) = split( ':', $data[$line] );
+   if ( $thisUser eq $user ) { # yes, so replace the line and mark found
+      $data[$line] = $newLine;
+      $found = 1;
+      last;
+   } # if statement
+} # for loop
+push @data, $newLine unless $found; # we did not find them, so add
+chomp @data; # remove all line endings
+# write the file back out
+open PW,">$pwfile" or die "Could not write the password file: $!\n";
+print PW join( "\n", @data ) . "\n";
+close PW;
+# tell user what we did
+print "User $user ";
+print $found ? "modified\n" : "added\n";
+;
+</code>
 ===== Setting up mail client =====
@@ Line 176: / Line 233: @@
 ===== Automation =====
-If I can find the script I wrote a long time ago, I'll include it here in the future. But, basically, you can use Perl's IMAP library to go through one or more active IMAP directories and move files from it into this storage area. I used that script to connect to a standard IMAP server and to a Microsoft Exchange server (via IMAP) to auto-clean older mail from the active storage, while maintaining the directory structure. It simply visited each directory (folder) on the active, looking for e-mail which was older than a certain date, then move (creating directories where required). If you write it yourself, be sure to only download the headers when checking a file! Save tons of processing and network traffic.
+I have written a script we use for ourselves and our clients which will automate archiving an active e-mail account to the archive you created.
+<code bash>
+svn co http://svn.dailydata.net/svn/sysadmin_scripts/trunk/archiveIMAP archiveIMAP
+</code>
+Installation consists of placing the script someplace, then installing some Perl libraries (see comment in archiveIMAP script, which has Debian installation line). It is released under the GNU Public License, though I'm thinking of changing it to one of the BSD's.
+Basically, it opens an IMAP connection to the //source// account, then finds any messages over a predefined relative date, copies or moves them to the archive server. I used that script to connect to a standard IMAP server and to a Microsoft Exchange server (via IMAP) to auto-clean older mail from the active storage, while maintaining the directory structure.
+The script can be configured to preserve the directory structure, or modify it as you like. It also has test runs, etc... The configuration file is YAML. The script can be run from the source, the target, or even a third server.
 The script could also be set up to remove MIME attachments and store them. A good place to start on that would be in the article [[http://www.perlmonks.org/bare/?node_id=525036]] where they describe how to pull a MIME attachment out and store it as a file. The script could then replace the MIME attachment code in the e-mail with a link to the extracted file.
+===== Errors =====
+Some e-mail, especially older ones or spam, have malformed dates, or dates which can not be processed by the Perl libraries. In this case, you may receive an error similar to
+<code>
+Use of uninitialized value $t[4] in addition (+) at ./archiveIMAP line 234, <GEN3> line 720660.
+</code>
+and the e-mail in question will **not** be processed. If you find lines like this in your logs, or if you see some older e-mail not being moved, you will need to move manually or delete them.