First, make sure that the DNS is configured correctly, then install and configure Postfix, Dovecot, MariaDB, SpamAssassin and ClamAV in turn. 1. Set up the system host name and DNS record; 2. Install Postfix and configure virtual user support; 3. Install Dovecot and set mail storage and SSL; 4. Deploy SpamAssassin and ClamAV to achieve spam and virus filtering; 5. Test SMTP/IMAP service and continuously monitor logs, and finally ensure that the mail server runs safely and stably by correctly configuring TLS and authentication mechanisms.

Setting up a mail server on Linux can seem intimidating, but with the right tools and a clear plan, it's entirely manageable. A functional mail server lets you send and receive emails under your own domain, giving you full control over your email environment. This guide walks you through the essential steps using widely supported open-source software: Postfix (SMTP), Dovecot (IMAP/POP3), and MariaDB/MySQL (for user and domain management), along with SpamAssassin and ClamAV for spam and virus filtering.

Note: Hosting your own mail server requires careful attention to DNS, security, and delivery ability. Poor configuration can lead to emails being marked as spam or rejected outright.

1. Prerequisites and Initial Setup

Before installing any mail components, ensure your system is ready:

• A Linux server (Ubuntu 22.04 or Debian 12 recommended)
• Static IP address
• Fully qualified domain name (FQDN) like mail.yourdomain.com
• DNS records properly configured (A, MX, SPF, DKIM, DMARC)

Basic System Setup

 # Update system
sudo apt update && sudo apt upgrade -y

# Set correct hostname
sudo hostnamectl set-hostname mail.yourdomain.com

Update /etc/hosts :

 127.0.0.1 localhost
YOUR_SERVER_IP mail.yourdomain.com mail

Required DNS Records

2. Install and Configure Postfix (SMTP Server)

Postfix handles sending and receiving mail.

 sudo apt install postfix mariadb-server mariadb-client -y

During installation, choose "Internet Site" and set the system mail name to your domain (eg, yourdomain.com ).

Edit Postfix Configuration

Edit /etc/postfix/main.cf :

 myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
mydestination = localhost, $mydomain, mail.$mydomain
home_mailbox = Maildir/

# Enable virtual domains and users
virtual_transport = virtual
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf

Create MySQL Configuration Files

Create /etc/postfix/mysql-virtual-mailbox-domains.cf :

 hosts = localhost
user = mailuser
password = yourpassword
dbname = mailserver
query = SELECT 1 FROM virtual_domains WHERE name='%s'

Repeat similarly for mysql-virtual-mailbox-maps.cf and mysql-virtual-alias-maps.cf (standard queries can be found in most mail server guidelines).

You'll need to create the database and tables for virtual domains, mailboxes, and aliases. Use a guide or script to set up the schema.

Restart Postfix:

 sudo systemctl restart postfix

3. Install and Configure Dovecot (IMAP/POP3)

Dovecot allows users to retrieve mail.

 sudo apt install dovecot-core dovecot-imapd dovecot-mysql -y

Configure Dovecot

Edit /etc/dovecot/dovecot.conf :

 protocols = imap pop3
mail_location = maildir:/var/mail/vhosts/%d/%n

Set up authentication in /etc/dovecot/conf.d/auth-sql.conf.ext :

 passdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}
userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n
}

Ensure the vmail user exists:

 sudo groupadd -g 5000 vmail
sudo useradd -g vmail -u 5000 vmail -d /var/mail/vhosts -s /sbin/nologin
sudo mkdir -p /var/mail/vhosts
sudo chown -R vmail:vmail /var/mail/vhosts

Enable SSL (strongly recommended):

Use Let's Encrypt:

 sudo apt install certbot -y
sudo certbot certonly --standalone -d mail.yourdomain.com

Then point Dovecot and Postfix to the certs in their config files:

 ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

Restart Dovecot:

 sudo systemctl restart dovecot

4. Add Spam and Virus Filtering

Install SpamAssassin and ClamAV:

 sudo apt install spamassassin clamav clamav-daemon -y

Enable SpamAssassin:

 sudo systemctl enable spamassassin
sudo systemctl start spamassassin

Integrate with Postfix via master.cf :

 smtp inet n - y - - smtpd
  -o content_filter=spamassassin
...
spamassassin unix - nn - pipe
  user=spamd argv=/usr/bin/spamc -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}

ClamAV can be integrated similarly or used with amavisd-new for a more robust filter pipeline.

5. Test and Monitor Your Mail Server

Basic Testing

• Send a test email from an external account to your domain.
• Use telnet or swaks to test SMTP:

   swaks --to user@yourdomain.com --from test@gmail.com --server mail.yourdomain.com

• Connect via Thunderbird or Outlook using:
  • IMAP: mail.yourdomain.com , port 993, SSL/TLS
  • SMTP: mail.yourdomain.com , port 587, STARTTLS

Check Logs

Monitor for issues:

 sudo tail -f /var/log/mail.log

Look for delivery errors, authentication failures, or rejection messages.

Final Notes

Running a reliable mail server requires ongoing maintenance:

• Keep software updated
• Monitor blacklists (eg, via mxtoolbox.com)
• Rotate DKIM keys periodically
• Back up MySQL data and configs
• Consider using tools like OpenDKIM and OpenDMARC for better authentication

While self-hosting gives control, it also demands responsibility. Many small organizations opt for third-party services (like Fastmail or MXRoute) for better delivery ability and less overhead.

But if you're learning, experimenting, or need full control—setting up your own mail server is a valuable skill.

Basically, get DNS right, secure services with TLS, and validate each component step by step.

The above is the detailed content of A Guide to Setting Up a Mail Server on Linux. For more information, please follow other related articles on the PHP Chinese website!