Set Up a Mail Server on VPS: Easy Step-by-Step Guide

Setting up your own mail server might sound like a daunting task, but it doesn’t have to be. If you’re looking to take control of your email hosting, maintain privacy, and avoid costly third-party services, you’ve come to the right place. In this comprehensive guide, we’ll walk you through how to easily set up a mail server on your VPS using straightforward techniques and proven best practices.

This isn’t just about getting email to work; it’s about building a secure, reliable, and professional mailing system that scales with your needs. Whether you’re a small business owner, developer, or tech enthusiast, this guide is tailored for you.

Why Set Up a Mail Server on Your VPS?

Before diving into the technical steps, it’s important to understand why hosting your own mail server on a VPS is worth considering.

• Full Control: Manage your email accounts, storage, and security protocols without restrictions.
• Cost Savings: Avoid monthly fees on external email services by leveraging your VPS resources.
• Privacy and Security: Keep your communication private without sharing data with third parties.
• Customization: Tailor spam filters, forwarding rules, and user permissions to match your needs.
• Professionalism: Use your own domain email addresses enhancing your brand identity.

Requirements and Preparations for Setting Up Your Mail Server

To start, ensure you meet the baseline requirements and understand what resources you will need for a successful mail server setup.

What You Need Before You Begin

• A VPS with Linux OS: Preferably Ubuntu or Debian, known for strong community support.
• A Domain Name: Required for sending and receiving professional emails.
• Basic Linux Command Line Knowledge: Comfort with SSH and terminal commands.
• Root or Sudo Access: Full administrative rights on the VPS.
• Updated DNS Settings: Ability to modify MX, SPF, DKIM, and DMARC records in your domain hosting panel.

Choosing the Right VPS Plan

Your VPS should have enough resources to handle your expected email volume efficiently. Here’s a quick guideline:

Pro tip: Start small and scale your VPS as your email volume and storage needs grow.

Step 1: Secure Your VPS for a Mail Server Environment

Security is non-negotiable when setting up your mail server. Early steps ensure your VPS is locked down against unauthorized access.

Update Your VPS Packages

Run these commands to update your package index and upgrade installed packages:

sudo apt update
sudo apt upgrade -y

Set Up a Strong Firewall

Limit open ports to only what the mail server requires (SMTP, IMAP/POP3, Webmail if any).

sudo ufw allow ssh
sudo ufw allow 25/tcp    # SMTP
sudo ufw allow 143/tcp   # IMAP
sudo ufw allow 993/tcp   # IMAPS (secure)
sudo ufw allow 587/tcp   # SMTP Submission
sudo ufw enable

This setup shields your VPS while allowing necessary mail traffic.

Create a New User for Mail Administration

It’s best practice not to work as root. Create a dedicated admin user:

sudo adduser mailadmin
sudo usermod -aG sudo mailadmin

Now, switch to this user before installing and configuring software.

Step 2: Install and Configure the Mail Server Software

Several software options exist for mail servers, but Postfix and Dovecot are industry standards for SMTP and IMAP/POP3 respectively.

Install Postfix and Dovecot

sudo apt install postfix dovecot-core dovecot-imapd -y

During Postfix installation, select “Internet Site” when prompted and enter your domain name.

Configure Postfix for Sending and Receiving Emails

Edit Postfix main configuration file:

sudo nano /etc/postfix/main.cf

Adjust the following lines (replace example.com with your domain):

myhostname = mail.example.com
mydomain = example.com
myorigin = /etc/mailname
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
relayhost = 
mynetworks = 127.0.0.0/8
inet_interfaces = all
inet_protocols = ipv4

Save and exit. Reload Postfix:

sudo systemctl restart postfix

Configure Dovecot for Email Retrieval

Edit Dovecot’s main configuration files to enable IMAP and set mail directory:

sudo nano /etc/dovecot/dovecot.conf
# Ensure this line is present
protocols = imap pop3 lmtp

Set mail location:

sudo nano /etc/dovecot/conf.d/10-mail.conf
# mail_location = maildir:~/Maildir
mail_location = maildir:/home/%u/Maildir

Create Maildir folder for your mail user:

sudo mkdir -p /home/mailadmin/Maildir
sudo chown -R mailadmin:mailadmin /home/mailadmin/Maildir
sudo chmod -R 700 /home/mailadmin/Maildir

Restart Dovecot:

sudo systemctl restart dovecot

Step 3: Secure Your Mail Server with SMTP Authentication and Encryption

To prevent unauthorized use and protect data in transit, securing your mail server is vital.

Enable SMTP Authentication in Postfix

Edit your Postfix configuration to enable SASL authentication:

sudo nano /etc/postfix/main.cf

Add or update:

smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes

Reload Postfix:

sudo systemctl restart postfix

Install and Configure SSL/TLS for Encryption

Encrypt email traffic with SSL certificates. The easiest way is to use Let’s Encrypt.

• Install Certbot:

sudo apt install certbot -y

• Obtain SSL certificate (replace mail.example.com):

sudo certbot certonly --standalone -d mail.example.com

Configure Postfix to use the SSL certificate:

sudo nano /etc/postfix/main.cf

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_use_tls = yes

For Dovecot also configure SSL:

sudo nano /etc/dovecot/conf.d/10-ssl.conf
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

Restart services:

sudo systemctl restart postfix dovecot

Step 4: Configure DNS Records for Mail Deliverability and Spam Prevention

Correct DNS settings are crucial to ensure your emails reach inboxes and are not marked as spam.

Update Your Domain’s DNS with These Records:

1. MX Record: Points to your mail server.
2. SPF Record: Authorizes your server’s IP to send email for your domain.
3. DKIM Record: Signs outgoing messages to verify they’re from you.
4. DMARC Record: Instructs receiving servers how to handle unauthorized emails.

Example DNS Records

Note: Generating DKIM keys requires additional steps using a tool such as OpenDKIM, which we’ll cover next.

Step 5: Set Up DKIM for Email Authentication

DKIM (DomainKeys Identified Mail) adds a digital signature to your emails, improving trust and inbox placement.

Install OpenDKIM

sudo apt install opendkim opendkim-tools -y

Configure OpenDKIM

Create and edit the configuration:

sudo nano /etc/opendkim.conf

Paste or update with:

AutoRestart             Yes
AutoRestartRate         10/1h
Syslog                  yes
UMask                   002
Canonicalization        relaxed/simple
Mode                    sv
SubDomains              no
Socket                  inet:12345@localhost
PidFile                 /var/run/opendkim/opendkim.pid
UserID                  opendkim
KeyTable                /etc/opendkim/key.table
SigningTable            /etc/opendkim/signing.table
TrustedHosts            /etc/opendkim/trusted.hosts

Create Key Directories and Generate Keys

sudo mkdir /etc/opendkim/keys/example.com
sudo opendkim-genkey -b 2048 -d example.com -D /etc/opendkim/keys/example.com -s default -v
sudo chown -R opendkim:opendkim /etc/opendkim/keys/example.com

Configure KeyTable, SigningTable, and TrustedHosts

echo "default._domainkey.example.com example.com:default:/etc/opendkim/keys/example.com/default.private" | sudo tee /etc/opendkim/key.table
echo "*@example.com default._domainkey.example.com" | sudo tee /etc/opendkim/signing.table
echo "127.0.0.1
localhost
192.168.1.0/24" | sudo tee /etc/opendkim/trusted.hosts

Integrate OpenDKIM with Postfix

sudo nano /etc/postfix/main.cf

Add:

milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:localhost:12345
non_smtpd_milters = inet:localhost:12345

Restart services:

sudo systemctl restart opendkim
sudo systemctl restart postfix

Add DKIM TXT Record to DNS

Retrieve your DKIM public key:

sudo cat /etc/opendkim/keys/example.com/default.txt

Add the contents (without the selector and domain part) as a new TXT record named default._domainkey for your domain.

Step 6: Test Your Mail Server Setup

Before relying on your mail server, thorough testing assures everything is running smoothly.

Send a Test Email

• Use an email client (Outlook, Thunderbird, etc.) configured with your new mail server credentials.
• Send mail to an external account (e.g., Gmail) and check delivery and spam folder placement.

Check DNS Records Propagation

• Use online tools like MX Toolbox or DNSChecker to verify MX, SPF, DKIM, and DMARC records are active.

Verify Mail Server on VPS

Test Postfix and Dovecot functionality directly on the server:

telnet localhost 25
telnet localhost 143

Step 7: Optional Enhancements and Maintenance

Your mail server is live, but further optimizations improve security, efficiency, and user experience.

Spam Filtering with SpamAssassin

Reduce junk mail with SpamAssassin:

sudo apt install spamassassin -y
sudo systemctl enable spamassassin
sudo systemctl start spamassassin

Webmail Access

Consider installing webmail clients like Roundcube for browser-based email:

sudo apt install roundcube roundcube-core roundcube-mysql -y

Regular Updates and Backups

• Keep your software updated with frequent upgrades.
• Back up configuration files and mailboxes regularly.

Conclusion

Learning how to easily set up a mail server on your VPS opens doors to complete email control, enhanced privacy, and significant cost savings. By following this step-by-step guide, you’ve not only built a functioning mail server but also secured and optimized it for reliability.

Don’t settle for third-party services that limit your freedom—take charge today. Start by securing your domain DNS properly, configure authentication, and continuously monitor your server’s performance. The benefits of running your own mail server are worth every effort.

Ready to elevate your business communication? Set up your mail server now, and experience email hosting on your own terms!