Home own mail server configuration to fix Cert issue.

 A "TLS/SSL handshake failed" error when sending email has nothing to do with your port 25 being open.

Here's the distinction:

• Port 25 Open (Inbound): This allows other mail servers on the internet to connect to your server to deliver email.

• TLS Handshake Failed (Outbound): This error happens when your server tries to connect to another mail server (like gmail.com or outlook.com) to send an email, and they cannot agree on a secure, encrypted connection.

This problem is almost always caused by one of two things: an invalid/missing SSL certificate on your end, or your server trying to use outdated security protocols that modern servers refuse to accept.

Here’s how to fix it.

Cause 1: Invalid or Mismatched SSL Certificate

When your hMailServer connects to a remote server, it says hello (EHLO) with a hostname (e.g., mail.yourdomain.com). If you then try to start a secure STARTTLS session, the remote server checks your certificate. The handshake will fail if:

• You don't have an SSL certificate configured at all.

• You are using a self-signed certificate that the remote server doesn't trust.

• The certificate has expired.

• The name on the certificate does not match the hostname your server is using.

Solution: Install a Valid SSL Certificate

You need a valid, trusted SSL certificate for the hostname of your mail server. The best option is a free one from Let's Encrypt.

1. Get a Certificate: On a Windows server, the easiest way to get and auto-renew Let's Encrypt certificates is with a tool like Certify The Web or win-acme. Install one of these tools and follow its instructions to generate a certificate for your mail hostname (e.g., mail.yourdomain.com).

2. Configure in hMailServer:

• Go to Settings -> Advanced -> SSL certificates.

• Click Add, give it a name (like "My LetsEncrypt Cert"), and browse to the certificate file (.pfx) and private key file generated by the tool. You will also need the certificate's password if you set one.

• Go to Settings -> Protocols -> SMTP.

• On the Delivery of e-mail tab, find the Host name field. Ensure this name exactly matches the name on your SSL certificate (e.g., mail.yourdomain.com).

• Just below that, in the SSL/TLS section, select the certificate you just added from the dropdown menu.

• Click Save.

Cause 2: Outdated Security Protocols (Very Common!)

hMailServer uses the Windows operating system's underlying security provider (SChannel). If your Windows Server is older or not updated, it may be trying to negotiate using obsolete protocols like SSLv3, TLS 1.0, or TLS 1.1. All major email providers have disabled these and now require TLS 1.2 or TLS 1.3.

Solution: Update Windows SChannel Protocols

Manually editing the Windows Registry for this is complex and risky. The easiest and safest way is to use a free, trusted tool.

1. Download IIS Crypto by Nartac. It's a simple GUI tool that configures the security protocols for the entire Windows operating system.

2. Run the tool as an administrator.

3. Click the Best Practices button. This will automatically select strong, modern protocols (like TLS 1.2/1.3) and disable the old, insecure ones.

4. Click Apply.

5. You must reboot your server for these changes to take effect.

After rebooting, hMailServer will now use the modern protocols, and the TLS handshake with servers like Google and Microsoft should succeed.

Action Checklist

1. Verify Hostname: Double-check the Host name in the SMTP Delivery of e-mail tab.

2. Install Certificate: Get a Let's Encrypt certificate for that hostname and install it in hMailServer.

3. Assign Certificate: Assign the new certificate for outgoing SMTP traffic.

4. Update Protocols: Use IIS Crypto, apply Best Practices, and reboot the server.

Completing these steps will resolve the vast majority of "TLS/SSL handshake failed" errors for outgoing mail.