Email has undergone a major transformation in the past decade . What began as a simple means to send a quick message has evolved into a business critical application. This increased use of email as a primary communication method has placed an emphasis on maintaining the availability of this service to the users. One of the best methods for diagnosing problems with email is what I refer to as “getting back to the basics”.
Getting back to the basics means separating the email application software from the protocol itself. Application software can be either an MUA (Mail User Agent) such as Outlook or Thunderbird, or an MTA (Mail Transport Agent) such as Exchange or Postfix. We will concentrate on working with the underlying protocols only. Troubleshooting software beyond basic configuration (proper user name and password, correct mail server IP and port number, mailbox setup, etc.) is beyond the scope of this article.
There are 3 main protocols we will deal with when troubleshooting email. SMTP (Simple Mail Transfer Protocol) is used to send email. The SMTP protocol “pushes” messages, whether it's from the desktop to the internal email server, or from one email server to another. It is important to keep in mind that this is a “best effort” or “store and forward” protocol which means that there is no guarantee of successful delivery. POP3 (Post Office Protocol 3) and IMAP (Internet Message Access Protocol) are used to retrieve email. POP3 and IMAP “pull” messages from an email server to the desktop. POP3 downloads the entire message to the desktop when an email is read. IMAP downloads only the email header information when a mail check is performed, the message itself is actually opened and read right on the mail server (making IMAP a bit more bandwidth friendly). Both of these protocols send login credentials to the mail server in clear text unless the secure version of the protocol (POP3S or IMAPS) is used. While there are other proprietary email protocols in use, we will concentrate on the aforementioned 3 which are considered the standards for internet email transport.
The most important tip I can give is to pay very close attention to any error messages you may receive related to mail delivery. If a user states “I sent an email, but I got an error message back”, you need to tell that user to copy the error message down verbatim. Email error messages are usually very specific about what the problem is. Here's an example:
This is the Postfix program at host mail.mydomain.com.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to <postmaster>
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The Postfix program
<joeshmoe@budweiser.com>: host mail1.synacor.com[64.8.70.127] said: 553 sorry,
your envelope recipient is in rejected. either it has been deactivated or
does not exist (#5.7.1) (in reply to RCPT TO command)
The above message comes from my mail server (mail.mydomain.com) and is telling me that my message to joeshmoe@budweiser.com could not be delivered. Specifically, the mail server for “budweiser.com” (mail1.synacor.com[64.8.70.127]) said that the recipient (joeshmoe) either doesn't exist or his account has been deactivated. Note the “553”, this is an SMTP reply code. Anything in the 500 range means “permanent error”, the 400 range means “temporary error/will try again”, and anything in the 200 range means “success/OK”.
Troubleshooting sending (SMTP) errors is done on the command line using the SMTP commands. If you suspect a problem between the desktop and the internal mail server you should do this from the desktop, if the issue is suspected to be outside of your network you need do this from the mail server itself. We'll make a connection to the recipients mail server and attempt to send a message “by hand” using some basic SMTP commands and see what happens. If you are testing from the desktop, you already know what the mail server's IP of FQDN is, if you are testing from your internal mail server to an outside server, you need to look up the name or IP of the recipient's mail server. This can be done with “nslookup” (Windows) or the “dig” command (Linux). Once we know how to get to the recipient's server, we'll make a telnet connection to port 25 (SMTP), then we'll introduce ourselves (the “helo” or “ehlo” command), and finally we will send an email and see what type of reply codes we get. Open up a terminal or command prompt and let's get started!
Look up the mail server:
thughes@thughes:~$ dig mydomain.com mx
;; QUESTION SECTION:
;mydomain.com. IN MX
;; ANSWER SECTION:
mydomain.com. 736 IN MX 5 mail.mydomain.com.
Make a connection (telnet) to that mail server on port 25 :
thughes@thughes:~$ telnet mail.mydomain.com 25
Trying 73.42.65.188…
Connected to mail.mydomain.com.
Escape character is '^]'.
220 mail.mydomain.com at your service ESMTP NO UCE
(notice the 220…that means OK)
Now, introduce yourself by typing “ehlo <something>” (I typed “ehlo howdy!”):
Trying 74.41.65.188…
Connected to mail.mydomain.com.
Escape character is '^]'.
220 mail.mydomain.com at your service ESMTP NO UCE
ehlo howdy!
250-mail.mydomain.com
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-STARTTLS
250 8BITMIME
If you get an error when using the “ehlo” (extended helo) command, the server probably doesn't support the extended version of the command so just use the standard “helo” command. Again, notice all the 250 codes, these mean that this server supports all of the listed options.
Let's send an email:
Type “mail from: <your email address>” (the “<” and “>” are necessary in this command)
Trying 74.41.65.188…
Connected to mail.mydomain.com.
Escape character is '^]'.
220 mail.mydomain.com at your service ESMTP NO UCE
ehlo test
250-mail.mydomain.com
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-STARTTLS
250 8BITMIME
mail from: <thughes@fwpm.com>
250 Ok
(There's that 250 again)
Type “rcpt to: <recipient's email address>”
Trying 74.41.65.188…
Connected to mail.mydomain.com.
Escape character is '^]'.
220 mail.mydomain.com at your service ESMTP NO UCE
mail from: <thughes@someotherdomain.com>
250 Ok
rcpt to: <thughes@mydomain.com>
250 Ok
(250 again)
Now we need to add some data to the email, so type “data” and hit the “enter” key, then type “SUBJECT:” (add a subject), hit “enter”, type “FROM:” (type the from address), hit “enter”, type “TO:” (type the to address), hit “enter” twice, and then type the body of your email. When you're done typing your message, hit the “.” key and you should get a message that the email was sent (“queued”) on the recipient's server, then type “quit”. All of the commands that I typed in the following example are in bold type:
Trying 74.41.65.188…
Connected to mail.mydomain.com.
Escape character is '^]'.
220 mail.mydomain.com at your service ESMTP NO UCE
mail from: <thughes@someotherdomain.com>
250 Ok
rcpt to: <thughes@mydomain.com>
250 Ok
data
354 End data with <CR><LF>.<CR><LF>
FROM: mickeymouse@donaldduck.com
TO: thughes@mydomain.com
SUBJECT: test email
This is a test email
.
250 Ok: queued as 56CBE3B018E
quit
221 Bye
Connection closed by foreign host.
This message was sent successfully (“250 Ok: queued as 56CBE3B018E”). If there was a problem, the reply code would have told me exactly what the problem was. You may also have noticed that when I typed in the “FROM:” field, I used an address other than the one that I used in the initial “mail from:” command, this is because the “FROM:” in the headers of the email do NOT have to match the original connecting (“envelope”) sender. This is how spammers spoof email (heh heh).
Alright, delivery works, now we need to test mail retrieval. Let's do some POP command line magic next. First, we need to telnet to our internal mail server on port 110 (995 for POPS):
thughes@thughes:~$ telnet mail.mydomain.com 110
Trying 192.168.12.80…
Connected to mail.mydomain.com.
Escape character is '^]'.
+OK dovecot ready.
Now we need to authenticate by typing “user <username>”, hit “enter”, then type “pass <your password>” and hit “enter” again:
thughes@thughes:~$ telnet mail.fwpm.com 110
Trying 192.168.12.80…
Connected to mail.fwpm.com.
Escape character is '^]'.
+OK dovecot ready.
user thughes
+OK
pass *********
+OK Logged in.
We're in! Now type “list” to see all the emails (they are listed numerically, the first number is the message number, the second number is the message size), type “retr (message #)” and hit “enter”, and your message will be displayed. Type “quit” to disconnect from the POP server.
Finally, let's try email retrieval using IMAP. Telnet to the internal mail server on port 143 (993 for IMAPS) and log in by typing “a01 login <user name> <password>”:
thughes@thughes:~$ telnet mail.mydomain.com 143
Trying 192.168.12.80…
Connected to mail.mydomain.com.
Escape character is '^]'.
* OK dovecot ready.
a01 login thughes ********
a01 OK Logged in.
Next, lets list our mailboxes. Type “a02 list “” “*”” and hit “enter”:
ao2 list "" "*"
* LIST (\NoInferiors) "/" INBOX
* LIST (\NoInferiors \UnMarked) "/" "Trash"
* LIST (\NoInferiors) "/" "Ebay"
* LIST (\NoInferiors) "/" "Sent"
ao2 OK List completed.
Now we can enter a mailbox:
a02 select INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft Junk $Label1 $Label2 $Label3 $Label4 $Label5 NonJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Junk $Label1 $Label2 $Label3 $Label4 $Label5 NonJunk \*)] Flags permitted.
* 1142 EXISTS
* 3 RECENT
* OK [UNSEEN 1131] First unseen.
* OK [UIDVALIDITY 1157335433] UIDs valid
* OK [UIDNEXT 26939] Predicted next UID
a02 OK [READ-WRITE] Select completed.
Fetch the headers for the first email in the list:
ao2 fetch 1 all
* 1 FETCH (FLAGS (\Seen) INTERNALDATE "18-Aug-2006 09:43:33 -0400" RFC822.SIZE 21975 ENVELOPE ("Thu, 17 Aug 2006 13:06:55 -0400" "Official: Jul-Aug 2006 (ISC)2 Newsletter" (("(ISC)2 Management" NIL "management" "isc2.org")) (("(ISC)2 Management" NIL "management" "isc2.org")) (("(ISC)2 Management" NIL "management" "isc2.org")) ((NIL NIL "thughes" "mydomain.com")) NIL NIL NIL "<LYRIS-1619473-1789-2006.08.17-16.13.08–thughes#mydomain.com@isc16.isc2.org>"))
ao2 OK Fetch completed.
Read some mail:
ao2 fetch 1 body[text]
(This should show you the message body)
Close the connection by typing “ao2 logout”:
a02 logout
* BYE Logging out
a02 OK Logout completed.
Connection closed by foreign host.
That's about it for command line troubleshooting using the basic protocols. The point of the preceding exercises is to confirm that mail can be sent and received. If you are successful using the command line to send/retrieve mail but it doesn't work when using applications such as Outlook or Thunderbird, you can now concentrate your efforts on troubleshooting the software application and configuration.
The logs on your mail server can provide a wealth of information to aid you in diagnosing problems. Becoming proficient in reading the logs is one of the most important things you can do. Take a look at the mail logs and learn how to read them. Try to follow an email through the logs from the time it hit the server until it was delivered to the recipient.
Email headers are also a valuable source of information. You can view the complete headers in most email clients by choosing an option such as “view > all headers”, right clicking the email and choosing “view internet headers”, etc. The exact method depends on which client you are using, Google it if you can't figure out how. Headers are read from the bottom up (use the time stamps as a reference). Below is an example of a full set of headers (I added the comments in bold):
Return-Path: <apache@isc18.isc2.org>
X-Original-To: thughes@fwpm.com
Delivered-To: thughes@fwpm.com
This is my mail gateway delivering the message to my mail server:
Received: from fw.fwpm.com (unknown [192.168.12.1])
by mail.fwpm.com (Postfix) with ESMTP id 280A23B019E
for <thughes@fwpm.com>; Mon, 1 Oct 2007 10:51:03 -0400 (EDT)
This is my mail gateway processing the mail internally:
Received: from fw.fwpm.com (localhost [127.0.0.1])
by fw.fwpm.com (Postfix) with SMTP id 4E54FAC07F
for <thughes@fwpm.com>; Mon, 1 Oct 2007 11:08:13 -0400 (EDT)
This is the spam processing and scoring on my mail gateway (spamassassin):
(sender vscan@fw.fwpm.com)
X-Spam-Checker-Version: mailDefender 3.1.3 (2006-06-01)
X-Spam-Level: S
X-Spam-Status: No, hits=1.9 reqd=5.0 tests=NO_REAL_NAME=0.55,
SPF_HELO_SOFTFAIL=0.1,SPF_SOFTFAIL=0.1,XD_DYNAMIC_IP=0.9, XD_MULTIPART=0.2
Bayes=0.5
X-Spam-Report: * 0.9 XD_DYNAMIC_IP At least one relay is a dynamic IP * 0.6
NO_REAL_NAME From: does not include a real name * 0.2 XD_MULTIPART Email
contain several alternative views * 0.1 SPF_SOFTFAIL SPF: sender does not
match SPF record (softfail) * [SPF failed: ] * 0.1 SPF_HELO_SOFTFAIL
SPF: HELO does not match SPF record (softfail) * [SPF failed: ]
This is my mail gateway receiving the email from the sender's server:
Received: from isc18.isc2.org (isc18.isc2.org [216.12.146.142]) (using TLSv1
with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate
requested) by fw.fwpm.com (Postfix) with ESMTP id E5E2BAC07E for
<thughes@fwpm.com>; Mon, 1 Oct 2007 11:07:48 -0400 (EDT)
This is the sender's server receiving the mail from an internal process:
Received: from isc18.isc2.org (isc18.isc2.org [127.0.0.1]) by isc18.isc2.org
(8.13.1/8.13.1) with ESMTP id l91F7mL0022786 for <thughes@fwpm.com>; Mon, 1
Oct 2007 10:07:48 -0500
Here's the message:
Message-Id: <200710011507.l91F7mmV022784@isc18.isc2.org>
Content-Transfer-Encoding: binary
Content-Type: multipart/related; boundary="_———-=_1191251268227820"
MIME-Version: 1.0
X-Mailer: MIME::Lite 3.01 (F2.73; B3.07; Q3.07)
Date: Mon, 1 Oct 2007 15:07:48 UT
To: thughes@fwpm.com
From: service@isc2.org
Subject: (ISC)2: Confirmation of change to CISSP CPE Record
X-AVAS-Signature: pfilter.pl Version 1.73
X-AVAS-EmailID: 20070901-110749-7603
Status: RO
X-UID: 26809
Content-Length: 761
X-Keywords:
X-Length: 3068
=== This is a system generated message from (ISC)2 ===
Dear T. Hughes:
This email message is blah…blah…blah.
As you can see, the headers provide a detailed record of mail routing. You can use the time stamps at each hop to determine where any latency in delivery may be. Another trick is to use the header information to verify a spoofed email. If an email appears to come from somebody you know but looks suspicious, view the headers to determine the IP address of the original sending server. Perform a “whois” lookup on that IP address; if it comes back as registered to a Chinese or Eastern European domain (for example) you can be certain it is a spoof.
Lastly, I want to comment about email security. Today's users tend to send anything (including very sensitive or personal information) via email without any thoughts of security or confidentiality. Email is not secure! As mentioned earlier, SMTP, POP3, and IMAP are all clear text protocols. This is akin to sending your snail mail on postcards instead of inside an envelope, safe from prying eyes. There are a few basic precautions you can take to secure your email. One would be to use POP3S and IMAPS to retrieve mail. Secondly, you should configure your mail server to support TLS (Transport Layer Security). TLS will allow encrypted SMTP communication with any recipient's server that also supports TLS. It should be noted that you must configure TLS to be optional, if the receiving server does not support TSL the communication will fall back to plain text.
Thirdly, I would suggest the use of an email encryption application (such as Zix mail) to ensure sensitive information is transmitted securely. Finally, user training is a must. Your users need to have a basic understanding of how email works. They need to know that it is neither secure (in it's default configuration) nor guaranteed to be delivered to the intended recipient.
I hope you have found the information and tips in this article interesting and informative. Feel free to contact me if you if you have additional input or would like further information about the techniques I have presented.
Todd Hughes is a Network Security Analyst and long time Linux fan in the Upstate New York area. Send your Questions, comments or feedback to: thughes@fwpm.com.
Copyright Todd Hughes 2007. Printed by permission.
{mos_fb_discuss:174}