NTLM Authentication Scheme for Mail


This develops and expands the description of NTLM Authentication Scheme for HTTP to apply to the Internet standard mail protocols POP3 and SMTP.

NTLM Authentication, sometimes called Secure Password Authorisation, is one of the only two methods of password protection available in Outlook and Outlook Express (I count myself amongst those who don't consider LOGIN to be any form of security). Kerberos is all fine and dandy but, it's a bit overkill for the problem at hand, it needs additional software and open ports on the firewall and it isn't as widely supported at NTLM. NTLM/SPA cannot be considered very secure but it's good enough to discourage the casually curious and anyway, it's the common denominator in the various versions of Outlook. Ok, there's MSN as well, but this is neither the time, nor the place, to go into that.

Much of this discussion is focussed on the behavior of Outlook but I make no apology for that. Since NTLM is an undocumented standard, the behavior of Outlook and Exchange Server must be taken as definitive.

The NTLM/SPA Message Exchange

NTLM exchanges three types of message between the client and server. Depending on the client, the format and use of the messages varies. The format of messages is the same whether used in POP3, SMTP or HTTP - a base64 encode data stream. Although they are discussed individually, usage is very similar across all protocols.

NTLM/SPA and the POP3 Protocol

Outlook and Outlook Express

Outlook clients don't support APOP, use the CAPA command (RFC 2449) or use the AUTH command properly (as described in RFC 1734).

In the absence of the CAPA command, Outlook determines whether the server supports NTLM/SPA by issuing the AUTH command without a parameter. The server should respond with a multi-line response listing the authentication mechanisms available.

S : +OK available authentication methods
S : .

If Outlook doesn't see NTLM in the list of authentication methods, it'll simply report that NTLM/SPA isn't available. It won't bother to try it anyway. And it won't use LOGIN.

The authentication exchange consists of three messages.

* The first is sent by the client to introduce itself to the server. This is message type 1. Amongst other things, it indicates which version of Outlook is in use and whether the authentication is using the user's logon password or mail password (more of which later).
* The second is the server's challenge. This is message type 2. It contains the 'nonce', a string of 8 bytes, unique to the session, which the client will encrypt with the user's password. The format of this message varies depending on whether the client is Outlook or Outlook Express. If the challenge format doesn't correspond to the client then Outlook will terminate the session.
* Finally, the client responds with message type 3. This identifies the user, domain (or work group) and host name. These may be ASCII or UNICODE strings.
The 'nonce' sent in the server's challenge is encrypted with the user's password and returned in this message. The server repeats the encryption process with the stored password for the user and if the response strings match, authentication is complete. The encryption process is explained at LanMan and NT Password Encryption in Samba 2.x and NTLM Authentication Scheme for HTTP. This message can contain the response string in either or both of the NT and LanManager formats.
Outlook Express uses ASCII and only sends the LanManager format response string.

C : TlRMTVNTUAABAAAABoI .... rest of client intro (message type 1)
S : + TlRMTVNTUAABAAAAA4I .... rest of server challenge (message type 2)
C : TlRMTVNTUAADAAAAGAA .... rest of client response (message type 3)
S : +OK logged on

Non M$ Clients

I've used Ximian Evolution in testing. Other Linux based clients seem to be similar. Almost all support APOP, use the CAPA command and use the AUTH command properly. And most support better authentication schemes than NTLM, e.g. CRAM-MD5. But Exchange Server only supports NTLM/SPA so you may be stuck with it. Note that Exchange Server doesn't support CAPA so your client may be unable to determine whether the server supports NTLM; it just has to try it.

For completeness, just in case you want to use NTLM/SPA with a server other than Exchange Server, using the CAPA command will elicit a response like this:

S : +OK eitha.net POP3 server capabilities
S : IMPLEMENTATION eitha.net 1.1
S : .

The client issues the AUTH command, specifying NTLM as the authentication method, but, unlike Outlook we dive right in to the authentication exchange. So the client introduction, message type 1, is not used and the client expects the server to respond to the AUTH command with it's challenge, message type 2. This conforms to the SASL mechanism (RFC 2222).

S : + TlRMTVNTUAABAAAAA4I .... rest of server challenge (message type 2)
C : TlRMTVNTUAADAAAAGAA .... rest of client response (message type 3)
S : +OK logged on

NTLM/SPA and the SMTP Protocol

Outlook and Outlook Express

Here there's much more conformance to the standards. Outlook clients send the EHLO command (RFC 1651) if the "outgoing server requires authentication" option is checked in the mail account settings. The server should list NTLM as one of the supported authentication mechanisms.

Thus :
C : EHLO test
S : 250-mail.eitha.net supports the following extensions

Outlook then issues the command AUTH NTLM and waits for a positive intermediate reply from the server. Any information in this reply, e.g the server challenge, is ignored. Outlook then sends the client introduction message, the server replies with it's challenge and the client sends it's response which is authenticated as above.
If NTLM does not appear in the list of authentication mechanisms listed by the server, Outlook won't try to use it. If LOGIN is listed, Outlook will use that as second best. If none of the methods Outlook knows about (NTLM, LOGIN or KERBEROS) appear, Outlook tries sending the message anyway. It's then for the server to apply it's authorisation rules to an unauthenticated connection. What's more, if the NTLM authentication fails, Outlook will try LOGIN. So much for keeping that password a secret!

S : 334 OK
C : TlRMTVNTUAABAAAAB7I .... rest of client intro (message type 1)
S : 334 TlRMTVNTUAABAAAAA4I .... rest of server challenge (message type 2)
C : TlRMTVNTUAADAAAAGAA .... rest of client response (message type 3)
S : 235 AUTH OK

Note that Outlook gets upset if the intermediate reply codes aren't 334, e.g. 332 will not do.

Message Formats

All Types

All messages start with the same information :

char method[8] = "NTLMSSP\0"
short int type = 1, 2, or 3 - the message type
short int z[3] six null bytes

Thereafter they differ accoding to message type.

Type 1 : Client Introduction

Message type 1 is sent by the client to introduce itself to the server. It indicates which version of Outlook is in use and whether the authentication is using the user's logon or mail credentials. SASL conformant clients don't use this introductory message.

NTLM is integrated into the Windows system. When Outlook is asked to provide NTLM authentication it first attempts to use the Windows logon information, i.e. the username and password you used to logon. If your mail username/password is the same as your logon username/password then this is fine and you'll go straight in. This is typically the case for a Windows workstation connected to a Windows server domain.
If your mail credentials are different, although you have painstakingly entered this information into the mail account settings, Outlook puts up a dialog box and asks for the username and password information for the mail server. No doubt there's an undocumented switch in Windows somewhere that modifies this behaviour but I haven't been able to find it (if you know, please pass the info on). For remote users who don't connect to a domain or for domain-free setups, keeping passwords in sync. can be tiresome. As a saving virtue, Outlook only does this when it's started, not every time it polls the mail server.
You'll be pleased to know that non-MS clients behave in an altogether more reasonable fashion and use the credentials you supply to the mail client.

flags[0] = 0x06 - Outlook Express ; 0x07 - Outlook
flags[1] = 0xb2 - Windows login ; 0x82 - mail client credentials
(The values for flags[1] suggest that this is some kind of bit mask with bit 3 set for Windows login information. Which implies that there's some more information in there. Anyone know?)

Type 2 : Server Challenge

The 'nonce' is a string of 8 bytes, unique to the session, which the client will encrypt using the user's password as the encryption key. It can contain non-ASCII bytes. Values should be unpredictable and, if it's repeated at all, only repeated over a long interval. Typically, the server will use the current date, time, and milli-seconds past midnight to generate the nonce. Alternatives might be a simple counter, perhaps combined with the clients IP address. Since the nonce is clearly visible, the objective is to prevent an attacker capturing the client authentication for a given nonce value and replaying it when that value is repeated.

Type 3 : Client Response

Null response

There is one final type of message that can be issued in reply to any of the other messages, or indeed, the initial positive server reply to AUTH NTLM. It's simply the single character '*' on a line by itself. This means 'I don't want to play anymore'. It's equivalent to QUIT and is usually issued by the client. Outlook sends it if it doesn't like the format of the server challenge or if the user cancels out of the password prompt dialog.
The action required is to send a reply similar to those below, and close the connection.
for POP3 :
S : -ERR AUTH terminated by client

and for SMTP :
S : 535 AUTH terminated by client

Stan Thomas / 24 June 2003 / stan@eitha.net.