Set up SPF

Simetrical · May 11, 2008

I've set up [wiki=http://en.wikipedia.org/wiki/Sender_Policy_Framework]SPF[/wiki] so that e-mails coming from TWC can't be forged (at least not very easily). Hopefully this should make recipients trust our mail more and be less likely to mark it as spam. However, if I messed up, mail might not work when this goes live, and it will continue to fail to work for some period even after I do what I can to fix it (the host maintains the DNS servers and they don't update them instantly, not to mention caching). I've set a low TTL for now, one hour, in case something screws up. Be on the lookout.

chriscase · May 13, 2008

If it helps, I sent a legitimate test message to my earthlink hosted account and it was delivered successfully.

If you want to see headers or want me to try a spoof test, feel free to PM me.

Simetrical · May 13, 2008

What do you mean by a legitimate test message? One supposedly from twcenter.net but not from an approved server, or what? I don't actually have any servers I can send e-mail from except TWC, so I can't try to spoof the mail unless there's some third-party service that will do it for me.

Anyway, the mail still seems to be working fine. Gmail is correctly picking up the SPF (addresses removed so as not to encourage spam bots):

Code:

[b]Received-SPF: pass (google.com: domain of [removed]@twcenter.net designates 74.53.195.226 as permitted sender) client-ip=74.53.195.226;[
Authentication-Results: mx.google.com; spf=pass (google.com: domain of [removed]@twcenter.net designates 74.53.195.226 as permitted sender) smtp.mail=[removed]@twcenter.net[/b]
Received: from localhost (odin.twcenter.net [127.0.0.1])
	by odin.twcenter.net (8.13.8/8.13.8) with SMTP id m4DHdpUa000684
	for <[removed]@twcenter.net>; Tue, 13 May 2008 12:39:51 -0500
Date: Tue, 13 May 2008 17:39:51 +0000
To: [removed]@twcenter.net
From: "Total War Center Forums" <[removed]@twcenter.net>
Auto-Submitted: auto-generated
Message-ID: <200805131751.8eb841528215@www.twcenter.net>
MIME-Version: 1.0
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Priority: 3
X-Mailer: vBulletin Mail via PHP
Subject: New Private Message at Total War Center Forums

chriscase · May 13, 2008

I used the TWC interface to send myself a message; I assumed that would neccesarily come from the appropriate IP range.

If you want a spoof test I can send a message to one of my accounts at either earthlink or verizon with an envelope from of test@twcenter.net. Presumably that should fail.

If you want to test google, here is a procedure to do so:

1) Obtain the MX records of smtp servers that accept mail for google.com:

G:\apps\putty>nslookup -q=mx google.com
Server: Wireless_Broadband_Router.home
Address: 192.168.1.1

Non-authoritative answer:
google.com MX preference = 10, mail exchanger = smtp4.google.com
google.com MX preference = 10, mail exchanger = smtp1.google.com
google.com MX preference = 10, mail exchanger = smtp2.google.com
google.com MX preference = 10, mail exchanger = smtp3.google.com

smtp1.google.com internet address = 209.85.237.25
smtp2.google.com internet address = 64.233.167.25
smtp3.google.com internet address = 64.233.183.25
smtp4.google.com internet address = 72.14.215.25

2) If you have a valid google.com email address, use it. I will call mine "test@google.com".

3) If you have a valid twcenter.com email address, use it. I will call mine "test@twcenter.com".

4) Log in to a system that's not in your twcenter.com SPF range.

5) telnet to the smtp port at google's inbound MTA and try to send a message as your twcenter.com address (input cmds in courier):

telnet 209.85.237.25 25
220 smtp.google.com ESMTP
ehlo twcenter.com
250-smtp.google.com Hello <removed>, pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE 20000000
250-STARTTLS
250-DELIVERBY
250 HELP
mail from: test@twcenter.com[/SIZE]
250 2.1.0 test@twcenter.com... Sender ok
rcpt to: test@google.com[/SIZE]
250 2.1.5 test@google.com... Recipient ok
data
354 Please start mail input.
Subject: test
From: test@twcenter.com

test

.

250 Mail queued for delivery.
quit
221 Closing connection. Good bye.

Now, if I could read the email at "test@google.com", I would be able to tell if the spoof worked. I suspect the SPF filter happens after the message is accepted for delivery, so there is no way for me to tell whether the spoof worked simply from the SMTP session.

Of course, the target domain, google.com in this case, is arbitrary. You can use this procedure to verify whether your SPF implementation is working for mail sent to any domain, provided you have a test account in that domain.

Simetrical · May 14, 2008

Yes, of course. But as I said, I don't have a computer to try that from. Google doesn't accept mail from residential computers, for obvious reasons. I could try from my college, but they probably block outgoing port 25 traffic from most computers, also for obvious reasons. I only have shell access to one other server, and that one blocks outgoing port 25 traffic too (at least from non-root users). So I don't have anywhere to test from.

Of course, you would probably want twcenter.net if you have somewhere to try your example, not twcenter.com.

chriscase · May 14, 2008

Simetrical said:
Yes, of course. But as I said, I don't have a computer to try that from. Google doesn't accept mail from residential computers, for obvious reasons. I could try from my college, but they probably block outgoing port 25 traffic from most computers, also for obvious reasons. I only have shell access to one other server, and that one blocks outgoing port 25 traffic too (at least from non-root users). So I don't have anywhere to test from.

Of course, you would probably want twcenter.net if you have somewhere to try your example, not twcenter.com.

Actually, inbound MTAs for all mail domains generally accept mail from the entire internet. SMTP does not distinguish between an MUA like Thunderbird and an MTA that wants to route mail. That was the point of looking up the MX records for the domain. Of course, if you behave badly you may have your IP blocked by an MTA, but one test message is unlikely to cause that. All you need is a computer somewhere on the internet to do the test.

Simetrical · May 14, 2008

The last time I tried that I got this response:

Code:

550-5.7.1 [xx.xxx.xx.xx] The IP you're using to send email is not authorized
550-5.7.1 to send email directly to our servers. Please use
550 5.7.1 the SMTP relay at your service provider instead. f4si2121932nfh.26

But you're right, it works now. It landed in my inbox (I'm too lazy to pick out all the e-mail addresses again, or for that matter IP addresses, so they're uncensored this time):

Code:

Delivered-To: simetrical@gmail.com
Received: by 10.100.232.8 with SMTP id e8cs50701anh;
        Wed, 14 May 2008 07:47:49 -0700 (PDT)
Received: by 10.82.178.3 with SMTP id a3mr122463buf.62.1210776468468;
        Wed, 14 May 2008 07:47:48 -0700 (PDT)
Return-Path: <simetrical@twcenter.net>
Received: from twcenter.net (cpe-72-229-28-14.nyc.res.rr.com [72.229.28.14])
        by mx.google.com with ESMTP id 35si11144271nfu.4.2008.05.14.07.47.24;
        Wed, 14 May 2008 07:47:48 -0700 (PDT)
Received-SPF: softfail (google.com: domain of transitioning simetrical@twcenter.net does not designate 72.229.28.14 as permitted sender) client-ip=72.229.28.14;
Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning simetrical@twcenter.net does not designate 72.229.28.14 as permitted sender) smtp.mail=simetrical@twcenter.net
Date: Wed, 14 May 2008 07:47:47 -0700 (PDT)
Message-Id: <482afb93.2315300a.08cd.09e4SMTPIN_ADDED@mx.google.com>
Subject: Test

Test

So it's just soft-fail, which doesn't seem very useful. Since we've now confirmed that everything is working properly in both directions, at least as far as Gmail is concerned, I've changed it to a hard fail. I'll try again in a few hours, maybe tomorrow, to see if everything is still working properly.

chriscase · May 14, 2008

Simetrical said:
The last time I tried that I got this response:

Code:

550-5.7.1 [xx.xxx.xx.xx] The IP you're using to send email is not authorized 550-5.7.1 to send email directly to our servers. Please use 550 5.7.1 the SMTP relay at your service provider instead. f4si2121932nfh.26

Were you using one of their mail exchangers before? Or their outbound smtp service? Outbound SMTP servers (the ones typically provided to customers as sending servers) are usually configured to prevent relay.

Simetrical · May 14, 2008

I was just using one of the top MX entries.

chriscase · May 14, 2008

Simetrical said:
I was just using one of the top MX entries.

Interesting. Well, ISPs do lots of sleight-of-hand to control spam and other forms of abuse, but the bottom line is they have no reliable way of knowing what is an MTA from some site on the internet that wants to legitemately route mail to them and what is not. SPF is a start, but as you can see, it is not part of the core protocol.

Simetrical · May 14, 2008

Yes, mail is a nightmare, from the naive old days when nobody thought about abuse. It really needs to be centrally brokered through a hierarchal system if we want to be able to reliably stop spam, but there's too much inertia for that. Anyway, it is perfectly possible to determine whether an IP address corresponds to a residential computer with few to no false positives, although many false negatives. Major ISPs make it known which of their addresses are given out to customers. I don't think there's any good reason for such IP addresses to send out e-mails, given that they can easily enough proxy it through any of a large number of free e-mail providers.

Anyway, Gmail now seems to be correctly putting spoofed e-mail in the spam folder, although it doesn't refuse delivery altogether, interestingly:

Code:

Delivered-To: simetrical@gmail.com
Received: by 10.100.232.8 with SMTP id e8cs97075anh;
        Wed, 14 May 2008 16:51:25 -0700 (PDT)
Received: by 10.210.22.16 with SMTP id 16mr1584552ebv.86.1210809080250;
        Wed, 14 May 2008 16:51:20 -0700 (PDT)
Return-Path: <simetrical@twcenter.net>
Received: from twcenter.net (cpe-72-229-28-14.nyc.res.rr.com [72.229.28.14])
        by mx.google.com with ESMTP id 7si1283558nfv.32.2008.05.14.16.51.03;
        Wed, 14 May 2008 16:51:20 -0700 (PDT)
Received-SPF: fail (google.com: domain of simetrical@twcenter.net does not designate 72.229.28.14 as permitted sender) client-ip=72.229.28.14;
Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of simetrical@twcenter.net does not designate 72.229.28.14 as permitted sender) smtp.mail=simetrical@twcenter.net
Date: Wed, 14 May 2008 16:51:19 -0700 (PDT)
Message-Id: <482b7af7.0716300a.08ca.ffffc202SMTPIN_ADDED@mx.google.com>

It seems like it defeats a lot of the point of SPF if a hard fail still permits delivery. Oh well. Google is apparently happier if SPF is provided, anyway, in terms of not marking real stuff as spam.

chriscase · May 14, 2008

Simetrical said:
Anyway, it is perfectly possible to determine whether an IP address corresponds to a residential computer with few to no false positives, although many false negatives. Major ISPs make it known which of their addresses are given out to customers. I don't think there's any good reason for such IP addresses to send out e-mails, given that they can easily enough proxy it through any of a large number of free e-mail providers.

Well, I have worked with ISPs and their email systems for the last 10 years or so, and I have never seen this seriously considered. For one thing, there are a lot of mailbaggers out there who run unix in their home networks and would not appreciate being locked out of the internet mail network. Perhaps more to the point, many of the staff who run ISP mail systems actually belong to this group

There is a very good reason for such IPs generally to be allowed to conduct SMTP traffic: because that is the intent if not the letter of the RFC that defines the protocol.

I suspect that your original session was rejected for some other reason. In any case, if you know of any major ISP that intends to use subscriber access type information to block IP ranges on their MTAs, I'd be interested.

Set up SPF

Set up SPF

Former Chief Technician

Chairman Miao

Former Chief Technician

Chairman Miao

Former Chief Technician

Chairman Miao

Former Chief Technician

Chairman Miao

Former Chief Technician

Chairman Miao

Former Chief Technician

Chairman Miao

Join the Total War Center!

Rome on TWC Wiki

Total War Center on the Wiki

Total War Series on TWC Wiki

Modding Information on TWC Wiki

Medieval II on TWC Wiki

Wiki Links

Thread Statistics

Site Polls

Axis & Allies

Battleship

Checkers

Chess

Clue

Go

Monopoly

Risk

Stratego

Other