SPF stands for Sender Policy Framework. Like you, that meant absolutely nothing to me. What an SPF record does, is tells the recipient email server that an email they just received, is legit "if" it comes from w.x.y.z IP address. This information tells the recipient that the email MUST be somewhat legit, because it came from an IP that is in the SPF record. You have to have access to the DNS Zone records to set the SPF record in the first place, so if you have access to the DNS Zone, the email "MUST" be legit because you manually added an IP address of the mail server who just sent the email. That allows the recipient to have a better trust of the email and gives it a lower spam score on the spam filters.
Let's say your server sends an email without an SPF record. The recipient server looks at it and says: "You just sent me an email from acme.com and the email came from 18.104.22.168 IP address. I have no idea if the IP 22.214.171.124 is actually supposed to send email on behalf of acme.com, so since I can't tell.... eh... we'll throw it to the wolves and protect the user who was supposed to receive it. Or, they might actually pass it through if the content of the email doesn't seem spammy. It can really go either way and it's all up to the receiving email server.
To fix this, you would create an SPF record in the DNS zone for acme.com, to tell the recipient that the email you just sent from IP 126.96.36.199, in fact has permission to send email on behalf of acme.com.
Things to Know
Several things you need to know about formatting and syntax of SPF records.
1) The "Record" is always TXT
2) The "host" can be pretty much anything. In other words, there's no specific host / subdomain parameter (unlike DKIM and DMARC that have specific host parameters). You can use _spf.acme.com, spf1.acme.com, myblocks.acme.com, netblocks.acme.com, _netblocks.acme.com, etc. AS LONG AS the main domains SPF record points to one of these as a domain include (covered later). For example, you need an SPF record on acme.com that lives on acme.com.
3) Always start the record value with v=spf1. (If the protocol gets updated in the future, you'll want to change this...)
4) Always separate the syntax with a space and semi-colons are NOT needed to separate the syntax (unlike dmarc and dkim records which require semi-colons)
5) Even though the SPF is setup and correct, some recipients will still reject your message if rDNS isn't setup (more on this later)
Something else to keep in mind is your hosting environment. Just because your domain is acme.com and the server IP is 188.8.131.52, doesn't mean that's the same IP address your email is coming from. For example, let's say you're on a shared hosting server. There might be 10 IP's on that server, spread out among 300 domains. But, they all use the same outgoing email server, using the IP of the server itself. What can be tricky, is figuring out exactly what that IP is. The easiest way is to send an email to yourself through your website webform, then inspect the headers of the email. It should tell you what the IP address was that sent the email. Gmail is really good about giving you quick access to the headers. If you look at the "original" message, it has a section for SPF and tells you exactly what the IP is that sent the message.
What becomes even more tricky, is cloud hosting services. Since the hosting isn't contained to any one single server, it can be spread out among all sorts of IP addresses and figuring THAT out can be a bit tedious. For example, I helped a client out who was on bluehost.com. For whatever reason, using bluehosts spf record threw up an error that there were too many DNS lookups, so I decided to condense their list to create my own. I had to dig into their SPF records to find the list of IPs that they specify and only add those IPs to my own SPF record. Sometimes, you'll get lucky if you're on a decent hosting platform and you can simply just point to their pre-defined spf record. Other hosting providers, probably don't make that information easy to find or they have a really bad SPF record because they didn't set it up properly.
Getting the Basics
Overall, SPF records are easy to deal with. You either point to an IP address or you include a domain that has IP addresses setup.
Imagine being in the DNS zone editor for acme.com with the IP 184.108.40.206. You would create an SPF record to look something like this:
v=spf1 ip4:220.127.116.11 ~all
The "Host" would be @ and the Record would be a TXT. (the @ means acme.com on some hosting providers like Godaddy. Just know that you need to point back to acme.com)
|acme.com||TXT||@||v=spf1 ip4:18.104.22.168 ~all|
What if acme.com IS the IP for the email server?
There's a option you can add to make things super easy. It's simply adding an a to the list. If acme.com is IP 22.214.171.124, you don't need to add ip4:126.96.36.199. Just add a.
v=spf1 a ~all
This says: "look at the public IP address of acme.com".
What if I use the same IPs that are associated with my MX records?
There's an option you can add to look at the mx records for the domain. This usually isn't needed because a lot of times, the MX records will be different than the outgoing server IP in most cases. Even if you had a server running at your home, you'd probably be fine with just using the a option because incoming and outgoing are both on the same server. Either way, you can simply set the mx option as follows:
v=spf1 mx ~all
If you wanted the MX and the primary domain, you would set it like this:
v=spf1 a mx ~all
Ok, but what if I have a bunch of outgoing email servers on different IPs?
Simple. Just list them out, separating each new ip with ip4:
v=spf1 ip4:188.8.131.52 ip4:184.108.40.206 ~all
What if my IP is ipv6?
Just change the ip4: to ip6 and you're golden :-)
Ok, but what if my server has a LOT of IPs?
This depends... Are the IP's all on the same subnet? If so, you can simply add the subnet mask to the end of the ip4: or ip6:
v=spf1 ip4:220.127.116.11/24 ip6:2000:6000:4000::/36 ~all
If the IP's aren't on the same subnet and they're scattered all over the place, it might be time to wrap them into their own SPF record. You can totally have multiple SPF records that chain into each other. You usually want to start with the top most SPF and use that one to daisy chain down into other records. This is where it gets interesting...
What if I don't need an IP, I need a domain because I was told by my email provider to use something like _spf.acme.com
Hold your horses!! We'll get there... Keep reading...
Wrapping SPF Records
Let's say you control 5 different domains on 5 different hosting servers. Each has it's own IP. They're YOUR servers that YOU control, so you trust them and know that each one is totally legit. BUT, you don't want to keep track of IPs for each and every server, to deal with the SPF records for each domain. Choose a domain that you would consider to be the primary domain of them all. Usually, you'll have a domain that kind of "houses" all of the other domains. Use that primary domain to create an SPF record that houses the IP addresses of all the mail servers for the other domains. (This same concept will work if you use 3rd party email services like mail-chimp, or a CRM like salesforce. They generally provide their SPF records for you to point to.)
Let's say the main company is acme.com and acme.com owns the subsidiaries acme-bricks.com, acme-steel.com, acme-wood.com, acme-stone.com etc, and each domain has it's own email server, therefore IP address.
You would create an SPF record on acme.com to look like this:
v=spf1 ip4:18.104.22.168 ip4:22.214.171.124 ip4:126.96.36.199 ip4:188.8.131.52 ip4:184.108.40.206 ~all
Now acme.com houses the IP addresses for each of the different servers, therefore they're all legit. The next thing we need to do, is tell the OTHER domains, to point THEIR SPF record over to acme.com. So instead of adding the IP address in the SPF for each and every domain, you can set it in one place, acme.com and if the email server IP changes for any given domain, you only have to change it on acme.com because all of the other domains point to acme.com.
This is accomplished by using the include: option for the other domains. So effectively the SPF record for each domain would look like this:
acme.com TXT v=spf1 ip4:220.127.116.11 ip4:18.104.22.168 ip4:22.214.171.124 ip4:126.96.36.199 ip4:188.8.131.52 ~all
acme-bricks.com TXT v=spf1 include:acme.com ~all
acme-steel.com TXT v=spf1 include:acme.com ~all
acme-wood.com TXT v=spf1 include:acme.com ~all
acme-stone.com TXT v=spf1 include:acme.com ~all
When an email is sent through acme-bricks.com, the recipient looks at the SPF record which says: "Hey, go look at acme.com for legit servers". So it looks at the TXT SPF record for acme.com which has 184.108.40.206. It's legit! And passes the email through.
You do need to be careful when crafting these types of SPF records though because you can only have 10 DNS lookups per record. If there are too many lookups, the recipient might reject the email.
Let's look at another situation, where you have domains to other email providers, such as Mailchimp for mass email, or Google Workspace for primary email but you also have your own hosting server that sends email through the website. This isn't very many things to keep track of so they would all fit nicely inside a single SPF record, but let's say you're a neat freak and want an SPF record that ONLY holds domains and another SPF record that ONLY holds IP addresses. You would create 3 total TXT records for this, one top one to house the bottom two:
Here's an example
You would first create an SPF record inside the _spf.acme.com that points to the two lower SPF records:
v=spf1 include:_netblocks.acme.com include:_netdns.acme.com ~all
Then, you setup the respective IP's or domains within each of the bottom spf records:
v=spf1 ip4:220.127.116.11 ~all
v=spf1 include:_spf.mailchimp.com include:_spf.google.com
From here, you set the acme.com SPF record to look at _spf.acme.com, so you can still point your other domains to the central spot include:acme.com:
v=spf1 include:_spf.acme.com ~all
So, if acme.com is pointing to _spf.acme.com and _spf.acme.com is pointing to _netblocks & _netdns, when you point acme-bricks.com to acme.com, it then inherits everything that is housed within acme.com (_spf _netblocks & _netdns).
Yup, it's confusing... That's why you need a tool to help you visualize it.
DMARCIAN has a freaking fantastic tool that shows all of the links for an SPF record and they will tell you if you have too many lookups.
Take a look at their tool against google.com: https://dmarcian.com/spf-survey/google.com
SPF and Subdomains
There might be times when you need to deal with different SPF records on different subdomains. There's a catch: Is the subdomain a CNAME or an A record?
If the subdomain is a CNAME of the primary domain, meaning it is using the same IP as the primary domain, then it will automatically include the SPF record for the primary domain.
Let's use the examples docs.acme.com and api.acme.com
docs.acme.com is a CNAME of acme.com so it points to the SPF record for acme.com and you don't need to change anything.
However, api.acme.com is an A record with it's own IP. In this case, you will need to create a new TXT record for api.acme.com and give it an SPF record. This record can easily just point to the acme.com SPF record or if you need something completely different, it can have it's own unique values.
api A 1h 18.104.22.168
api TXT 1h v=spf1 a include:spf.acme.com ~all
SPF and SMTP Relays
I'll be honest and say that SMTP relays are a bit foreign to me, but from my understanding, you can route all of your email through an SMTP relay and the email will use the relays SPF record to authenticate the message. However, for proper alignment (different from authentication) you still need to add an SPF record that points to the SMTP relay service IP in your own domain DNS records.
If you want a more advanced way of controlling SPF records, you can use what is called DNS SPF macros that are designed to somewhat automate the lookup process dynamically. This is something that is out of the scope of this article, but here are a few places to read up on the topic:
BlueHost DKIM Setup
You spoke of your struggles with BlueHost for one of your clients. I'm with Bluehost and was able to set up my SPF records, but I'm struggling to enable DKIM for my domain. I reached out to support and they simply told me they don't set up DKIM for their customers and just told me to visit http://www.dkim.org/... Did you manage to set it up for your customer finally? If so, can you share how you did it?
I hate to say it, but I honestly don't remember exactly what the outcome was. (I've slept since then) I want to say the client did get BlueHost to comment on the issue, but I can't remember if they were able to set it up, or if he was told that they don't support it at this time but it was in the works... I'm pretty sure it was the latter.
Considering that the DKIM protocol began around 2004 and was finalized to be the protocol it is today in 2011, I'm going to throw it out there and say that: "Any hosting company that doesn't offer DKIM signing as part of their hosting platform, isn't worth using."
Hosting companies that use the true cPanel platform and let you control everything through the actual cPanel interface, should already be setup to offer DKIM as long as you have access to the "Advanced DNS Zone Editor". Companies like BlueHost or MediaTemple want you to use their own in-house DNS GUI and DKIM isn't really on their radar.
The fact that the customer support at BlueHost sent you over to dkim.org, goes to show how clueless they are about how it actually works. The private keys MUST be setup on the originating server and tied to the email portion of that server.
Given the importance of email authentication, it is flabbergasting how many website and email hosting companies, aren't even setup for it. Intuit's online Quickbooks for example, a billion dollar company, doesn't even offer DKIM keys for when a customer sends an invoice on behalf of their domain.
I wish I had better news for you on the matter, but unfortunately, my ultimate recommendation is to dump BlueHost. I tried to talk my client into moving somewhere else, but wasn't successful :-/
Thanks for your reply. Really
Thanks for your reply. Really loved the article by the way. It was the most succinct summary I could find on e-mail authentication.