Made some significant improvements in the AI learner this last week. it’s now been 5 days in a row since anyone release a spam from quarantine or complained about a false positive. Whenever a spam is released from quarantine I get a copy of it and I look into it to figure out what the system did wrong. I had to release one just for grinz to make sure the system was still working.

This is something I was doing in SpamAssassin but SA is a crude tool. What I’m doing is making files with regular expressions where if a number of matched occur the file mane is used as a token indicating a match. for example, I test for lotsofmoney, truseme, africa, banks, religion, etc. All of which are harmless by themselves but when combined can be very effective at classifying spam and ham.

I’m throwing the tokens into my Evolution AI and it creates the associations and scores them automatically. All I have to do it create these files as a way of pointing out what is interesting and the AI does the work. Amazing how well it works. I wrote up a detailed explanation here.

http://wiki.junkemailfilter.com/index.php/Concept_Parsing_Spam_Filter

I decided to post all the details in my support wiki. You can read all about it here.

The new filtering system continues to be beyond amazing. And it’s still getting better. I just filed a provisional method patent on it and I’m hoping to license it to big companies and competitors for royalties. I’m not yet ready to disclose how it’s done but hope to soon.

This isn’t just about blocking more spam. This system is good enough to put spammers out of business.

I have made a serious breakthrough in spam filtering technology. This is far bigger than when Bayesian filtering was considered a breakthrough. It’s so good that I’m looking into a software patent.

I coded it up initially and put it online the evening of December 24th. I’ve been improving it since then. This trick is so accurate that it’s nearly 100%. By “nearly” I mean that one has to assume 100% can’t be achieved. But I expect that in the near future as this evolves that I will see an entire day where not a single spam gets through and not a single good email is blocked.

This new trick (sorry – can’t tell you about it yet until patent pending) is especially good at actively detecting good email. I has a learner so I’m looking forward to next week when everyone comes back to work and starts sending business email. I’m thinking that by the middle of January all false positives will be gone.

I’m thinking I’m going to call it the “Evolution Spam Filtering System”.  I think what I’ve essentially done is code the formula for evolution into software. It’s a recursive learner so the longer it runs the smarter it gets.

Right now the few false positives I’m seeing is from my more traditional spam filter and the Evolution Filter is pointing out errors in my system where I have been getting things wrong. Users should see a serious improvement when everyone comes back to work next week.

I think this new feature – whitelisting subjects – is going to be extremely powerful to avoid false positives. There are a lot of words that spammers would never use in the subjects of their emails that are related to your specific business. Suppose you are an auto repair shop. You might white list phrases like:

• brakes
• alignment
• wheels
• oil change
• paint job
• appointment
• drop off

Unless the spammer is targeting your industry they would never use these words and customers who do use these words would always get through. All you have to do is go through the subject lines of your common emails and look for words that are used a lot but never used in spam.

I’m not sure who is more incompetent, Yahoo or Microsoft, when it comes to email systems. Right now Outlook is down. Outlook is not totally down but can only accept a single message at a time.  I’m a small operation doing spam filtering and email hosting and I have better uptime and email services than Microsoft does. Go figger!

“421 4.3.2 The maximum number of concurrent server connections has exceeded a limit, closing transmission channel”

So – if your hosted at Microsoft and not getting your email – let them know

What the NSA is doing is so morally wrong to rise to the level of a crime against humanity itself. I have decided to put the NSA on notice. If I get an NSA letter I will not keep my mouth shut as instructed. I will immediately publish every detail of all communication with the NSA immediately regardless of how classified or secret they might claim it is.

One can not allow the government to use secrecy as a way to conceal criminal activity. So I will not cooperate with any National Security Letters.

This is funny. More people should do this.

I will be gone from July 12th through July 20th to the ICANN 47 meeting. I’ve been part of an email group for several years on the Non Commercial Stakeholders Group (NCSG) that advises ICANN on policy.

Generally I don’t go to these meetings but two key people couldn’t make it so I was asked if I could fill in. Trying to come up to speed on the issues. I am also concerned about when the NSA is going to make a move on ICANN and compromise the DNS system. This could also be the starting point for a one world government, and if it is I need to help see that it starts out right.

However – during the trip I will be less available by telephone to do tech support. I will be available by email and I’m bringing 2 laptop computers and should be as functional there as I am here in California. But if my response time is slower I apologize in advance.

A message to my customers and to the world,

As most of you are aware of the government of the United States has a spy program called PRISM where they are tied into companies like Verizon, AT&T, Google, Microsoft, Apple, Facebook, Dropbox, Yahoo, AOL, etc. These companies have been given papers from the secret FISA court requiring them to turn over information to the government, and they are required to lie to everyone about what they are doing.

For what it’s worth I can assure you that we are not participating in any kind of spying on the part of anyone. I consider secret courts making secret laws to be unconstitutional and immoral. I have written about and spoken out a lot on this issue. Below are some links expressing the way I feel about what is going on.

Changing the meme on NSA spying

My Interview at Netroots Nation

Democrats Push back on NSA spying

21 Questions Congress should be asking the NSA

For all our international customers (foreigners) I want to apologize on behalf of sane America for what we are doing you the rest of the world and I want to assure you that we try to service all our customers the same regardless of country, race, religion, or any other arbitrary distinction. I personally consider privacy and liberty to be my religious values. We are a little bit English oriented here but that’s just because we’re small.

However ….

Under the law if I were served with a secret order by a secret court or a national security letter it would be a felony for me to let you know that and like Google, Apple, and Microsoft I would be required to lie to you about it. Therefore everything I said to you so far might be a lie. The government has by definition make me into an untrustworthy liar because anything I say might be a cover for a vast conspiracy that I’m forced to be part of.

In fact – if I’m not lying now – but in the future I’m required to lie to you, if I stop fighting the NSA that would, and should look suspicious, as if it were a clue that someone got to me. So – in order to not look suspicious I’m going to be required to keep fighting them. Or at least pretend that nothing has changed. But I’m not as good of a liar as I am at telling the truth so maybe if it happens you’ll notice a difference.

But – the people at NSA just aren’t very bright. If they were they wouldn’t be doing what they are doing because clearly they haven’t thought it through to realize what a very very bad idea it is. So if you see me change my position on this issue then you’ll know that they either got to me or that I have lost my mind.

I would like to think that I’m tough enough to stand up to government threats. I would like to think that I have the courage to do what’s right when the time comes to do the right thing. But it would be foolish to declare myself unbreakable. But it is my hope in posting this notification that I create a logical paradox that the NSA can’t break, so that even if they break me that I you will be able to tell the difference.

Marc Perkel

Support

Junk Email Filter Inc.

I’ve been seeing the DNS ANY attack against my DNS servers. I’m using PDNS but this should work with BIND or any DNS server. My DNS servers are not recursive, but they are attacking me anyway. I guess I’m the DDOS amplifier.

If you think the same thing is happening to you you can detect the attack as follows:

tcpdump -n udp dst port 53|grep ANY

If you are being attacked you will see something like this:

07:32:20.736483 IP 95.130.170.57.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:20.744356 IP 203.217.178.77.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:20.884926 IP 95.130.170.57.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:20.932583 IP 80.241.214.70.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:20.975589 IP 203.217.178.77.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:20.991550 IP 95.130.170.57.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.207135 IP 203.217.178.77.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.349899 IP 95.130.170.57.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.399836 IP 85.25.119.152.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.432679 IP 203.217.178.77.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.454784 IP 95.130.170.57.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.605888 IP 80.241.214.70.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.663643 IP 203.217.178.77.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.884477 IP 85.25.119.152.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)
07:32:21.895233 IP 203.217.178.77.25345 > 184.105.182.241.domain: 10809+ [1au] ANY? isc.org. (36)

After hunting around on the internet I found some code that actually blocks it. This code works:

iptables -v -I INPUT 1 -p udp –dport 53 -m string –from 50 –algo bm –hex-string ‘|0000FF0001|’ -m recent –set –name dnsanyquery

iptables -v -I INPUT 2 -p udp –dport 53 -m string –from 50 –algo bm –hex-string ‘|0000FF0001|’ -m recent –name dnsanyquery –rcheck –seconds 10 –hitcount 3 -j DROP

This allows two ANY queries from an IP over 10 seconds. You can adjust it any way you want. The above code inserts these rules into the first 2. You might also want to just use append and put these in early in the list.

iptables -v -A INPUT -p udp –dport 53 -m string –from 50 –algo bm –hex-string ‘|0000FF0001|’ -m recent –set –name dnsanyquery

iptables -v -A INPUT -p udp –dport 53 -m string –from 50 –algo bm –hex-string ‘|0000FF0001|’ -m recent –name dnsanyquery –rcheck –seconds 10 –hitcount 3 -j DROP

 Is it working? You can test it as follows:

dig @dns.yourdomain.com isc.org ANY

It should word 2 times in a row and fail on the 3rd try. This will allow some diagnostic ANY queries but block the high volume DDOS attacks.

I also have some recursive name servers and I use them heavilly internally. But I also use them lightly externally. These name servers are still open to the world but I threw some rate limiting on them for external use. That way they can be open and yet not abused.

First you have to allow your own network full access:

iptables -v -A INPUT -t filter -s 192.168.1.0/24 -j ACCEPT

Then you rate limit other addresses:

iptables -v -A INPUT -p udp –dport 53 -m recent –set –name dnsanyquery
iptables -v -A INPUT -p udp –dport 53 -m recent –name dnsanyquery –rcheck –seconds 1 –hitcount 10 -j DROP

Here’s I’m allowing 10 hits per second which is far less than an attacker would send.

SPF known as Sender Policy Framework totally sucks. Especially for people who are in the store and forward spam filtering business. That’s because SPF totally breaks email forwarding. But I’ve developed some rules to detect when a server is going to possible reject our good email based on SPF. Basically I’m changing the sender to spfsucks@junkemailfilter.com and sending that and it seems to work.

The worst one to forward to is of course GoDaddy. They apply SPF like rules to web sites that don’t even have SPF records so I’m having to mine the logs for SPF like rejections and put them in a special GoDaddy list that tricks them into accepting our good email. Hopefully it will continue to work. For now it seems to be solving the delivery issue.

Google is migrating Postini to Google Apps and we’re beginning to get more phone calls from people what really don’t want to go that way. We maintain a system for the more independent minded user who doesn’t want to get sucked into the Google cloud. One thing people like is that you don’t have to set up mailboxes. You just turn it on and it runs.

Ever since Yahoo made the mistake of storing password in clear text and having them stolen by hackers it’s been a real headache. Spammers log into yahoo accounts, mine them for email addresses, and then send email from those accounts. Then they have been creating other fake accounts that are similar using those names so even if you change your password there may be a Gmail or Hotmail account owned by the spammer using your contact list and name.

Dear MX Backup Users,

I am contacting you today to let you know about changes to our free MX Backup service. We have implemented a new feature that automatically detects if your normal server is active and will return a 4xx error on the backup server if it is. We only accept and store email for your domain if your primary server is down. The advantage of doing it this way is that it reduces spam that might sneak through or backup servers when spammers try to use us as a back door to get to you.

In addition to our free MX backup service our black lists and white lists are also still free. In fact if you use our backup service we are harvesting spam but data from viruses that would otherwise be attacking your servers and our blacklists are optimized for your domain(s). Here’s information on using our blacklists:

http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists

Why is this free?
===============
People ask why we at Junk Email Filter are giving these services away for free. There are several reasons. The first is that in giving away the free backup service we collect data on virus bots trying to spam you and it builds our black lists that we use in our commercial spam service making it a better service for our paying customers.

We also spend no money on advertizing. Our business is word of mouth and having valuable free services is a good way to generate good word of mouth. To that end if you like the free service we are providing we would appreciate links to us so that we get better index positions on search engines. Our advertizing model is – we make you happy, you tell people how happy you are. We save ad costs and pass the savings on to our customers in affordable pricing.

Spam filtering along with many open source projects is very cooperative. We have created a culture among competitors in this business where we share a lot of technology. That allows all of us in the spam filtering business to provied better products and services allowing the world to get the good email and block the junk. Much of the technology we use we got for free from other people. We therefore feel that we should give services away for free to compensate those who we receive from for free and support the spirit of Open Source and shared technologies.

Outbound Filtering
================
We have a new product. In addition to inbound spam filtering we are now offering outbound filtering and relaying. Who needs outbound filtering? If your email server has been blacklisted because someone in your office got a virus you might need it. If your IP is in a dynamic range and you need to relay through an IP with better RDNS we can be that solution. If yo are an ISP and your users are getting hacked due to weak passwords we can detect the stream and intercept it early so that you don’t get blacklisted.

Some people ask how we do outbound so accurately. It is based on two principles. First – spammers never send spam slowly. So if the messages are coming slow – it’s not spam. If it’s coming fast then we look at the ration of bad recipients and if it’s high – it’s likely a spam stream. In reality it’s more complicated than these two rules but it formed a basis for our system that is working really well.

Governments and your email
=========================
One of the biggest threats to your email isn’t from spammers or hackers, but from governments who want to pass laws to control the Internet. We at Junk Email Filter believe the Internet should be free, private, self regulating, and that your email privacy is something we consider next to sacred. We support the efforts of the Electronic Frontier Foundation (eff.org) to protect online freedom and privacy from unnecessary government intrusion. I personally worked for EFF for 2 1/2 years as their first full time sysadmin.

EFF is moving into a new building and they are looking for donations to help them remodel and expand. I would like to encourage you to give generously to them as they are extremely effective in protecting our online rights. I consider it critical to the future of humanity to keep the Internet protected and hope you feel the same way.

https://www.eff.org/deeplinks/2011/11/double-your-impact-take-eff-mission-challenge

Holiday Wishes
=============
happy Holidays from us at Junk Email Filter wishing you and yours a good fortune in the coming year.

Marc Perkel
Fearless Leader
Junk Email Filter Inc.
http://www.junkemailfilter.com

I’ve been doing more work on my outbound spam filter lately. I’m good at catching a hacked account but it was taking me 3 minutes to figure out that the account was hacked. This allowed the spammer to get a few thousand outbound spams off and that wasn’t good enough. But when you first see the stream you don’t know for a while if it’s good email from someone’s list or a spammer.

So I developed a 10 minute delay queue that I run new streams into. That gives me some time to figure out what’s going on. So by the time the messages come back out of the queue I’ll either send them on or delete them. In theory that should stop all but maybe the first 25 or so. I might have to live with that.

Registered another domain name in addition to Free MX Backup. The new domain is http://www.outboundspamfilter.com.

I have made a LOT of progress recently filtering outgoing spam. Outgoing spam filtering is very different than incoming spam and requires a new strategy. But I’ve figured out a few easy rules that seem to work very well. As you all know I prefer behavior based rules rather than content based rules (like SpamAssassin).

In blocking outgoing spam I make the following assumptions:

1) Spammers send a lot of email. so if the sender is not sending a lot of email, they aren’t spamming. So if the outbound rate is really low I don’t have to even look at it. It’s good.

2) Spammers tend to have a lot of bad recipients in their lists. So if it’s high volume and lots of bad email addresses then it’s probably spam.

This isn’t everything I do but it is the basic rules for outgoing filtering and it’s working really well so far.

I’ve been thinking about what it would take to actually eliminate spam or reduce it to less than 10% of what it is now. One of the problems is the SMTP protocol itself. And a big problem with that is that mail servers talk to each other using the same protocol as users use to talk to servers.

Rather than get all users to change maybe it would be easier to get server software to change. This transition can be done by making server software that can do both protocols to maintain compatibility but will use the new protocol if both sides are capable of talking at that level.

I’m not sure what the specification of the new protocol should be but it should at least be different than what email clients use so that server to server communication isn’t the same as client to server communication. Perhaps server protocols can have more authentication information that would protect them from being spoofed. But having something different – even if it’s just a port change – is better than what we have now.

I’ve noticed a sharp increase in spam coming from Yahoo, Hotmail, Gmail and other free mail providers. Not sure what the cause is but the spammers are gaining ground on these services.