Spam Protection

How I Rebuilt My Dynamic Website as Static HTML..and how that helps fight spammers.

Static HTML. Regenerated dynamically. Whenever I feel like it. Via a custom admin interface.

I added phpBB V2 to my site shortly after it went live. It was highly customized. So it had a similar style to the rest of the site. As I don't check the site frequently, PhpBB auto approved users, allowing them to post immediately. I added a few rudimentary spam detectors to block spammers from registering. Ideas so stupidly simple I am surprised they still work.

Over six years I recorded:
- 5,459 blocked submissions
- 1,759 allowed submissions
- 23 spam submissions that were allowed
- 3,876 blocked registrations
- 1,625 allowed registrations
- 14 spam registrations that were allowed

So my stupid simple trick blocked 99.6% of spam... Until it didn't.

After ten years, the spammers figured out how to bypass my super simple spam detection, and I was overwhelmed with spam accounts, spam submissions and spam comments. So I took phpBB down. I vaguely considered redoing the customizations to phpBB v3, which came with captcha... but what if captcha wasn't enough? So I just kept the simple static HTML, and the site slowly withered away.

Claude Code eventually got me thinking. What could I do with AI? Quite a lot as it turns out. AI allows static HTML to truly scale. AI imported the puzzles from my original HTML into MySQL. I had AI write a script to dynamically generate new HTML straight from the DB, and it looked just like my old hand crafted site. I then migrated all the old phpBB puzzles into the DB and regenerated again. Now all my original puzzles, and the community added puzzles were all on the same page. Literally on the same page, instead of different pages, like they were before. I also added the old phpBB comments to the DB, along with some other feedback.

I then added some dynamic scripts to allow users to submit feedback, new puzzles, or comments. All the content now lives in the MySQL database (DB). Everything is rendered via Python scripts via HTML templates into static HTML pages. Visitors get fast, simple pages. The underlying content can be reviewed, managed, and deleted all from a Python admin dashboard. If everything looks good, I regenerate the HTML, server side. This works really well for spam. If things look spammy, I can fix it before regenerating. Nothing hits the website until I review and regenerate.

I went a little overboard with spam protection this time around. I hate spam. I really hate spam. Hate is actually too mild for my feelings on spam. I loathe spam. Spam is an abomination. I would much rather throw out one genuine submission, than wade through thousands of spam submissions. So. I have multiple spam detection gates. And this is where it gets a little interesting...

Rate Limits: Once spammers figured out my anti-spam techniques in 2019, I was flooded gates of hell with spam. I am still not entirely sure, but my best guess is they wrote scripts on top of a light weight browser and effectively emulated real humans. So, my new system hopefully solves that. If the new system detects unusual activity, the rate limits kick in, and everything is treated as spam. I can use AI to sort it all out at my leisure. None of the spam will appear on my site until I manually regenerate.

Looks Like Spam: I added every automatic spam detection routine I could think of, and anything Claude Code could think of. If it looks like spam, I log it and drop it. I get so much spam, I don't even bother storing it in the database, it goes straight into a log file. I hate-read the log file, to make sure it is operating correctly, before I purge it. Funnily enough, 99.8% of spam is still caught by the same simple spam check that worked in 2008. Yup, the same simple check that failed me in 2019. Why did it fail in 2019, if it is still so effective? My guess is it only takes one spammer to dedicate a little extra horse power to figure your site out, then they can open the floodgates of spam hell. Out of over 1,000 spammers stopped so far, only 2 made it past those same simple checks. What really surprised me, is over the span of the last month, I got as much spam as I used to get over an entire year, back in 2015 (see numbers above).

Over the last two months I have recieved about :
- 600 blocked submissions
- 8 allowed submission
- 1 spam submissions that were allowed

Smells Like Spam: Some of my automatic spam detection routines are a little... aggressive. Plus I am curious. Who is really getting past that first gate? It has a number of simple routines, as well as some surprisingly sophisticated routines for detecting spammers. So if a submission passes the first gate, I don't simply drop the spam, I store the submission in the DB but mark it as spam. From here I can simply delete all spam. I can also individually flag submissions as not spam, and then they move along to the next step.

LLM: Artificial Intelligence (AI.) If the submission makes it past the first two gates, it gets looked at by an LLM. I've thrown $50 at an LLM. With that $50 I should be able to semantically detect spam in over 5,000 submissions. If I get more than 5,000 submissions, I am probably doing something wrong. The LLM categorizes submissions into Approved, Pending, or Spam. Approved means the LLM thinks the submission should be automatically added to the site.

RI: Real Intelligence (RI.) I am the human in the loop. I visually review everything first. If the approved items all look all right, they will show up on my website the next time I regenerate the HTML. The pending submissions either need to be approved or rejected as spam manually by me. Pending means the LLM was not sure what to do. If the LLM gets things wrong too often, I just tweak the prompt. The Spam submissions need a quick glance before they all get deleted. The LLM is really good at spotting spam.

Summary: Is this overkill? No. My hypothesis is there are two types of classic spammers, and a newer breed.

Simple Spammers: These guys search the entire web for a webform with commonly used keywords, and they automatically fill the webform. They flood the internet with low cost spam. Easy to detect even with simple routines. They mostly are in the old business of SEO.

Sophisticated Spammers: These guys use a light weight browser that acts like a mechanical human, or maybe they also use an actual human. It costs a little more, so they are a little more targeted. On a small site like mine, it took ten years for one of them to even bother with my site. That was in 2019. I am not even sure they even bother with small sites anymore. I'll bet their focus is more on hacking into existing social media accounts, then spamming those.

AI Spammers: Eventually, I suspect spammers will use AI. Not just to bypass spam detection algorithms, but also for better AEO. Even if spammers use AI on my site, I have one true advantage. Time. In order for spammers to know if they have successfully spammed me, they need to see their spam on my site. How long will that take? It could be days. Weeks. Months. Years. Because who knows when I will regenerate my static HTML. By which point they will hopefully have moved onto easier prey.

So that's how I rebuilt my dynamic phpBB site with static HTML... and why publishing slowly is a natural impediment to spammers.

What do you think? Let me know.

Send me some feedback. Remember to check the checkbox if you don't want to be published on the wall of fame/shame