I’m trying to understand the bot problem in the internet and finding more ways to defend myself. One thing that I can’t seem to understand is why most bots, scrapers and crawlers seem to have residential IPs.
- Is it that ISPs are being paid by tech-bros to assign them these IPs?
- Is it that residential devices have been hacked /contain malware that does this?
- Is it trivial for companies to assign themselves residential IPs?
- Paid volunteers are doing this for AI companies?
Or is there is some other reason for this?
Obviously this is a problem because one can rotate / cycle through residential IPs and if I aggressively block each offender in my logs permanently, then the next person assigned this IP who may be a legitimate user will be unable to access my site.
You should consider looking into fail2ban or Crowdsec.
I recently added Crowdsec to my firewall and it seems to work pretty well. I don’t host any public facing services on my network. But maybe someone else with more experience can chime in about it. I believe both tools are pretty good for stopping bots
Both those tools analyze service logs for suspicious activity and block based on patterns. If you aren’t hosting anything externally, they probably aren’t doing anything.
Crowdsec is probably more effective than fail2ban overall, but both are only part of a defense in depth strategy and are really last resort protection.