Most of them are very useful like the GoogleBot, crawlers, and spiders. But others are not so good and usually end up showing when you do not want to.

I «recently» have been doing all sort of tests, attunements, and changes. This with the objective to make the website much more resilient to attacks and solve problems that were left unchecked from the past iteration as mentioned before.

That is when I started to notice how much rampant was the problem of the bots at this place.

For example, here is a graph showing the fall of traffic when changed the «Not found» errors to «Permanently Moved».

Sudden drop around April 24th

That drop sure was dramatic.

Why was it so?

Well, in short, most of the traffic that I had did come from bots and crawlers. They were trying to access parts of this website that did not exist anymore. Especially for a gallery that I set up for photo sharing with people.

That and some links from nodes, created back when using Drupal, that were still indexed at google.

But of course, not all traffic was from nice behaving bots.

Weird Requests from bots

During the time I checked the logs, started to see internal links to stuff like «Get X medicine», «shop for X and such», and so on.

I did not ever remember to have had such stuff on this website!

Since the attempts to access such content was quite often, it makes me wonder if maybe it was some sort of malicious bot, as I did not recognize the «browser string».

The string just ended pointing to a corporation that offered «analytic solutions». I’ll not mention it the name, but it was really weird to see such thing in the logs.

Maybe the bots were just hiding themselves to look less suspicious.

Other requests were much more bizarre. They were attempting to access and change stuff at the old Drupal administration core.

But the prize comes to the one that had me puzzled the most: «/_/_/».

No idea what was the intention of this specific address, probably some sort of exploit attempt but that just came up with nothing now…

 Just to end

While bots are very useful, from crawlers to news feeds fetchers, some of them just have very bizarre and obscure intentions.

Keeping a website safe can be easy if you use the correct software on it and if you keep it up to date. This is maybe the core element in matters of security, alongside a strong password.

But of course, no software or account is safe. Yet, you can do a lot of things to protect yourself from it.

The post Keep out the bots appeared first on Endless Manifesto.