Googlebot "attacked" my site!

**Availor** · 22 September 2007, 10:29 PM

I had the strangest thing happen to me recently.
This is my server's statistics.

I had about 8,000 visits per day. At first I though it was a counter error, but later on I saw that my website traffic reached 1 gb in a few days !!! I thought that someone is trying to hack my website, but only after checking the ip's of the "intruder" I found out it was no else but googlebot!!

I have immediately uploaded robots.txt file to my server telling googlebot to go away, and the day after the visits ceased.

What went wrong? Why did googlebot keep visiting my site so many times?
I didn't even have meta information prior to the "attack".

These visits took much traffic from my server. That is some strange behaviour from something that should be the "friendliest" thing on the web.

Does anyone know what might be the reason for it? This is the first time such a thing happened eventhough it's not the first time I work with php-fusion on the same server, on different domain (with the same settings).

**RTK** · 23 September 2007, 12:26 AM

I am confused too...

Where in the statistics does it say 8000 visits per day?

Where in the statistics does it say 1GB?

Are you sure you posted the right stats?

Risto

**Manwe** · 23 September 2007, 06:27 AM

He did post the right statics, though you might find that it doesent say exacly 1GB or 8000, I found 15000 and 56000kb...

This behavior is indeed wierd for a googlebot...I've accually never heard of anything similiar...
I even searched a forum for Hackers, and found nothing...(they usually have info on security problems and such)

**RTK** · 23 September 2007, 06:53 AM

Originally Posted by Manwe

He did post the right statics, though you might find that it doesent say exacly 1GB or 8000, I found 15000 and 56000kb...

This behavior is indeed wierd for a googlebot...I've accually never heard of anything similiar...

Ok, so what's the problem again? This is what I see:

19981 35.14% 11141 40.74% 325488 56.23% 38 12.38% crawl-66-249-72-50.googlebot.com And that's for the month.

Looks normal to me? What's weird about it?

Availor, here's what you can do to limit impact from SEs if BW is an issue for you:

(1) Implement a proper robots file. One that asks SEs to stay away from admin / structure files, and only look at content. Why ban them from all of them? I assume you still want to be visible in search engines, and for people to find and come to your site? Ask SEs to stay away from big files (big downloads, movies and games etc.)

(2) Use Google Webmaster tools to tell the robot to slow down its crawl rate.

Both 1 and 2 can take more than a few days to kick in.

Also, once robots learn about your site's update pattern - they will adapt accordingly.

You asked: What went wrong? Why did googlebot keep visiting my site so many times?

It has only visited you 38 times so far? That's nothing...

Risto

**Availor** · 23 September 2007, 08:04 PM

Maybe I posted a wrong statistics file

Trust me on that one I had about 8000 visits per day, and I found that it's not an uncommon problem

http://www.google.com/support/webmas...571&topic=8460

So I'm not the only one - it happens. At least i know how to solve the problem.

**RTK** · 23 September 2007, 10:08 PM

Maybe you were thinking "hits" at 8000? The hits can get high as the server counts everything accessed - every file (including those used to build the page) - every single piece of graphical element etc.

Looking at the stats you had 38 separate visits from Google. This is low... If you have frequently updated site, you would like to see more frequent visits that. The more visits, the quicker your content (will most likely) end up in the serps.

Again, you need a good robots files.

Risto

**RTK** · 23 September 2007, 10:27 PM

Originally Posted by Availor

So I'm not the only one - it happens. At least i know how to solve the problem.

I know it happens too - but it's not you. It would be in the stats.

This is really only an issues for sites that have an incredible amount of content, especially those with big files. Again even more so for sites hosted with limited BW (that are tapdancing around the allowed quota). I guess, server load can become an issue too, if you are being bombarded with SEs and traffic (and have all kinds of (and sometimes poorly designed) backend scripts running).

Risto

**Egg Bramhill** · 24 September 2007, 03:31 AM

Availor, try clickin on the home page of talk graphics where it says "currently active users "and see how often this site is hit by the Yahoo! Slurp Spider. It's quite an eye opener.

**Availor** · 26 September 2007, 05:26 PM

I had 18,000 visits in two - three days I don't think that's normal. Now things are better once I have set robots.txt file.

**RTK** · 26 September 2007, 06:00 PM

Originally Posted by Availor

I had 18,000 visits in two - three days I don't think that's normal. Now things are better once I have set robots.txt file.

Availor, are you sure don't mean "HITS"?

A robot normally looks at more than one page per visit... Also, one single page can result in hundreds of "hits" depending on what is contains - images, php files etc. used to build that one page.

Looking at the updated stats for your site (yesterday the 25 th.) I see:

104 visit (people and robots) - they viewed a total of 703 pages - a total 6402 files were accessed - 12192 hits (note that this number includes the tiniest spacer image) - the bandwidth is negligible (almost nothing).

Where do you see the 18,000 visits?

Risto