. Topicala Page Index Token

A Journal about the experiences I have developing little applications in C#, Perl, Html and Javascript and talking about things new things that I use. Always Geeky; Always Nerdy; Always poor Grammer!

I am a Software Analyst Developer working in Southport, England but living in Liverpool. I develop mainly in C# and ASP.Net. I have been developing comercial software for several years now. I maintain this site (hosted at SwitchMedia UK) as a way of exploring new technologies (such as AJAX) and just generally talking about techie geek issues. This site is developed through a host of Perl scripts and a liberal use of Javascript. I enjoy experimenting with new technologies and anything that I make I host here.

Quick Search

Web www.kinlan.co.uk

Wednesday, August 24, 2005

MSN Bot

There is an article over on the Search Engine Journal about MSN bot hammering peoples bandwidth. I noticed this too on my Atom.xml file, it is one of the reasons that my most prominent RSS syndication link is on Feedburner.

MSN Bot, was hitting my Atom XML file once every minute or so from what I could tell, and it had used about 100 MB of my transfere limit. Not much really, but this was only 15 days into the month. My traffic this month has gone through the roof in comparison to previous months. I thouhgt I had more readers, turns out it might have been MSN bot.

Supposedly MSN Bot is a syndication search engine, (the artcile only mentions RSS, but it was getting my ATOM feed). It is searching for new data all the time, thus it hits your feed to see if it has changed. There is information here, and also at MSN about how to slow it down. Basically it involves creating a Robots.txt file in the your directory, which tells the MSN bot to come back in x minutes time.

I created a Robots.txt file, and it reduced the problem substantially.

Technorati Tags
[feed], [feed], [feed], [feed], [feed], [feed], [feed], [feed]

Comments: [Add New]