Yes, robots are approaching your website! But don’t worry, they are well-meaning robots – so you shouldn’t fear some Arnie lookalike in a leather jacket turning up, and you don’t have to look over your website to delete any mention of Sarah Connor. We are instead talking about the crawlers that Google sends out to analyse websites for its search listings. Google intends for these bots to crawl your site’s JavaScript and CSS files – hence recent advice it has sent out regarding robot.txt files…

Google’s bots will be back…

Many websites have robot.txt files, but many particularly old sites with these files have, it seems, concerned Google. That’s largely because many older robot.txt files have been put in place to direct Google’s bots not to crawl the site’s JavaScript or CSS. Google has recently revealed that, for mobile searches, it will penalize websites that are not optimized for mobile, but a site’s JavaScript and CSS are good for helping Google to determine the site’s suitability for mobile.

It is understandable, then, that the search giant has advised certain sites to update their robot.txt files – and Jumping Spider Media can do this for your own website as part of its services in technical SEO.