10: Search Engine must crawl frequently

- Tips For Blogging

When you search breaking news on Google, do you ever wonder why websites like CNN, BBC and co always come on top? It’s actually not just because of good SEO (which shall be treated later), it’s because Google has discovered that they have content related to the breaking news you searched for.
Other sites might have this breaking news, but their sites haven’t been visited by Google’s (Ro)bot or spider. Spider is a web crawler that visits a website and saves its content in a database to be used by search engines to learn what a website has and what it does not. It is through the crawled contents that Google is able to search through when you feed it a keyword. Some sites are crawled by Google once in a day, once a week, once a month or even yearly.
You can instruct Google bots to crawl your website or specific pages in a certain frequency. It can be hourly, daily, weekly, monthly or even yearly. You can do this through an XML sitemap. Find out more on Google Webmaster Tools for XML sitemap.
Remember, Google might end up not honoring your request if you set a frequency of “daily” for a specific page but the page is actually updated weekly. Set the crawl frequency to match your update frequency. Setting hourly frequency where it is not necessary will only add up to your server load, and with time, Google will learn to ignore your request.