Google's Crawler Now Automatically Detects Smartphone Content

Google today announced that its crawler for mobile sites, Googlebot-Mobile, can now detect smartphone-optimized content. The idea here is to ensure that Google’s mobile search can now direct searchers immediately to the smartphone-optimized version of a website. Typically, these sites redirect users to their smartphone sites when they detect a mobile browser. This introduces a slight amount of latency (generally around 0.5-1 seconds) and given Google’s general efforts to speed up the web, this is obviously something the company would like to avoid as it tries to get searchers to their destination as fast as possible.

According to Google, the content crawled by its smartphone-optimized crawler will obviously “be used primarily to improve the user experience on mobile search. For example, the new crawler may discover content specifically optimized to be browsed on smartphones as well as smartphone-specific redirects.”

Google is also reminding publishers to treat Google’s bots just like they would treat a human user. The company’s search team has long frowned upon sites that try to manipulate the search rankings by presenting a special site to Google’s crawler. Some sites, however, treated the mobile crawler differently in the past, as it only looked for feature-phone content. Now, Google is reminding these publishers that it’s time to start “serving the appropriate content based on the Googlebot-Mobile’s user-agent, so that both your feature phone and smartphone content will be indexed properly.”

Here are the user-agent strings that the updated Googlebot-Mobile uses to crawl smartphone sites:

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)