Custom Search Engine adds new features

Google Custom Search Engine (CSE) just announced that they now power AdSense for Search. Another nice new change is that if you provide a Sitemap, Google will use that to improve the coverage of the custom search engine.

Now let me nip one idea in the bud, because I can already feel a few people thinking it: this is not a backdoor way to get more pages in Google’s index or to improve your search ranking. The official blog post mentions that. But I think this is a win for webmasters all around. If you use AdSense for search, the product is now better and more tweakable. And if you provide a sitemap, your custom search can get much better as those urls are incorporated into a CSE-specific index.

I’ll just give you a quick anecdote to point out how easy-to-use this product is becoming. My Mom runs a charity that has a web site, and she asked me about site search. We had this email exchange over the course of several hours:

Mom: I wonder if I can add a gadget to search our website like I see on other websites. Is that hard?
Me: This is what people use to add a search box to their site: http://www.google.com/cse/
Mom: It works. I have a search button on my blog!

At my grad school, the goal was to create something so useful that a faculty member would want to use it, and so easy that a faculty member could use it. 🙂 As the email exchange above shows, Google Custom Search Engine is so easy, it’s Mom-approved.

The Custom Search Engine team keeps pushing the technology further and further, and these two changes are major improvements. If you haven’t taken a look at using CSE for your site search recently, it’s worth taking a fresh look.

33 Responses to Custom Search Engine adds new features (Leave a comment)

  1. There does seem to be some confusion over the wording in the post on the AdSense site.

    I read it as saying that it improves indexing over what they used to use to run AdSEnse search but others are interpreting it as saying that it improves indexing in Google.

    Is what I’m reading here that you’re saying it doesn’t improve Google search indexing and that perhaps the AdSense blog wasn’t quite clear?

  2. Darren, it definitely doesn’t help your indexing in Google’s main web search. The custom search engine team essentially is creating a side index and they are willing to use Sitemaps to improve the coverage of custom search engines. So AdSense for search gets better and the coverage of your custom site search gets better. But you won’t get extra urls placed in Google’s main web index. Does that make sense?

    If you want more detail from the webmaster central blog post: “Note that this change will not result in more pages indexed on Google.com and your search rankings on Google.com won’t change. However, you will be able to get much better results coverage in your CSE. …. We’re now maintaining a CSE-specific index in addition to the Google.com index for enhancing the performance of search on your site. If you submit a Sitemap, it’s likely that we will crawl those pages and include them in the additional index we build.”

    I hope that makes sense. We want to improve the coverage for sites that use AdSense for search or a custom search engine, but we didn’t want that to unfairly give a preference in our search results to sites that use these services, either.

  3. Matt,

    “If you submit a Sitemap, it’s likely that we will crawl those pages and include them in the additional index we build.”

    In the same spirit, can one assume that submitting a sitemap would enhance crawling and indexing a site’s pages on GOOG main index (google.com, google.dk ..etc) too?

  4. O.K. that is just ridiculous.
    I once read someone write at webmaster world that if a site did not have a search function on it, that he would just click out. Of course my wordpress blogs have them, but the rest of my site has not.

    This will be the perfect addition.

    It was also funny messing with it and finding that I had four articles with the word lemons in them. Who knew?
    dk

  5. Well one thing about it is it gets me to put up a site map – until now there did not seem to be much point – Google does it job, so why should I bother generating one each week – now I have an answer to that.

  6. I just played around with it. This has come a good way since my first look. I don’t remember the refinements tool at my last look. Looks like I have some testing to do.

  7. I’ve been using Google CSE for a while on all my sites and always wondered why it didn’t power Adsense for Search, the search box looks better 🙂

  8. Matt many people think as you have also said that if they use this search, there are more chances of their web pages to be listed in Google search. But on my experience, there is nothing like that. Google only index you by generating your page rank report and who links you!

  9. Agreed, it is so easy to toss on a site now that it isn’t funny. Either I am just getting more and more familiar with Google Aps or they are getting easier and easier to use (or both). I do need to be better at keeping my sitmap updated though. I have been trying for my company to add CSE to their site since I started working here and they won’t even test it.

    Nice job on giving respect to your Mom.

  10. I think the crucial thing for me is, how much more frequent is the custom search crawl? How much deeper can it go?

    I have to weigh up using my own site search, which won’t be as sophisticated as Google’s but will at least be up to date, against using a Google Custom Search.

  11. It would have been useful if the CSE would let you customise the adsense you were running adsense and a CSE on the same account. If I want to benefit from the new adsense for search, I would have to recreate my CSE via Adsense.

  12. Neato, I mentioned this over a year ago in their discussion group… http://groups.google.com/group/google-custom-search-suggestions/browse_thread/thread/aa69dded951b90d3/0ed0d03d785643cc#0ed0d03d785643cc

    The side index is strange though…is Google trying to decentralize its search? I know that I prefer more comprehensive results when I do a search so now I’m going to have to find or create a CSE when I’m researching a topic. If that’s the case, then perhaps results for relevant CSEs can show up at the top whenever somebody does a regular google search. And/Or it could be an additional tab in the upper left corner in between Images and Maps. Which would mean page rank for CSEs?

    Counter to the post though, I’ve found CSEs results to be less comprehensive than regular search results for individual sites. For example, when I do a search for site:sborchid.com in my CSE (http://www.google.com/coop/cse?cx=007213763730197914006%3Aqnrad4lwhzo) there are 14 results but when I do the same search in regular google there are 1,930 results.

    Also, instead of having to add sites to my database and then add the sites to my CSE…it would be super great if I could create a simple dynamic webpage that displayed the list of sites in my database and then enter the URL of that page into my CSE. That way, I could add sites to the database, my webpage would reflect the changes and every couple of days my CSE would spider that webpage and update the list of sites searched.

  13. If I want to benefit from the new adsense for search, I would have to recreate my CSE via Adsense.

    Is that true? That’s going to kind of be a pain; I have been using CSE on many many sites for over a year now. Bleah.

  14. Siddharth, that’s correct.

    “In the same spirit, can one assume that submitting a sitemap would enhance crawling and indexing a site’s pages on GOOG main index (google.com, google.dk ..etc) too?”

    Harith, that’s not the way it works now, and that’s by design.

    Ros, I don’t know how much deeper/fresher the custom search will be now; I only know that it’s better than before. But if you had low coverage before and added a Sitemap, I would imagine that it could improve quite a bit.

  15. Hmm, I’m comparing the original CSE page to the AdSense for Search page, trying to figure out if it would make sense for me to move from one to the other. (It would take some amount of time, for sure)

    I’m not really worried about coverage, I have that pretty much nailed down already.

    And ease of use will be great in the future, but I already have bunches of these in place.

    One thing I notice – on the AdSense for Search interface, you only get the choices of using the sites you select, or the entire web. On the CSE side, you can also select to search the entire web, but emphasize the sites you select. That’s the way I have mine set up currently; I don’t think I’d want to move to AdSense for Search if that one didn’t work the same way.

    Anybody know?

  16. From a marketers perspective, keeping it sticky is the best feature. It certainly is nice to get more content into the results more quickly, but keeping visitors on the site or using the paid exit (ads) creates an ideal mechanism that is both useful to the visitor and clearly beneficial for the site owner. I like it and will be working to get it implemented on several sites.

  17. Is there a good way to check index coverage with CSE? Using the “site:” operator doesn’t return a total page count for me.

  18. I was going to ask why my CSE does not open the results page in a new window, or tab. It has been like that for at least the last two or three weeks.

    Instead, it shows the text “1 to 10 of about…” in the iphone-styled portal window but you can’t scroll that text in any way to actually see any of the results.

    Before anyone asks; No, it is not any sort of popup blocker in the browser, or some sort of computer protection program. This happens on dozens of machines, at multiple locations, and with all browsers (Mozilla, Safari, Opera, IE) tested so far.

    And then, yesterday, as if by magic, the problem is no longer there.

    I did some edits in the .htaccess file for the site yesterday. I am wondering if that had some effect, but I really cannot see why that would be.

    Finally, can anyone explain this very odd effect. Sometimes, instead of seeing the Google CSE text in the Google CSE mini-window, I actually see the text of the robots.txt file from the site there instead. What causes that to happen, and why?

  19. Thank you!!!
    Thank you!!!
    Thank you!!!

    We were just talking about using Google for site search on Mahalo and we found that many of our pages were not indexed so we had to discount the idea all together. now that we can get all our pages into a CSE we’re golden.

    I love it. Google FTW!

    best j

  20. This is great news. I’ve always hated the way Adsense for Search forces ads to the top of the results. In some cases it means there aren’t even any legitimate search results above the fold, and it’s been my main reason for not using it on numerous sites.

    I’m going add this to at least half a dozen sites right now. Thanks team 🙂

  21. This is probably the best Google freebie ever:

    Instead of having to use a database driven website and an SQL query to get poor quality search results from your site, you can now use Google CSE on your domain, and get paid when you don’t have any relevant content.

    Love it.

  22. Matt,
    Shouldn’t Google be concerned about all of the extra bandwidth that this will consume if it takes off, people put it on their site, and it is widely used?

  23. I’ve been using the CSE for quite a number of months already and it is and always has been monetized by AdSense. It does just fine as it is.

    Why would I want to switch to this new AdSense version? The instructions seem so much harder than those for the regular CSE.

  24. Matt,

    Thanks for feedback.

    “My Mom runs a charity that has a web site,”

    I have deep respect for your privacy. However, don’t you think its time to grant your Mom charity site and charity blog few backlinks to support that charity cause 😉

  25. Dave (Original)

    Now let me nip one idea in the bud…

    Nice try 🙂 You should know by now that most only read what they want to hear.

    Been using MANY CSE on my site for a while now. I have them search by category of each site on the same theme as mine. Great stuff.

    However, I must say that the paid CSE VS the free CSE, doesn’t come close to justifying the cost.

  26. From Netmeg – “One thing I notice – on the AdSense for Search interface, you only get the choices of using the sites you select, or the entire web. On the CSE side, you can also select to search the entire web, but emphasize the sites you select. That’s the way I have mine set up currently; I don’t think I’d want to move to AdSense for Search if that one didn’t work the same way.”

    The process I follow under the new system is:
    1. Set up AdSense for Search under one’s AdSense account.
    2. Use the “Manage Ads” interface to edit the AdSense for Search you just set up.
    3. Go over to the CSE side of things through the “View more advanced search features” link on the AdSense side.
    4. Set up the search engine as if it were a CSE.

    The new CSE settings will then be applied to the AdSense for Search engine you just set up.

    AdSense for Search is DEAD, long live AdSense for Search! :-()

    I do have to say though that setting things up under the old CSE for AdSense seemed more intuitive and straight forward than what currently exists for AdSense for Search.

    The problem is though that I wouldn’t have a clue as to how to straighten it out without simply duplicating the CSE interface for the AdSense for Search interface, although that is already done to a certain extent anyway.

    I know, use a 302 Temporary Redirect from AdSense for Search to CSE so people have everything in one place without becoming confused as to where they are. 😉

  27. Glug. Thanks; I don’t think that solution would scale for me. I so far can’t really see what would be gained by me for changing, so I think I’ll just let it all be for now.

  28. Matt,

    Is there any way to have the custom search engine just search specific areas of google like Google Products (http://www.google.com/products). When you try to add http://www.google.com/products to selected sites it errors out with “Please enter a valid URL.” Tried wildcards as well. It does seem to allow subdomains for Google like maps.google.com but products.google.com is not a valid subdomain.

  29. CSE is a great tool for all of our clients who built sites back in the “who cares if I can never do anthing to it” category. Nice opportunities w/ site search for companies to see how search engines view their site as well.

  30. Dave (Original)

    Matt, considering users have always been able to show AdSense on CSE results, what is the new feature? I know you said Sitemaps, but you also said “*Another* nice new change…”

  31. Any improvement in Google’s site search is a welcome improvement as far as I am concerned. I’ll test it out but I, too, am concerned about crawl frequency (as well as depth of indexing — it does visitors no good if the new CSE site search index doesn’t catalogue all the words on the page).

  32. Matt,

    Running an ecommerce site, I have always had a problem with Search. None of the modules I implemented before, did satisfy my search requirements. So just last week I implemented CSE Business edition. Here is what has happened so far:

    The Good:

    – The search results are the most relevant I have had. I am extremely happy as to the speed and the relevance

    The Bad:

    – I feel like, if you pay money for a service, you should get adequate support. the support link in the CSE main does not provide you with any means to contact google for support. I ended up emailing to the email address on my google checkout receipt and venting frustration until someone responded
    – The Group where you post your question to is full of questions but no answers. It seems no one from google bothers to read the posts
    – The documentation are extremely lacking

    The Real Bad:

    – The implementation is only for http protocol. There is no implementation for secured pages. Thus, your result pages always warns you about privacy issues and cookies being blocked. It seems to me that at least the “Business Edition” must be able to adhere to the https protocol so as not push custometrs away from a Business Site by elluding that there are some privacy concerns about the site they visit
    – There is no documentation as to how to change the look and feel of the result pages. Nor as to how remove Google Branding, even though what documentation exists, tells you that you can remove the Branding. I spent 1 whole day searching through the Group to find an answer.
    – The support team finally got back to me saying: “The issue will be prioritized by our Project Managers and Developers, but
    I’m not able to give you an ETA for this.” Meanwhile, my customers are hesitant to purchase anything from an ecommerce site with a “Privacy Issue”

    Enhancement Requests:

    – It would be great if in addition to text search, the CSE could do an image search as well to list the image beside the text of the result. That would be extremely helpful for an ecommerce site
    – It would be great if you could do other things than simple search. I know that you can do Boolean and other filtering but if you could change the look and feel to something akin to Google’s own advance search, it would make it a lot easier

  33. We are using Google CSE for some time to search free pr and news sites. I remember seeing some Google ads above and below the search result page. So I am confused when you say that “Google CSE powers Adsense for Search”. Isn’t it already using Adsense to generate some revenue?

css.php