Week 10 Notes

David Hawking , Web Search Engines:Part 1&2
Current developments and future trends for the OAI protocol for metadata harvesting. Library Trends
Michael Bergman, “The Deep Web: Surfacing Hidden Value”

These three articles were very informative and interesting. They brought a lot of insight to what I do nearly every day - search the web.
Part 1 of the first article relayed how the ideas of search engines have changes over the years. They have improved greatly. The explanation of the infrastructure and the illustrations of the generic search engine greatly helped my understanding. I also thought the explanation of why some information is not fetched was good. "Before fetching a page from a site, a crawler must fetch that site's robots.txt file to determine whether the webmaster has specified that some or all of the site should not be crawled." This was new to me.Part 2 on the indexing algorithms was more technical and I was glad again for the graphics.
Of all of the articles, I particularly liked Bergman's comparing a web search engine to a net going over the ocean. This brought to mind a very clear picture, even before I got to the graphic. The "deep web" is information packed, but right now our access to it is limited by most search engines.