Bots & search tools

By Drew RobbSpecial to GCNOnline purchasing, in theory, can deliver the highest-quality products at the best available price because online shopping puts the entire market at your fingertips. The problem is that the Internet contains approximately 575,000 times as many pages as the Los Angeles Business-to-Business Yellow Pages. Your fingertips would have to do a lot of walking to get through them all.That's where bots come in.A robot, or bot for short, is a program that acts as an agent for a user or another program, or simulates a human activity.Robotic software generally is believed to have originated with Eliza, an artificial intelligence program developed by Massachusetts Institute of Technology professor Joseph Weizenbaum in 1966 to study natural language communication between humans and machines. Eliza was programmed to act like a psychoanalyst, answering questions with further questions, and it reportedly could be just as frustrating to talk to as a real shrink.Bots have come a long way. They provide dozens of different online services, assisting not only with purchasing but also with such things as finding news on specific subjects or keeping track of companies within a specific, regulated industry. Some could be of particular use to government information technology shops and other offices: Newsbots continuously monitor online news services and notify a user when articles meeting programmed criteria appear. Users specify both the types of news being sought and the sources to be watched to tailor the news to their exact needs.A programmer can easily set one up to keep the IT department up to date on computer news, one to inform the public relations department of agency press coverage and another to track relevant legislative developments. Although shopbots are often promoted as last-minute shopping tools, they can be harnessed effectively by government purchasing departments to locate the best price on a Sun Microsystems Ultra 60 Model 2450 server, for example. Some shopbots track the price of an item over time, with graphs showing if the price is going up or down.Shopbots also can be set to notify you bye-mail when a product is available at the price you specify, check if the item is available on auction sites, perform automatic price comparisons with other suppliers and compile ratings on the technical support provided by online suppliers.Because each shopbot specializes in specific types of products, an office might use separate shopbots for computer purchases, travel arrangements and other purchases of services or products. At the very least, they can make it easier to get the requisite three quotes for every purchase order. Spiders are robots that follow hyperlinks and make copies of the linked pages to create directories that can be used by search engines.In maintaining a site for an agency or department'whether it's an intranet, extranet or Internet site'a spider can be used to index your own Web site, as well as those of other specified agencies or organizations, to keep the site's search engine completely up to date.For example, the Occupational Safety and Health Administration works closely with the National Institute of Occupational Safety and Health, and its search engine pulls up documents from both organizations' sites. These monitor various sites and inform the user when information has been updated. Depending on the type of bot selected, the user can be notified by an e-mail message, through a customized Web page or by accessing a file that tracks all changes since it was last accessed.For example, an update bot can track new information posted by major companies or trade groups of the industries an agency regulates. Similarly, in managing a centralized Web site to which people in different offices can post data, the update bot monitors the Web site for changes to post on the 'What's New' page. There are several types of data management bots. One stores personal information and automatically inputs it into online forms so they don't have to be filled out manually each time. Besides saving time for one's own staff, the forms could also be offered by an agency's site to improve the public's interface with government.Other data management bots take documents and mine them to create keyword lists or to generate summaries. These keywords and summaries are then used by search engines or internal document storage and processing systems. This enables users to readily locate data without requiring someone to manually summarize documents. Intelligent service agents act like human employees to answer frequently asked questions. They are a step up from voice mail information systems in that people can ask questions verbally and get answers to those exact questions.Although these bots cannot handle all questions, they can reduce the number of calls and e-mail messages agency personnel have to answer. Although hundreds of bots are readily available, there also are bots that assist in creating robots or intelligent agents to meet exact needs, which can save programmers time.There are two excellent online reference sites on bots, explaining how to get them and how they can simplify tasks.The BotSpot, at , contains articles explaining the different types of bots and how they work, as well as reviews of several hundred of them.Webcrawler has a database containing technical data about more than 200 Web robots at .With the Internet now approaching 1 billion pages, it is impossible for one person to keep track of what is happening even in one sector of it.Using robots and intelligent agents, however, makes it possible to get the most out of the vast amounts of data available.We are still a few years away from Jetsons-like robots that clean the house and do the laundry, but software robots are at least available to help with procurement and other routine tasks. XXXSPLITXXX-