The search is on for ways to navigate invisible Web sites
Some of the newsiest sites on the Internet are the ones you can't see. Conventional search engines never visit a vast area known as "the invisible Web." Instead of Hypertext Markup Language pages and eye-catching graphics, this invisible world consists of databases searchable only by special forms. Say you routinely check airline schedules for government travel. Much of the flight information appears on the Web but not on static pages. To set up a flight via the
Some of the newsiest sites on the Internet are the ones you cant see.
Conventional search engines never visit a vast area known as the invisible
Web. Instead of Hypertext Markup Language pages and eye-catching graphics, this
invisible world consists of databases searchable only by special forms.
Say you routinely check airline schedules for government travel. Much of the flight
information appears on the Web but not on static pages. To set up a flight via the
Internet, youre forced to visit an airlines site, enter your request and check
the database to see whats available. Then you must travel on to the next airline
site.
The government has its own invisible Web. Stop by http://www.fedworld.com
and see the dozens of databases listed. Click on any of them and youll see a form
that must be filled out before you can find what you want. Search engines do index some of
the databases, but for rapidly changing information, a general-purpose engine isnt
the best tool. What you need is a universal form that queries multiple Web databases,
saving you the time and trouble of visiting the sites yourself.
To index and navigate the invisible Web, you would send a query that executes in real
time against multiple databases.
Problem is, because we search databases in so many ways, its nearly impossible to
create a universal form similar to those used by search engines. What works for an airline
site wont work for, say, an IRS database of tax information.
The trick is to break databases down into subsets of similar sites and make a limited
form to query just those sites.
Visit http://www.webdata.com for an example. Look
for the entries under Web 4D, then click on the word Auctions.
From this interface, you enter a search term once, then click on the various icons to
search across multiple auction sites.
Say youre shopping for a fast computer. Simply enter 450 MHz to see whats
available.
Id like to see FedWorld and other government sites adopt this approach for
searching government databases. Its a logical interface and a great time-saver for
databases with similar content and form.
Here are some other navigators for the invisible Web:
Shawn P. McCarthy is a computer journalist, webmaster and Internet programmer for
Cahners Business Information Inc. E-mail him at smccarthy@cahners.com.