[Comp.Sci.Dept, Utrecht] Note from archiver<at>cs.uu.nl: This page is part of a big collection of Usenet postings, archived here for your convenience. For matters concerning the content of this page, please contact its author(s); use the source, if all else fails. For matters concerning the archive as a whole, please refer to the archive description or contact the archiver.

Subject: Internet Search FAQ 1/2

This article was archived around: Sun, 29 Jun 2003 20:21:39 GMT

All FAQs in Directory: internet/research-faq
All FAQs posted in: misc.writing, misc.writing.screenplays
Source: Usenet Version


Archive-name: internet/research-faq/part1 Posting-Frequency: last week-end of each month Last-modified: 27 June 2003. Copyright-Notice: see end.
Internet Search FAQ Part 1 of 2 ******************************************************************** Part 1 1. WHAT IS THIS FAQ? 2. DISCLAIMER 3. WHY USE THE INTERNET AT ALL? 4. HOW CAN I FIND...? 4.1 How Can I Find Specific files, text, multi-media or people? 4.2 How Can I Find Specific information? 4.3 How Can I Find More General Background Information? 5. HOW CAN I FIND INFORMATION FASTER? 6. SHOULD I PAY FOR INFORMATION? 7. WHERE CAN I GET FURTHER HELP? 8. HOW CAN I VALIDATE WHAT I FIND? 8.1 How reliable is the Net? 8.2 What can I do about it? 8.3 How should Internet sources be cited? 9. WHAT ABOUT THE FUTURE? Part 2 10. URLS FOR A RAINY DAY - useful links for research 11. END CREDITS This FAQ is available on the web at <http://www.search-faq.com> along with an archive of changes. It is updated and posted roughly the first weekend of each month, circumstances permitting, to: misc.writing, alt.movies.independent, alt.union.natl-writers, misc.writing.screenplays, alt.answers, misc.answers and news.answers. At the same time, any changes to the FAQ, including new resource links, are posted to all except the *.news groups in a separate message "Internet Search FAQ - What's New", to save users having to download the entire FAQ each month just to find out what's been added. This can also be found at <http://www.purefiction.com/pages/wotnew.htm>. Keen searchers are urged to check this out regularly for new ideas and links, and those with clever browsers (Navigator and Internet Explorer 4 and above) can set them to "subscribe" to the page on a regular basis - if they can work out how. All suggestions and comments are welcome. Please send to charlie@harris.u-net.com. ******************************************************************** 1. WHAT IS THIS FAQ? Although this posting was compiled originally for writers, it has become increasingly clear that this FAQ (Frequently Asked Questions) list is of use to anyone who wants to find their way around the Net. It grew out of a cry for help that I sent out, in desperation. As a professional writer, I wanted information of a variety of types. One day I might want specific dates, another day just background information. I wanted to know if I could use the Internet to find these different types of information quickly and reliably. And I wanted to know which of the many different bits of the Internet would be good for which different type of search. However, the vast majority of books, articles and Usenet postings do not address the question from the point of view of the user, and tend to be obsessed with either vague surfing or searching out free software. The last thing I wanted was yet more software. I was pleased to receive a number of responses to that original cry for assistance - useful and supportive answers, which gradually became the foundation of this FAQ. The FAQ tries to look at the Net from the point of view of the user. So it is divided into the kinds of questions that Net searchers might have. It also includes "worked examples" where possible, to clarify the methods that can be used. Finally there is a list of useful URLs (Urls For A Rainy Day) which includes most of those mentioned in the main text plus a load more, and is also available at <http://www.purefiction.com/pages/res2.htm>. I haven't tried to explain what all the technical terms mean (eg: URL, ARCHIE, FTP...) These are very adequately explained in a thousand postings, books and magazines. The problem is knowing which to use in which circumstances. The Internet is constantly changing, and so I welcome any suggestions, criticisms and additions. However, most Net users are snowed under with URLs, etc, so please send personal recommendations, or that of someone you know, and say why or how it is useful. (For example, state that a particular URL is good for geographical queries, or how you used Gopher to research background for your romantic novel). ************************************************************************ 2. *DISCLAIMER* Information, advice, URLS, e-mail addresses, etc, are generally included on the recommendation of satisfied users. They are passed on herewith without prejudice! I've not checked all of them out, and make no guarantees that they are accurate, useful, or still appropriate, or in fact ever were. I take no responsibility for any loss, damage or waste of time in using them. Sorry. But please do tell me if an URL turns out to be useless, or non-existent, so that the information can be kept up- to-date. ************************************************************************ 3. WHY USE THE INTERNET AT ALL? 3.1 If you want to use the Net effectively, you need to be prepared for what it can and can't do. The Internet is not a substitute for a good library. The Internet can be very frustrating. The Internet is very variable. The Internet is not well indexed. And the Internet is not comprehensive. So is it worth using at all? Well... 3.2 The Internet is an additional source of information, which often can't be found, or isn't as up-to-date, elsewhere. "Searching for data on Internet can be frustrating but what you find often can't be found in a library -- the same is true in reverse. I didn't stop using the library when I started using the Internet." (writer Laurence A.Moore) 3.3 The Internet is convenient, and supplies information in usable form. "One handy thing about Internet research is that when I'm done, the results are on my computer. With the library, the best I can do is photocopy what I find, or bring the books home and type the data in. "Looking out the window above my computer, I see birds and autumn-colored trees and calm, quiet, gently-falling rain. As soon as I send this, I'm going to bring a mug of fresh coffee back from the kitchen and take off on Internet. Can't do that at my local library!" (Laurence A.Moore) 3.4 However, the Internet has to be worked at. The "superhighway" is still substantially under construction. As one writer put it: "the Internet is an enormous library in which someone has turned out the lights and tipped the index cards all over the floor." (Or, variously, "Like trying to work off the librarian's notes after discarding the card catalogue," Allen Schaaf). 3.5 Be realistic and focused about what you want to find. Do you want a precise fact, or more general background material? How will you know when you've found enough information - or when to stop trying? Faced with the enormous size of the Net, it's tempting to believe that the ideal link is just around the next corner, but some types of information simply aren't there, while other information may exist on the Net, but be extremely difficult to locate. Sometimes, to be honest, there are easier ways: a phone call, the local bookshop, a friend of a friend. Nevertheless, the more you learn about the Internet, the more you become aware of what it can and can't do. The most difficult way to approach the Internet is when you already have a large and urgent piece of research to conduct. Better to check out small areas of it without stress, for a few minutes at a time, on a regular basis. Give yourself a chance to play about with the Net when the pressure is off, so that when the pressure is on you can find what you need quickly and efficiently. *********************************************************************** 4. HOW CAN I FIND...? What's the best and most efficient way to look for what I need? (Here we look at some ways of finding the different kinds of information that's on the Net.) 4.1 How can I find Specific Files, Texts, Media (images, sounds, etc) or People? ------------------------------------------------------- 4.1.1 How can I find specific file by name? The more precise you can be with your search, the better. So if you have a precise filename, you've got the best chance of finding what you want. Many search engines and meta-search engines now have facilities for searching for software files, etc). Try Google, for example <http://www.google.com/> or many of the others listed in URLs For A Rainy Day (9.3.4). There are many books, articles, etc, on the Internet which show how to search for specific filenames, using Archie, etc, so this is not dealt with further in the FAQ. However, researchers rarely have a precise, or even imprecise, filename. So.... 4.1.2 How can I find a specific text? There are an increasing number of web and FTP sites which hold public domain copies of a wide range of classic texts, song lyrics, etc. Some links are given in part two of this FAQ - URLs For A Rainy Day (Section 9.7). You can also link to some of these via: <http://dspace.dial.pipex.com/jane.dorner/jd_links.htm> There are history archives on the Internet and a number of libraries on the Net. For example, David Brager suggests the Library of Congress' American Memory section - "Large collections of primary source and archival material relating to American culture and history." <http://rs6.loc.gov/>. In addition, increasing numbers of search engines will allow you to search across a number of search engines for specific items such as lyrics. One such is OnlineSpy <http://www.onlinespy.com/>. See Section 4.2.2 for discussion of other such "metasearch" engines and 9.3.4 for a list of metasearch engines to use. 4.1.3 How can I find a specific image, movie or sound? Many metasearch engines, such as OnlineSpy (see above) will allow you to search for images specifically - or even sounds or movie clips. You may however need to be very precise with the terms you search with (see 4.2.1 below for how to use search engines with precision). One particularly useful site is Image Surfer <http://isurf.yahoo.com> recently developed by Yahoo. Image Surfer is a search engine which you can search by category or using search terms, but instead of giving its answers in text form it produces a series of small thumbnail images. Much the most useful image searcher I've yet seen, Image Surfer's capacity is still small, but Yahoo promise it will grow in size. Well worth checking out. ImageFinder <http://sunsite.berkeley.edu/ImageFinder/> gives you a number of different databases to search for a variety of types of image - eg: the Smithsonian Photographic Collection or Colombia University Image and Video Catalog. Useful for both pictures and sound is the search engine HotBot <http://www.hotbot.com> which provides tick boxes to allow your search to include still images, video or audio sound clips, or even shockwave animations. Said to be one of the best MP3 search engines at the moment. 4.1.4 How can I find specific people? There are many resources on the Net that can help you locate and even make contact with specific people - famous or not, individuals or companies. Whether they'll be of any use to you will depend on a number of factors, not least geographical. As with so much on the Internet, the vast majority of resources are devoted to the USA. So there's little difficulty in finding directories and databases with look-up or even reverse look-up facilities covering just about every member of the US population, alive or dead. (Particularly intriguing, in passing, is Ancestry.com <http://ssdi.ancestry.com/> which among its useful resources for genealogical research allows you to find the social security number and other details of any dead American.... and then offers a facility to write a letter! Do they know of some postal service that we don't?) More wide-ranging are the directories of email addresses. However these are far from all-inclusive, even assuming your target has an email address. Some Internet Service Providers - such as CompuServe and AOL used to provide a look-up service which included all subscribers (and probably still do) but only for other subscribers, as I understand. For the rest, directories such as BigFoot <http://www.bigfoot.com> rely on finding email addresses of those who have web-pages or post regularly to newsgroups. By no means does this include everybody. Expect to have to try a number of sites before you find a lead. In Urls For A Rainy Day - Section 9 - there are numerous search facilities. 9.3.4, 9.3.5 and 9.11.2 give a number of meta-search engines, people searchers and reference sites which offer specific people-finding databases. Particularly useful are those such as All-In-One <http://www.allonesearch.com/> or Langenberg <http://www.langenberg.com/> which have links to many different "people" sites on one page. There are also databases devoted to certain types, eg: politicians (9.13.2). Organisations are generally easier to find through a search engine. But even then it is not always easy - especially if the organisation doesn't have a web page of its own. However, David Brager writes to inform us that if you know a domain name you can use it to find all kinds of details, from contact e-mail and snail-mail addresses to phone numbers at <http://www.vservers.com/before/dnscheck/>. Whether looking for people or organisations, in difficult cases you may need to try the more refined methods for finding information by using Search Engines, or posting questions on Newsgroups or Mailing Lists, as described in the next section. 4.2 How can I find Specific Information? ---------------------------------------------- (eg: dates and places. Or questions like: "what is a...?" "who is...?") 4.2.1 SEARCH ENGINES are popular for this. You type in a key word or phrase (such as Spain, or Spanish Civil War) and wait to see what they provide. The popularity of search engines on the Net can be changeable. When I started this FAQ there was no clear winner. Then Alta Vista appeared <http://www.altavista.com/>, and for some time beat all the others hands down. For at least two years Google has taken over at the top <http://www.google.com>. Google has many strong points, including simplicity, a lack of adverts and the ability to check its own "cache" of pages if the page you're looking for has temporarily disappeared. But no search angine is perfect and different people have their different favourites. You can find many other good search engines, each with its own paticular strengths in our list of links - Urls for a Rainy Day. The trick with using a search engine, is to know what each is good for and to look carefully at the hints and tips that they offer. For example some engines will only search for a precise phrase if you put it in quotes - such as: "Spanish Civil War." Planning is necessary for any search. Do some advance work with a Thesaurus and list a fair number of relevent search terms. Remember that search engines aren't like "Find" facilities on word processors. So you can afford a scattergun approach, trying a number of related words at the same time in case one of them hits home. For example: in starting a search for items on dealing with tiredness you might type the following related terms into the search box: fatigue overwork tired exhausted exhaustion sleep. Most search engines treat search words as potential parts of words. So in the above example 'fatigue' could also find web sites containing the word 'fatigued', and 'sleep' will find 'sleepy', 'sleepless' and 'sleeping pill'. But while 'exhaust' might have found 'exhausted' and 'exhaustion' it has been avoided so as not to pull out articles on car engines and pollution! If you find you've got too many articles, you can often make the search more specific by adding words you want to see (eg: 'overwork') or conversely specifying terms that you *don't* want to see (eg: 'Seattle'). Often you do this by using symbols (+ and -) or logical terms (AND and NOT). Check the rules for the search engine you're using. Some require wild-cards such as * to stand for missing letters. And many of them allow increasingly sophisticated ways of refining your search, by suggesting useful key words or popular web pages. Note: you may not get access to the hint pages (or even be able to use all the available refinements) if you're not accessing directly via the search engine's home page. In addition to using the search engines' own help pages, you can find a brisk and useful guide to the top search engines and how they work from the Web Search Cheat Sheet <http://www.colosys.net/search/>. Some examples follow, but please be aware that things move fast on the Net. The link that was found today may not be there tomorrow - or may not work in the same way! Richard Broke described such a search for information on the Spanish Civil War: "There are remarkable amounts of free information on the Internet. I looked up the 'Spanish Civil War' in Lycos <http://www.lycos.com> (in my view the best search engine on the Net). It came up with (inter alia): "http://www.civeng.carleton.ca/SiSpain/history/civil.html http://tigerden.com/~berios/spunk/Spunk336.html http://press-gopher.uchicago.edu:70/CGI/cgi-bin/hfs.cgi/99/beacon/88043317.ctl http://www.nypl.org/research/chss/subguides/milhist/home.html http://www.nypl.org/admin/../research/chss/subguides/milhist/home.html http://www.anatomy.su.oz.au/danny/book-reviews/h/The_Last_Mile_to_Huesca.html " Laurence A.Moore started with Yahoo: "First, I went to <http://www.yahoo.com> and ran a search for Spanish Civil War "Then I went down to the bottom of Yahoo's home page where there are several other search-engine links, and ran a search with Lycos. "Those two searches pulled several interesting items, and each had links to other sites. The most interesting site from this quick search was http://www.civeng.carleton.ca/SiSpain/history/civil.html "If I'd really been doing research on the Spanish Civil War, I would have followed the links, and also used the other search engines." However, TJ had mixed feelings: "I find that using a keyword or Yahoo gets me much more than I wanted. "For some reason, I feel as if all I have to do is type in a subject and I'll find everything referenced on that one subject. Doesn't happen that way, does it?" As I said, each search site has its particular strengths. Nick Tompkins writes to tell us about Google <http://www.google.com>. "I am currently training employees at a major TV company, so I have been showing them a few search engines - running searches for the same subject matter to see how much relevant material is found. Google came out tops and a producer found his own name on a site selling videos of a film he made a few years ago. He went straight to the legal dept to see if they owed him any money....." Look in particular for the "search within results" link at the foot of Google's results page, which gives you a chance to narrow down your search, if your first search pulled out too many sites, or too many that weren't relevant. Pat Marcello adds, "I also like the fact that Google achives web pages, so if a page is offline, I can still get to the information." Mike Casswell writes: "The best feature of all, in Alta Vista, is the Advanced Query Page, which is a different page (linked from the Simple Query page). This has a number of clever search tools. I often use 'near' which is both simple and powerful. There is also a help page for the Advanced Query syntax." If you want to find a lot of search facilities in one place, Ellie Kuykendall says, "I very much like http://www.beaucoup.com. There are over 600 search engines on one page...everything from government to email addresses. Takes a bit of time to load, but I use it all the time." 4.2.2 META-AND MULTI-SEARCH ENGINES Alvaro Ramirez recommends the Meta Crawler: "It may not be very accurate, but fast... it sure is!" <http://www.metacrawler.com/> Such "meta" search engines use various techniques to search across a number of engines at once. They can often be customised for different types of search allowing you to select which search engines you want to use, and many offer a number of specialist categories, including many databases that are not covered by normal search engines. One excellent meta-search engine is Profusion <http://www.profusion.com/>. Profusion combines search results in a single list (avoiding duplication). It's fast and easy to use, and can also check that the links are still live. Often confused with meta-search engines are "multi-" search engines. I bet you're confused already. Essentially they do the same job, sending your search terms to a number of search engines at once, but they don't try to combine them. Instead they display the results from each search engine in separate windows. Two excellent multi-search engines are Search Spaniel <http://www.searchspaniel.com/> and theinfo.com <http://www.theinfo.com>. When you enter your search terms, they open a new window for each of the search sites individually. Beware, this can be a bit overwhelming if you've selected all possible sites! They also make no attempt to combine search results, so you have to be prepared for a fair bit of sifting, but that can be an advantage in some cases, as different search engines rate sites in very different ways. So this approach is useful for those more difficult searches, where your search terms may be less easy to narrow down. For example, I tried to use theinfo.com to find a site relating to the feature film "Go". Now being both a verb and a game the word "Go" is likely to appear on a million pages, even capitalised, so I wasn't surprised to find zilch on the first attempt. I closed the myriad new search windows that theinfo.com had opened (and selected a few less sites!) and tried "Go AND cinema". Some metasearch engines have difficulties with search terms that use expressions such as AND, OR, +, - etc. In this case, some of the sites used by theinfo.com still came up with nothing useful, but others put the film "Go" at the very top. Note: If I'd looked carefully enough I'd also have noticed they have a set of specialist search sites, including a category for "Movies". There are ten specialist categories - good, but not as wide-ranging as Search Spaniel which offers 23 as well as a "personalised" list (not that I've ever managed to get that feature to work!). Theinfo.com doesn't pretend to be the only search engine you'll ever use. Like any search source, you need to have the right kind of query. There is even a page on the site that explains when it's useful to use theinfo.com as opposed to other sites, an excellent feature I wish more search engines would adopt: <http://www.theinfo.com/about/whyuse.html> Most of us tend to be lazy and stick with just a couple of search engines that we are used to. One bonus that comes with using a multi-search engine is that you get a chance to see new engines you may never have seen before, and to catch up on the latest advances of those you may not have touched for months (or even years!) Thus it can be a quick way to test the strengths and weaknesses of different search engines against each other. Thus both Multi- and Meta- search engines can be extremely useful, but be aware that they don't always provide all the facilities of the original search engines. For example, searching recently on the title of my own new project - "Paradise Grove" - I found our own film website on Alta Vista but not on Profusion (which uses Alta Vista) because Profusion treated the capital letters differently! However, these are minor drawbacks. On the whole, the good meta/multi-search engines are well worth using. Find a list of them at 9.3.4. But the Internet offers far more resources than just search engines... 4.2.3 USENET NEWSGROUPS can be extremely useful for asking specific questions. The World Wide Web is by no means all there is to the Internet. Indeed there are other areas which can be just as useful for finding information, and perhaps the most important of these is Usenet. There are currently 24,000+ newsgroups, covering just about every subject you can imagine, and a few you probably can't! Beth Porter says: "Post messages in classy newsgroups and fora. Can be dodgy, but it's paid off for me quite a few times." And Yvonne Hewett: "I use the Net for research by the simplest method possible: going into the list of Newsgroups and searching it for the topic I'm interested in, and then posting to the group. "I've found that the Net is like most places where there are people with expertise: if I approach them properly and ask intelligent questions, the answers are usually forthcoming. If answers aren't, I often get pointers to people who are in the know. And like any other research, it takes time and patience to work through the masses of non-indexed information." You can subscribe to an appropriate newsgroup using specialised software, or you can search one or more newsgroups using either AltaVista (selecting "search Usenet") or Google: <http://www.altavista.com/> <http://www.google.com> If you don't know which newsgroup(s) will be best for your subject, then try putting an appropriate query into Google or AltaVista and noting what newsgroups come up. Almost all newsgroups have a FAQ (Frequently Asked Questions list) which can be a mine of useful information, or at least tell you if you're in the right place. FAQs can be searched for at Infinite Ink <http://www.ii.com/internet/faqs/> or downloaded from ftp sites ftp://ftp.uu.net/usenet/news.answers and ftp://rtfm.mit.edu/pub/faqs. Writers Sal Towse and Marty Fouts used DejaNews and Alta Vista in quite a sophisticated way to search a very specific question: what style of calligraphy was used to write the journal entries in Myst and Riven (they are computer games). [Note - since this was written DejaNews has been taken over by Google] Sal went to Deja News' interest finder and entered "fonts". Result: 99% chance in the Newsgroup comp.fonts (talks about "fonts"). Sal's comment: <duh!> She then conducted a Deja News quick search on "riven" restricting it to comp.fonts. Result: one hit, a posting with the subject: "Riven Font" which read, "If you're looking for the unusual font used for the handwritten-looking text in many of the documents in Riven as well as in the in-store displays, you can find it online at <http://www.ragnarokpress.com/scriptorium>. Look around for the Fiorenza font -- there's even a shareware version." Sal: <Yowzaaa! Is Deja News cool or what?> Marty added the following rider, though: Deja News is cool, and finding that the _font_ name is Fiorenza is probably the answer to the intended question. However, for those interested in calligraphy and typography, the question was about "style of calligraphy," which requires a slightly different answer. "Fiorenza" is the name of a typeface designed to give a calligraphic appearance. Most typefaces are named by or after their designer, so this one appears to be named by someone interested in Florence, hinting that the calligraphic style _might_ be Florentine. As Sal has pointed out, quite a few sites have the Fiorenza font. A quick check with Alta Vista for "+Fiorenza +calligraphy" turns up http://www.cgrl.cs.mcgill.ca/~luc/fontlinks.html where we are told that ...the Renaissance calligraphy shown in Fiorenza,... which, along with firing up my copy of Riven and comparing the first book's calligraphy to a calligraphy sample book suggests to me that an answer to the original question _might_ be: Riven uses a computer font called "Fiorenza" that is available from several sites, including one found by Sal Towse: http://www.ragnarokpress.com/scriptorium from the comment mentioned above and a visual inspection of the font, I would _guess_ that Fiorenza is based on an Italian monastic calligraphic style that is full serif with a 40 degree nib angle(? - I didn't measure,) rounded uncils, and concave downstrokes. If you want to know in more detail, you should check a good art supply store, book store or your local library for books that will contain samples of calligraphy from the time of the Renaissance. You might also look into typography books, one of which may give you the history of "Fiorenza," if it is based on an existing typeface, rather than being generated entirely for computer use. Other possibilities: do a more detailed inspection of the results of the Alta Vista search. Post the question in comp.fonts. Check to see if there is a calligraphy related newsgroup, and if so, post there. The site I found has quite a few links, including one to "The Society for Italic Handwriting" which may be a good source of info. I've included this at length because it's an instructive illustration of the many ways there are many ways of skinning a cyber-cat, and of the importance of asking the right question in order to get the right answer, and also that sometimes answers found on the Net will send you to different parts of the Net or even resources outside the Net altogether. IMPORTANT NOTE: some newsgroups don't take well to being "used" by strangers. If you're thinking of posting a question, lurk for a little while first, to check out the prevailing mood and make sure your question hasn't already been answered in the group's own FAQ. Avoid assuming that other users are only there to provide you with free answers to your questions. And do make sure that your subject line is a useful guide to what you're asking: "Information wanted" is not so effective as "Who won the Battle of Bosworth Field?" 4.2.4 MAILING LISTS can be helpful in similar ways. For those who haven't met mailing lists yet, they are the equivalent to newsgroups, but you receive all the postings (or a digest of them) by email. There are even more mailing lists than Usenet groups, and some are very highly specialised indeed. Excellent places to start searching for appropriate mailing lists are Topica <http://www.topica.com/> which also allows you to read the lists, messages and discussions on-line, Liszt <http://www.listz.com/> and Windweaver Web Resources <http://www.windweaver.com/searchpage2.htm#Discussion>. For more resources check 9.2 below. Alternatively, you can obtain a list of mailing lists by sending an email with the single word HELP in the body of the email, to mail-server@sri.com or mail-server@rtfm.mit.edu. 4.3 HOW CAN I FIND MORE GENERAL BACKGROUND INFORMATION? ---------------------------------------------------------------------------------------------------- As the searches become wider and less specific, the Internet becomes more tricky to use. 4.3.1 NEWSGROUPS and MAILING LISTS remain useful nonetheless. Either for posting specific queries, obtaining FAQs, or just lurking and seeing what ideas crop up. Jane Dorner: "Quite honestly the best thing is to join a newsgroup dedicated to the research subject in hand and trawl that until you find what you're looking for." 4.3.2 GOPHER, ETC. Few writers mentioned Archie, Veronica, Gopher, WAIS or any other use of FTP or Gopher- space. However, before I go much further I should say that this could be for a good reason. As Steve Hunt writes, "For all practical purposes they are dead. "I think all the Veronica and Archie servers are down, and the only gopher I know of that is still running is the one at University of Minnesota, probably because they created the gopher protocol: <gopher://gopher.tc.umn.edu/>" I suspect that this too has now gone the way of all other gophers, but if do you find one that works, or are just interested in ancient history, this is how it used to work in the early days: Marnie Froberg researched police corruption using Archie (for ftp file site searching) and Veronica (for searching worldwide gophers) and WAIS based search engines (einet.galaxy and harvest both of which run WAIS in the background). "AmyWriter" found great success with Gopher: "Through Gopher, I've downloaded some great files that go beyond what the encylopedia has, e.g. for Haiti, I got info on all the political stuff that is happening NOW from news articles, white papers, etc. This is info that would be dated in the encyclopedia. "Basically, I go into Gopher and type in, "Jamaica," for example. This brings up a list of reports on many topics of interest which I scan and select and then print out. For example, there might be a college professor's report on current Jamaican politics." Gopherspace could give very quick and informative answers to queries, but gradually disappeared as more organisations have moved over to the Web. However, gophers covered a large number of databases that were not on the Web and which contained a wealth of information and texts. You access gopher space using dedicated gopher software or from a Web browser by typing the gopher's address (it starts with gopher:// instead of http://) usually followed by a port number (typically 70) as in <gopher://gopher.ic.ac.uk/70>. Gopher search engines are called Veronica or Jughead. Veronica was the more recent. A typical Veronica search (on the single word "uranium") brought 146 items within seconds. Some of these (again typically) were out-of-date or the connections didn't work. The rest gave me everything I might have wanted to know about uranium, from its elemental properties to the latest uranium mining figures from various parts of the world. Gophers were generally run by universities and government departments, so seemed to be best for academic and governmental type searches, although there was some quite non-academic stuff there as well. 4.3.3 WEB DIRECTORIES may be better for vaguer background research than search engines because they allow you to follow through a line of thought on a root and branch principle. Try clicking on the appropriate "branch" of Yahoo (for example) and then narrowing down: <http://www.yahoo.com>. However, Yahoo can look rather limited at times, with a relatively small database. To an extent this is a problem with all directories, which can't be compiled as automatically as search engines, and so tend to be smaller and less up-to-date. Galaxy <http://galaxy.einet.net/galaxy.html> is much more clearly laid out than Yahoo - easier to see where the different sub-headings are, and seems generally a better choice at the moment. It should be noted, though, that the distinction between search engines and directories is becoming increasingly blurred. Many directories now have a "web search" option, while a number of search engines, offer directory-like services - check out Google <http://www.google.com> and Alta Vista <http://altavista.com/>) 4.3.4 VALUE-ADDED GUIDES are often more helpful than directories. They offer fewer links, but pre-select them to filter out the dross. In addition, their descriptions are generally more detailed than ordinary directories and search engines. Clearinghouse, for example, <http://www.clearinghouse.net/> provides topical guides to the Internet. They aren't as comprehensive but provide value-added descriptive and evaluative information ideal for researchers." Those Encyclopaedia Britannica people also run an excellent value-added combined search engine and directory <http://www.britannica.com/>. It's actually quite good. It offers to give only good quality sites, but if it can't find any in its own database it can be set to search Alta Vista, so you should always find something. 4.3.5 SEARCH ENGINES, however, appear to grow less useful as the query becomes vaguer. Much of the problem lies in knowing how to phrase the key words. On wider background searches these can become confusingly all-encompassing. 4.3.6 WEBRINGS WebRings are an interesting new development which could be useful for general browsing. One of the most difficult things to duplicate on the Net is the ability to browse around a subject, slowly but thoroughly building up a solid base of knowledge. Somehow it's a great deal easier in a physical library where you can find lengthy books on specific topics. Web pages have a tendency to be lighter in content than most books, and following links can be a remarkably hit and miss affair. With WebRings, groups of sites on a topic are linked together so that you can move easily through the sites, forwards and backwards or even at random. In theory, a WebRing should also provide a certain guarantee of quality. You can search for WebRings at <http://www.webring.org/>. Worth checking out to see if there's something on your chosen subject. 4.3.7 WEBLOGS A new development coming over the Internet horizon is the weblog, or Blog to its fans. Blogs vary enormously, but essentially they are a blend of on-line newsletter and discussion group - sites where individuals and/or subscribers can post news stories, links, discussion points or the detailed minutiae of their life. Some are riveting, some are less so, and the range of interests is idiosyncratic. The form is still in its infancy as far as providing useful research material is concerned, but the infant shows promise. To poke around among the bloggers try looking at the Blogs listed at: EATONWEB BLOG PORTAL <http://www.eatonweb.com/portal/> BLOGGER <http://www.blogger.com> Look for the option to list by categories, or search on a particular search term. In addition, should you feel the urge to set up your own Blog, the last two sites listed above will help you do it for free. 4.3.8 PROXIMITY SEARCHING A more unusual tool for research comes from NameBase <http://www.namebase.org/>. In addition to a database of useful articles in a number of fields, particularly social, political and commercial, you can perform a "proximity search". Search on a name, and their database creates a "network diagram", linking a wide range of related names, grouped according to how close or frequent the link is found to be. Easier to use than to describe, try it out. 4.3.9 THE "DEEP" OR "INVISIBLE" WEB When conducting a search on the Net, the main search engines can be very good - but be aware that there is a great deal of the Web that they are totally unable to search. This has been called the "invisible web" or the "deep web". It exists for a number of reasons. Most important is that many excellent databases cannot be searched by the search engines' automatic software ("web spiders"), either because the spiders cannot access the databases, because otherwise ordinary webpages are constructed in ways that interfere with the workings of the spiders or for other technical reasons. Frames and dynamic pages interfere with the way that the web spiders access information and return useful addresses for the search engines to use. And the text content of images or Adobe pdf files cannot be examined. In addition, some databases simply won't work with spiders or refuse access. For example, you can't search on a phone number in Alta Vista and get an answer from Anywho.com. Financial databases, newspaper archives, government information, almost every kind of resource or database is affected by the "invisible web" problem. If you rely solely on search engines, you are limiting your resources to a tiny fraction of what is out there. The best way to deal with this is to have a good supply of databases which are specific to your subject. These will lead you faster and more surely to the information you need than any search engine, which will on current estimates only cover 1/500th of the 500 billion pages now on the Web. Many of the links in "URLS FOR A RAINY DAY" (FAQ part 2) will return information that search engines cannot find. Other useful strategies include searching Usenet and other discussion groups, posting queries on discussion groups, and using expert resources (see sections 4.2.3 and 10.2.) There are a slowly increasing number of sites for researching the invisible web. Two of the best include: - INVISIBLEWEB.COM <http://www.invisibleweb.com> - developed by Intelliseek - COMPLETE PLANET <www.completeplanet.com> - 20,000 approx invisible web databases Further information on the invisible web (and links) can also be found at these two sites: <http://www.searchenginewatch.com/sereport/00/08-deepweb.html> <http://websearch.about.com/internet/websearch/library/searchwiz/bl_invisibleweb_apra.htm> Thanks to Steve Hunt for pointing out the invisible web issue. ********************************************************************** 5 HOW CAN I FIND INFORMATION FASTER? 5.1 First, there's the obvious: get a faster modem, or an extra-fast connection like an ISDN line, ADSL or cable-modem connection. Or upgrade your processor, RAM and video RAM. However, these cost money, and you're still at the mercy of a slow connection somewhere the other side of the world. 5.2 If you don't need pictures, then set your browser to load web-pages without them. Unfortunately, there are still some sites which are virtually unusable when displaying text only. Luckily, not many. 5.3 Less obvious is the question of efficiency. The Net is so large that it takes time to get to know any one subject area - to suss out some databases you can trust, assess which sources are best for which kinds of information. You can make on-line life easier for yourself if you focus on relatively small subject areas for relatively long periods of time. It's more difficult if your work or inclinations lead you to research civil engineering one day, single parents the next... 5.4 BOOKS AND BOOKMARKS Ultimately, you can't beat a good set of URLs in a well-maintained (and backed-up(!)) bookmark list. Some of the best URLS come from experience. Others can be culled from books, newspapers and magazines. Beth Porter: "Get hold of Computer Life's Road Map to the http://www, which is sweetly laid out in category globules [Sports, Media & Entertainment, Politics]; there's also the Internet White Pages, published by IDG Books [Godin & McBride] ... more URL's than you've had hot dinners." 5.5 Some popular sites now have one or more "mirror sites" in other parts of the world, carrying the identical information. If accessing one of these, choose the mirror site in a time zone which is likely to be least in demand, eg: between 11pm and 7am local time. 5.6 Consider going on-line at more expensive times of day (if you have to pay for phone calls) or using Internet providers with better bandwidth and modem/user ratios. The extra cost may well be outweighed by the greater efficiency and faster access times. Talking of money.... ******************************************************************************** 6. SHOULD I PAY FOR INFORMATION? 6.1 OUTERNET. Richard Broke: "One of the problems of the Internet is that it is free! So, basically, you get what you (don't) pay for - much of the time. The Outernet is the name given to pure knowledge databases which are SUBSCRIPTION ONLY. Probably the biggest is called Dialog (sometimes aka Knowledge Index). "Because they are selling data, these outfits are reliable (by which I mean accurate) and up- to-date." However this probably is mainly of use to those whose work can justify the expense. Dialog begins with an annual sub of 30 UK pounds (or equivalent). However, to that you must add on-line charges which depend on where you live and which database you access. Some databases charge $12 per hour, while others go as high as $225. Then there's charges for displaying documents (say 60c per document), extra charges if you print stuff out, connection charges if you're outside the US... Web: <http://www.dialog.com/> or phone: UK: 0171-930 5503 USA: 800-334-2564 or 415-254 7000 I have phone and fax numbers for other European countries if wanted. 6.2 HIRING RESEARCHERS There are other ways of paying too. Alex: "As with traditional research, you may find it pays to hire someone to find the information for you." Alex gave details of Mindsource....an organisation that finds information for people. Costs start as low as $50/Quarter, but you get more the more you pay. At the moment Mindsource is probably not as useful as it could be - but that may change. For details of Mindsource: send blank email to mindsource@memo.net. 6.3 INTERNET SUBSCRIPTION SERVICES There are subscription on-line services on the Internet itself, eg: Microsoft Encarta. Concise definitions are free, the full encyclopedia payable <http://www.encarta.msn.com/>. For variety and topicality, check out the Electric Library <http://www.elibrary.com>. Electric Library allows you to use "plain English" searches across a wide range of newspapers, magazines, TV and radio transcripts, dictionaries, encyclopedias, maps, photographs, and literary and artistic classic works. When I've used it, it's been fast and efficient at pulling out relevant information. You can also download special software (Windows and Mac) to search the library without needing an Internet browser. (Note: Electric Library goes offline each day from 04:30 to 6:30 EST for uploading new material. Depending on where you are in the world, and what times you like to work, this down-time may or may not be an inconvenience. For example in Britain, this is 09:30-11:30). Irritatingly the site doesn't display the current subscription rate (it used to be $9.95 per month, unlimited use - $59.95 p.a.) but there's a seven day free trial. 6.4 DOWNLOADABLE DATABASES Finally, you can download whole databases to keep on your hard drive, for a fee. One example is NameBase - an index of individuals, corporations and groups compiled from 600 investigative books published since 1962, and thousands of pages from periodicals since 1973 - covering the international intelligence community, political elites Right and Left, assassinations, scandals, Latin America, big business, and organized crime. Download a 10-day free trial of NameBase's entire index from <http://www.namebase.org>. ********************************************************** 7. WHERE CAN I GET FURTHER HELP? -------------------------------------------------- 7.1 ALCS has a dedicated writers' server, put together by Jane Dorner and Chris Barlas. <http://www.alcs.co.uk> Chris writes: "One of the features is a writers' information directory, a series of hyperlinks that writers have found useful for research purposes." Society of Authors have also developed a site <http://www.writers.org.uk/society/> put together by Storm Dunlop. As has the Writers Guild of America <http://www.wga.org/>. (Note: the Society of Authors has temporary server problems, some users may find problems for a few weeks). 7.2 WRITERS' PERSONAL RESOURCE PAGES There are a number of personal writers' pages on the Net, offering useful links. Some are listed below (9.8). 7.3 SCRNWRIT MAILING LIST Marty Norden tells of the screenwriters list called SCRNWRIT: "There are plenty of folks there who might be able to direct you to the right sources. If you'd like to join, just send the one-line message "Subscribe SCRNWRIT" to Listserv@tamvm1.tamu.edu. Be aware, however, that SCRNWRIT is an unmoderated and *very* active list. You'll easily receive 50- 100 messages *per day* from it, sometimes more." 7.4 RESEARCH BY REAL PEOPLE A growing number of free resources offer searching by soft, warm-blooded sentient beings, rather than computers. ProfNet <http://www.profnet.com> is a collaborative of 3900 public information officers (PIOs) linked by Internet to give journalists and authors convenient access to expert sources. There are a number of ways of submitting queries, not restricted to the Net itself: Phone (from US): 1-800-PROFNET (1-800-776-3638) Phone (from outside US): 01-516-941-3736 Fax (from US and Canada): 1-516-689-1425 Fax (from beyond US and Canada): 1-516-689-1425 Email: profnet@profnet.com CompuServe: 73163,1362 Ask An Expert <http://www.askanexpert.com/> offers similar expert facilities. KnowPost <http://www.knowpost.com/> is run by people from all over the world with the aim of helping "guide those who are lost on the Information Superhighway". The service is free, but you have to answer a question for each question you ask. A clever new variation on human-based research resource, SourceNet is a private research tool that lets journalists post anonymous queries on any topic that will be distributed daily to nearly 10,000 corporate and agency PR professionals. Journalists worldwide use the service to round out stories, find guests, test new story ideas, or find expert sources for stories in progress. SourceNet say, "It's like having an army of research assistants helping you, for just a minute or two of effort on your part! All queries are completely anonymous; personal contact information is never available to PR people through SourceNet queries." Free to all working members of the media: <http://www.mediamap.com/media> 7.5 INTERNET BY EMAIL If you can't get at all these resources in the normal way, you can get just about anything you need by email. Loyd Colston writes: By sending email to mail-server@rtfm.mit.edu with text in the BODY of send usenet/news.answers/internet-services/access-via-email one will get a file on how to do archie, FTP, http://www, WAIS, etc. by email only. In other words, you can FTP a file from an email only account. This file is also useful in learning how to use internet in general. The author also publishes the e-zine The Internet Tourbus which is a free tourguide of the Internet. Bob Appleton can also supply updated files which show in specific detail how to get just about anything by email. The files are free and everything mentioned in them is also free. To get these files individually or together in zipped format as well as in text format, send a message to: agora@dna.affrc.go.jp and in the body of the message put: send ftp://ftp.crl.com/users/iv/iverham/XXX Where XXX stands for one of the files listed below. email4u.txt getit4u.txt fun4u.txt pix4u.txt email4u.zip getit4u.zip fun4u.zip pix4u.zip 4useries.zip Repeat the line for each additional file requested. In addition the .txt (but not the .zip) files are available here: send http://members.aol.com/bombagirl/freeware/XXX send http://www.wireworm.com/4useries/XXX Another way is to send a blank message to: 4useries@wireworm.com. When you get the email information, please give attention to Email by news groups. Vigilant and InReference both allow for keyword searches of USENET being sent to your mail box. This saves a LOT of time having to read every article about items. Both allow filtering so you can specify exactly what you want. 7.6 Information Research FAQ For those who want to go into more academic detail about research - online and otherwise - I can recommend the massive, if slightly sprawling, Information Research FAQ <http://cn.net.au/> - not to be confused with our own production. Articles on a wide range of topics related to finding information fast. 7.7 Windweaver Web Resources An excellent site well worth checking out for help in research is Windweaver Web Resources <http://www.windweaver.com/> with over 100 pages of useful Internet search guides and links. Created by an Internet trainer who specialises in research resources, it offers useful comparisons of the strengths and weaknesses of the different search engines, etc, along with guides, help pages, and many, many links. Highly recommended. 7.8 Articles Sal Towse, contributor to the FAQ, has a good article on web searching in "Computer Bits". Well worth a read: <http://www.computerbits.com/archive/1998/0700/web_searches.html> as is the computerbits site in general. 7.9 Language Problems? If the material you find is not in your own language, don't despair. There are even sites that will translate individual words, or better whole chunks of text or web-pages between different languages. Don't expect perfection, or the most obscure dialects, though. As a starter, try Babelfish, from Alta Vista <http://babelfish.altavista.digital.com/> which covers French, German, Portuguese, Italian, Spanish or English. ********************************************************* 8. HOW CAN I VALIDATE WHAT I FIND? 8.1 HOW RELIABLE IS THE NET? Things are clearly changing all the time. The Internet is growing bigger - and as many have discovered a web page or Usenet posting can look the same whether created by an world authority or a student. The key issues are * ACCURACY - Most traditional media have standards of fact-checking, which need not be followed by the creator of a web page. The same applies to discussion groups. In misc.writing, for example, a writer accused 90% of the advice posted about copyright law of being wrong. However, we shouldn't overstate the case. Mistakes also occur in venerable legal textbooks. The problem is that we grow up learning to judge the validity of traditional media. Often this comes from the context in which it appears - we value information in a medical journal, for example, over a teen magazine. On the Net, that context is often missing or severely limited. * AUTHENTICITY - How authentic is a website or posting? Many health sites, for example, are created by drugs companies, but don't reveal that fact on the site itself. There are important-sounding history sites which are run by right-wing extremists. Even the domain name is no proof of authenticity, as constant legal wrangles continue to prove. Although the Net is subject to laws of libel, misrepresentation and advertising standards codes (despite suggestions to the contrary) these laws are not always easy to enforce, and take time. * AGEING AND FLUIDITY - Books and magazines have the grace to look their age. There's no dust on a website that hasn't been updated for over two years. Many sites don't even show a date. Or you may find the reverse problem. The web is so fluid that the solid site you relied on for information on a regular basis may simply disappear overnight. And every search engine is filled with out-of-date links to sites that no longer exist. * ACCESSIBILITY - The information you want may exist, but may be buried under a load of dross. Top of my personal hate-list are sites that offer to rig search engines so that your personal URL appears "near the top of any lists." Using techniques which fool search engines into thinking a page is more relevant than it is, they turn the usefulness of the web on its head. 8.2 WHAT CAN I DO ABOUT IT? 8.2.1 DOUBLE-CHECK Journalists always double-check information unless it comes from a totally secure source. The same should apply for any information you need to verify from the Net. Either find another reliable Internet source, or use traditional means - books, telephone, etc. 8.2.2 LOOK FOR "BRANDED" SITES "Branded" sites from organisations you know and trust are likely to be among the more reliable - although even they should still be treated with care, and you should not take the domain name as proof on its own. Many authentic-sounding domain names have been bought up by others. Government sites generally provide reliable statistics, as reliable as government statistics ever are! Electronic versions of publications such as established broadcasters, newspapers and specialised journals are likely to have been prepared as carefully as their traditional counterparts. Academic departments of universities can be good, too, but check to see if you're reading the work of a professor or student... 8.2.3 USE "FILTERED" DIRECTORIES Search engines use automatic web-crawling "spiders" to trawl for pages, and generally make no attempt to judge the value of the sites they find. However, the better search engines have developed a range of strategies for excluding sites that try unfairly to raise their profile, with a variable degree of success. By contrast, directories are selected by humans, giving a greater reassurance that the sites will at least be relevant to your query. Better still are value-added guides such as Britannica <http://www.britannica.com/> and About <http://www.about.com/> and in particular directories which give star ratings to valued sites such as Clearinghouse <http://www.clearinghouse.net/>. Most promising of all is the growth of specialised high-quality subject-based services, if there's one that covers your area. ELib <http://www.ukoln.ac.uk/services/elib/> is dedicated to developing such subject-based review lists, including a medical resource at OMNI <http://omni.ac.uk/>. Another medical resource can be found at the appropriately named Achoo <http://www.achoo.com/>. 8.2.4 LOOK AT INTERNAL EVIDENCE In all cases, you should check websites for internal evidence of quality: writing style, language, range of content, level of detail, clarity of design - all can give important clues as to the expertise of the provider. Writers with depth of knowledge and experience tend to be precise, rather than vague in their use of language, and will normally include detailed material and evidence to back up what they say. And while slickness is no guarantee of quality, a badly organised site suggests that the content may be sloppy in other ways too. In addition, look for references and citations, clear identification of who "owns" the site, the date of the last update, contact information and e-mail links. Lack of any or all of these should make you increasingly suspicious of the validity of what you find. 8.2.5 ASK WHO'S PAYING Good content takes time and effort, and while some people are happy to provide this for nothing, there's no incentive quite like hard cash to ensure the website is kept accurate and up-to-date. In some cases, it may be that the site relies on advertising, and therefore has a built-in incentive to keep visitors happy and supplied with good content. However, in others the paymaster may be a drugs company, a political organisation or an individual with an axe to grind. In the long run, if you can afford it, you may feel it's safer to pay yourself, and use one of the subscription-based services. However, even if you do pay, all the above considerations still apply. 8.3 HOW SHOULD INTERNET SOURCES BE CITED? Standards for citing the Internet resources are not yet established. However, you can find some useful suggestions at A Brief Citation Guide for Internet Sources <http://www.h-net.msu.edu/~africa/citation.html>. ********************************************************* 9. WHAT ABOUT THE FUTURE? 9.1 PLUS CA CHANGE... Clearly, the Internet will change - commercially, technically, philosophically. There'll be a need for services to pay for themselves, and that may mean more subscriptions, as discussed above. On the other hand, there are other ways of skinning a cat. Many sites will pay for themselves by advertising, while sites set up as corporate PR will find they need to offer more than pretty pictures to attract the browsers. There are also publication spin-offs, which need not necessarily be financially damaging to the provider, even if free. Times Educational Supplement found that their free Internet site actually led to increased sales of the printed publication rather than the decreased sales that might have been expected. When it comes to reliability of information, new techniques of encoding "watermarks" and using encrypction programs and digital signatures could be an important step towards ensuring that users trust the information they find. The energetic prosecution of legal safeguards will also be increasingly necessary - whether over trademarks and domain names or "passing off". And search engines will need both legal and technical ways to stop their search results from being rigged. It's perhaps as difficult to predict commercial developments as technical developments. The Net has an anarchic way of confounding predictions all the time. It wasn't so long since everyone was hyping "push" as the latest transformation. So far, the push revolution has been put on hold. 9.2 TECHNICAL DEVELOPMENTS ON THE NET? 9.2.1 INTELLIGENT AGENTS There's new software developing all the time. Intelligent Agents (IAs) are held to be the coming thing, according to some (the developers of Intelligent Agents, perhaps). IAs are not writers agents with nous, but computer programs that help with searching. There are a number of different types of IA, but basically they aim to make internet searching easier by (a) learning your preferences in an "intelligent" way and/or (b) going onto the net and searching while you are offline, thus saving time and money and/or (c) (theoretically) returning with a more targetted, useful list of hits - by avoiding duplication, eliminating dead links, and generally being more efficient than your average bog-standard search engine. When I posted a query about IAs, at first I received a grand total of one reply (from a reviewer) and zero replies from researchers and writers, from which I concluded that either IA's are not much use, or that no-one who uses them saw my message or (most probable) that researchers and writers are too busy researching and writing to take valuable time wrestling with unknown bits of software. Since then I've heard from Vic Justice who recommends "a search device called Web Ferret, which is a free download. He says: "It calls up 500 responses to the subject query and does it faster than, say Yahoo or others. To my computer illiterate mind, it searches the search engines. "A major advantage of Web Ferret is the contents index that pops up as the mouse cursor touches the item title, so that you only need call up items specific to your search. This saves time." <http://www.ferretsoft.com/> Bruce Krulwich who is professionally employed in bringing agents to market has also written a number of articles on the subject <http://www.geocities.com/ResearchTriangle/9430/> Meanwhile, for Mac owners only, Mike Shields tells us about Sherlock, 'another method of research, for those of us using a Mac with OS 8.5. Sherlock, performs the same feats as Metacrawler. Allows fuzzy searches as well. 'What makes this really powerful, is that you can create a Sherlock plugin for your site, so that your site will be included in the searches. 'You only search the plugins that you yourself load. So, for instance, if Charles Deemer creates a plugin for his site, and you check it off, then you search his site, if you've loaded his plugin before hand. You can also choose which of the search engines to utilize. 'It's a very fuzzy search. I can ask a question like, "Who is Charlie Harris?" And set it off and running, and it should come back with a few interesting things.' More information from <http://www.apple.com/sherlock/> To find out more on all kinds of IAs, you can also try: Software: <http://www.autonomy.com/> Autonomy <http://www.kryltech.com/> SSSpider <http://www.agentsoft.com> AgentSoft <http://www.alexa.com/index.html> Alexa Internet <http://www.botspot.com> BotSpot - try "Best of the Bots" Reviews and Discusion: <http://www.cs.umbc.edu/agents/>UMBC Agent Web <http://www.research.ibm.com/iagents/> IBM Intelligent Agents Home Page <http://www.webtechniques.com/archives/1997/05/burner/> Crawling towards Eternity (Web Techniques, May 1997) <http://cws.internet.com/reviews/agents-reviews.html> Stroud's Internet Agent Reviews (Most of these links courtesy of Jen - thanks.) The jury is still most decidedly out. Please tell me if you've tried IAs and find them scintillatingly useful, totally useless, or somewhere between the two. 9.2.2 WATCHING THE NET To get the best out of your Internet searching, it's worth spending a little time keep track of developments. Directories and search engines are constantly looking for ways to improve their service. For example: Alta Vista <http://www.altavista.com> has been offering a translation service for some months, giving rough and ready translations into and out of the more common languages and has now started suggesting key words to help refine or clarify your search.. The Excite directory <http://nt.excite.com/> has added to its news searching. The NewsTracker option helps you search 300+ publications, with the ability to save search terms for re-use. HotBot <http://www.hotbot.com/> now offers the Direct Hit ranking system. Perform a search, and Direct Hit will offer (if appropriate) a list of the most popular pages relating to your subject. The usefulness of course can depend on your chosen search topic. Read more about it in Counting Clicks and Looking At Links <http://www.searchenginewatch.com/sereport/9808-clicks.html> and at <http://www.directhit.com/> One of the best ways of keeping track is the excellent "Webmaster's Guide to Search Engines" <http://searchenginewatch.com/>. This is really aimed at website developers, but is no less useful for Internet Researchers, as a way of learning about search engines and what makes them tick. In particular, it has a twice-monthly update news page "What's New". If you subscribe (for free) you can join the mailing list and receive regular news of updates by e-mail. ********************************************************* (Continued in part 2 - URLS for a Rainy Day - loads of useful links for research) (c) Charlie Harris 1996-2003 (This FAQ may be copied in whole or in part for non-profit making purposes only, provided adequate credit is given to those who helped towards it, and the home address is given <http://www.search-faq.com>. In addition, if anyone wishes to post any part anywhere on the Net, please inform us <charlie@harris.u-net.com> and commit to keeping the posting up-to-date on a monthly basis - there's too much out-of-date stuff on the Net already) --------------------------------------------- Charles Harris - Internet Search FAQ www.search-faq.com - charlie@harris.u-net.com ---------------------------------------------