Researcher beware: consuming information with a grain of salt
We've heard the warning before: buyer beware. Consumers are told to be on their toes because shoddy merchandise and dishonest sales practices abound. The same advice can hold for consumers of information. The world is full of inadvertently sloppily inaccurate information and misleadingly biased information. And yes, even deliberately fraudulent information.
This is the doubled-edged sword. We live in an information-rich world that people in just the recent past could not have imagined. Thanks to computers and the Internet, we've seen the greatest information revolution since Gutenberg invented the printing press. An American Library Association Presidential Committee a few years ago stated that no other change in American society has offered greater challenges than the emergence of the Information Age. Along with this wonderful explosion of knowledge, we have to deal with the inevitable fallout of junk information.
We've heard the expression of being "street smart" or "street savvy." Well, in a similar way, on the information superhighway as well as on the off-road, we have to be "information savvy." Another term we could use is "information literacy." Information literacy, as defined by the American Library Association, is the ability by individuals to "recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information."
Whether you are an academic researcher or more casual general consumer, there is no sure way to completely protect yourself from the dizzying array of junk fallout. But you can minimize the chances of falling prey to bad information by striving to be savvy and discerning. Being aware and cognizant of the how and why of faulty information is a big first step in achieving savviness.
First, we have to guard against blind naivete and easy submission to myths and misconceptions about knowledge and information. The first rule is that nothing is sacrosanct and infallible. In days of yore (and even before then), it was assumed the earth was the center of the universe. But Copernicus and Galileo dared to question that well-established piece of information, even though it was passed down from very high sources. If we are easily prone to placing certain elements of academia on a pedestal, we are asking for trouble. As I will show, academia has been known to have some blunderers and outright charlatans.
Another misconception we have to guard against is that print resources are automatically superior to online sources of information. Wrong! In addition to the obvious disadvantage that print resources will never have the currency of electronic information, print resources can be outright shoddy. Don't fall into the trap of assuming that anything is automatically reliable. Keep that shaker of salt nearby at all times, in all places.
Print resources are superior in one resounding way: notwithstanding the explosion of the Web, there is still a much larger amount of information in print than online. And probably we'll never see everything in print to become available electronically.
One can argue that print sources might be on the whole more reliable than those online, but that doesn't mean you should cast a blind eye to the credibility of anything that emanates from the printing press. And we're not talking just about the tabloid press. Even textbooks and respected scholarly and medical journals have fallen prey to junky information.
Let me give you some sobering examples. First, textbooks. For some time it's been known that some school textbooks (written, of course, by teachers and academics) have been alarmingly shoddy in content.
Recently the Associated Press reported that a two-year study found that 12 of the most popular science textbooks used at middle schools across the nation were riddled with errors. Errors included maps showing the equator passing through the southern United States and a photo of singer Linda Ronstadt labeled as a silicon crystal. None of the 12 textbooks had an acceptable level of accuracy, according to the report's author, a physics professor at North Carolina State University. "These are terrible books," the good professor said. "These are basic errors. It's stuff that anyone who had taken a science class would be able to catch."
Even respected encyclopedias don't escape unsullied. In 2006 one of the top three science journals in the world, Nature, did a study of Encyclopedia Britannica. The inquiry showed that Britannica articles averaged about three errors per article. Interestingly, a free and openly-edited competitor that's fashionable to bash, Wikipedia, was almost as accurate (or inaccurate), with about four errors per article.
With the advent of computers and desktop publishing software, just like with the Internet, just about anyone can become a publisher. When I was self-publishing my book, I remember reading in one of the top books on the subject that anyone who publishes a book is automatically considered by the public to be an "expert." Wow, instant credibility. Self-published books ("independent publishers") are a fast-growing segment of the publishing industry. When you pick up an attractive, impressive-looking and sounding book to read, you might be surprised to find out it was produced by a high school dropout in his basement. Beware of those small publishers, and place your trust in the large publishers with esteemed reputations with eagle-eye editorial staffs who settle for nothing less than the highest quality. Uh-oh. Did you know the worst offender of shoddy science textbooks in the above-mentioned study was the highly-esteemed big-time publisher Prentice Hall?
And now let's turn to scholarly journals. The highest form of information is gotten from "primary sources," journals that report first-hand original research: studies penned by the researcher himself. It's better to read the diary of a famous or significant person than to pick up the information in a "secondary" source such as Time magazine that tells you what was in the diary. Likewise it's better to read the original research report in a peer-reviewed journal, rather than somewhere else. Peer-reviewed means that scholars on the editorial staff screen submitted articles and pick out those most deserving of dissemination.
Well, did you hear about the much-heralded publication of a multi-volume set of Hitler's lost diaries, pronounced authentic by an eminent British historian and expert on Hitler? It turned out the diaries were fake. We could also cite examples of respected psychologists, sociologists, and anthropologists, among others, who submitted studies that were later found to be phony. And then there's the relatively recent situation of studies published in JAMA (The Journal of the American Medical Association) and other journals that have been seriously questioned as being bogus because the research was sponsored and dishonestly manipulated by the greedy and devious drug manufacturers. Then, if things can't get topsy-turvy enough, the king of the tabloids, The National Enquirer, occasionally crows "I told you so" about stories they published that were at first discredited by the mainstream, only to be embraced by the major media a year or so later.
I'm not indicting anyone and I'm not trying to bash the academic community. I'm merely pointing out that nothing is sacred, and nothing is perfect. Nothing should automatically be taken as the unquestioned gospel truth. We should never place anyone on a pedestal or in some utopian ivory tower. We should be ready to question everything we come across we read and hear and see, whether we're consuming it as recreation or as scholars working on a paper. Like Copernicus and Galileo, we should be ready to be skeptical and, with documentation, be able to back it up. We should remember there are potentially two sides to every story. One study is not necessarily conclusive and can be contradicted by another. One resource is not necessarily infallible and can be challenged by another. Good scholarship entails corroboration from multiple sources. Experts can be wrong (and debates among experts is not uncommon, whether it's expert witnesses in a trial or via the written word). A researcher should perform his own "meta analysis" and reach a defensible conclusion. The researcher is, in some ways, like a criminal investigator, and at the same time being the judge and jury. We should heed President Reagan's advice, "Trust, but verify."
There's a mnemonic acronym I use to sum up the criteria one needs to use to judge information: AACCUSO. This acronym stands for Accuracy, Authority, Credibility, Currency, Usefulness, Scholarliness, Objectivity. It's a checklist approach to try to verify the trustworthiness of information. With minor variations, those are the basic criteria espoused by information professionals. This approach is not perfect and not infallible, but it certainly is better than mindlessly gobbling up information, better than not giving thought to what you are looking at. (For the weaknesses of the AACCUSO approach, read Marc Meola's article, "Chucking the Checklist: A Contextual Approach to Teaching Undergraduates Website Evaluation," published in Portal: Libraries and the Academy, July, 2004, vol. 4, iss. 3, p. 331). Usually these criteria are pointed out when dealing with websites, but they apply equally to print materials. We should be no less vigilant when we consume print information as when we consume online information, lest we allow ourselves to be lulled into traps of misinformation.
Yes, the Internet is known for its lack of controls, which is what makes it exciting and full of unexpected gems as well as being pesky and dangerous at the same time. Welcome to the WWW: the untamed Wild, Wild West of communications, aka the Wild, Wild Web. It's the notoriously slipshod collection of both the good, bad, and ugly.
Another W in the three Ws could stand for wacky. As in, your dog can have his own website. But that's not always so bad. Besides being tickled by the entertainment factor, it's reassuring in this perilous world that at least some sites will tell you up front that Rover is the webmaster. Or that all this wondrous information on a particular site is the work of an 14-year-old. Ironically, though, a site lauded by many as the best homework helper site around, was actually created by a 14-year-old.
But it gets sticky and tricky when somebody deliberately tries to pull the wool over your eyes. Sometimes the successful efforts are very elaborate and ingenious. Phony educational and government sites, for instance. Like the bogus IRS site that fished for people's social security numbers and other personal information. When I bought a .org domain name a few years ago, no one bothered to ask me if I truly was a nonprofit organization.
One of the best examples of sheer skulduggery that I can think of is a totally made-up report of far-fetched research studies regarding cats. The report is titled, "Feline Reactions to Bearded Men." It's very scholarly-looking, with an abstract, methodology, footnoted findings of prior investigators, data, notes, bibliography, and so on. The findings of the report are as follows: Cats do not like men with long beards, especially long dark beards; cats are indifferent to men with shorter beards; cats are confused and/or disturbed by men with beards that are incomplete and to a lesser degree by men whose beards have missing parts. To cap off the hoax, all the names that were footnoted and cited in the bibliography were those of celebrities, including actors and pop/rock music stars. For example, some of the entries in the bibliography were listed as "Schwartzenegger, A." "O'Connor, Sinead" "Ciccone, Madonna Louise" and "Seuss, Doctor." When a group of college students were shown the report, most of them believed it was legitimate. (If you want to see the site yourself, it's at www.improb.com/airchives/classical/cat/cat.html.)
The above shows that reasonably intelligent people can be taken in by outrageous and blatant chicanery. But, fortunately, websites created on such a deliberately deceptive scale are relatively few. Most websites are done with sincere intentions to impart truth and helpful information. That should be comforting. But don't let your guard down yet. This fact notwithstanding, here's something alarming and scary: a major study published by the The Journal of the American Medical Association (JAMA) concluded that only 63 percent of the Internet information on breast cancer was correct and complete, while for depression, the figure was even worse, 44 percent. So much for sincerity and good intentions.
If we extrapolate these JAMA findings, we are led to believe that, on the subject of medicine at least (and why not with other topics?), we may have about a 50-50 chance of finding a reliable website. Kind of like throwing darts, or what amounts to the luck of the draw.
But it doesn't have to be so haphazard. We can, to start with, exercise caution. But that alone isn't enough. Just plain caution can still get you killed walking through a dark alley or across an uncharted mine field. It's got to be intelligent caution. With an attitude of caution we can combine knowledge (savviness) that specifically arms us to know how to best search for online information and how to evaluate it when we find it.
Incidentally, recent surveys show that users of the Internet are becoming more skeptical about what they find online. In a 2003 study by the Center for Communications Policy at UCLA, only 53 percent of users said they believe most or all what they read on the Web, down from 58 percent from the previous year. The survey also showed that Americans using the Internet consider it at least as important as newspapers and books, and more important than television, radio and magazines.
Knowing how to best search for online information means, ideally, knowledge of the various search tools on the Internet and their characteristics and how to best use them. Once you've supposedly found the Web pages with the desired information, you should be able to evaluate them. Knowing how to evaluate information means understanding the sources of information there are on the Web, knowledge of the kinds of websites there are according to upper-level domains, and what criteria to use to judge websites and their information.
First, let's talk about the different sources of information out there on the Web, and understanding the types of websites according to kinds of domains. Sources and domains types are inextricably bound together.
Sources are the originators of the website or the information on the websitemost significantly, individuals, businesses, government agencies, educational institutions, nonprofits, trade associations, special interest groups, and the media.
At the same time, let's remember that with the sender, there is a receiver. For each type of producer, there is a corresponding potential consumer. The information consumer, or user, can be classified in various ways in terms of socioeconomics, demographics, and information motives: professional, scholarly, business, general consumer, male, female, young, old, rich, poor, geography, politics, information, entertainment, and so on. The interface of all these variables determines who uses what sites for what purpose.
Another part of the big picture is the type of site according to upper-level domain name. The upper-level domain names (such as .com, .org, .edu) are part of the URL, or Web address of each document found on the Internet. The domains in great part reflect the source and vice versa. By source, we are referring to the creator of the information, the page author, and the sponsoring organization, if any.
The domains have been principally .com, .net, .org, .gov, .edu, and .mil, plus several hundred that identify foreign countries. Recently a few more upper-level domain names have been added: .biz, .coop, .name, .info, .aero, .pro, and .museum. For our purposes, we'll restrict our discussion to this core set of upper-level domain names: .com, .net, .org, .edu, .gov, and .mil. Most websites fall under those domains, with all the other domain names just variations of the same thing.
It's important to understand that .com refers to a commercial site. Mega corporations in manufacturing, service, sales, as well as one-man businesses use this domain name. So do individuals who simply have a personal Web page. The reason individuals with personal Web pages are under .com also is because individual pages are paid for by persons using commercial Web hosting services. The new .name (for individuals) and .biz (for businesses) domains are supposed to help make this distinction clearer. Individual pages are also posted on sites other than .coms, such as .edu and .org., and are indicated with a ~, a mark that is called a tilde. The tilde indicates the authorship of a particular page by a specific individual who works for an entity or organization sponsoring the website.
A word about .net sites. The .net domain name originally indicated sites that were part of a network, such as community networks. Nowadays the .net name is all-inclusive, and any individual, business, or organization can use that domain name for a site.
Sites with .org addresses are created by a nonprofit organizations such as clubs, special interest groups, and trade associations. Educational institutions (schools, colleges, universities) use the .edu domain. Government sites have .org, while military pages use the .mil domain.
When evaluating sites for quality and credibility, I employ the pyramid principle. The best sites are at the top. As you go higher on the pyramid, there is less area covered. There's less area as you go upward because quality is less numerous, while mediocrity (at the base of the pyramid) is much more common.
Domain names can be rated according to the pyramid. Sites with .gov and .edu predominate at the top of the pyramid, followed by .org and then .com. There are some .com sites at or near the top, but most of them are at the bottom.
A number of criteria can be used to help measure the quality and credibility of a site. As mentioned above, these criteria can be summarized with the following acronym to help remember them: AACCUSO: Accuracy, Authority, Credibility, Currency, Usefulness, Scholarliness, and Objectivity.
Accuracy, authority, and credibility go together. Authority helps ensure accuracy and credibility. Accuracy and credibility flow from authority. Let's define what accuracy, authority, and credibility mean.
Accuracy means a minimum amount of error. It means reliability that the information presented can be trusted. Authority means the source of the information is trustworthy and competent. Authority means the author of the information has the credentials and background to present the information accurately. This creates believability, or credibility. Credibility is also enhanced by reputation, especially in the case of a publisher or sponsoring organization.
An author can be an individual or a sponsoring organization (government, corporation, or nonprofit organization). Think of a job resume. An individual's competency can be gauged by his relevant education and experience. Likewise the author of information on a Web page should be open to scrutiny. An organization carries credibility just like an individual. Some organizations, like accredited educational institutions and the government, automatically carry a lot of weight in terms of credibility. These sources are identified by their .edu and .gov domain names. Some organizations, such as businesses or nonprofit organizations, can be viewed as both good and bad sources of information. These sources use .com, .net, and .org domain names.
Are the authors clearly identified? Are the authors or maintainers of the site authorities in their field? Are their names, credentials, and e-mail address given?
The second "C" stands for currency. Currency and timeliness are obviously very important in many fields, topics, and disciplines. Does the site tell when it was last updated? Better yet, does each individual page or document have a date of update or copyright? Your Web browser will tell you when the document was last modified even if there is no visible date on the page itself. In Internet Explorer, click on File, Properties. In Netscape, go to View, Page Info.
Usefulness (or usability) is an all-inclusive term that covers a number of things. Is the information presented appropriate and thorough enough for your needs and the purpose of your research? Is the site easily navigable with working links, and is the presentation reasonably attractive and readable? A website is less useful if the information is hard to locate within its pages or difficult to read because of bad design (such as clashing colors of fonts and background).
Scholarliness is reflected by a number of factors. In the purest sense, scholarly information is information presented by an authority in the field or by someone with acceptable credentials. This information may also be "peer-reviewed" and edited by authorities or sponsored by a reputable organization. Scholarly information offers documentation in the form of cited sources. The information may be "primary," offering original research, or it may be "secondary," which is second-hand reporting of the research. Secondary information that is general and without citation of sources sits low in the pyramid of scholarliness. "Popular" is the other side of the coin of scholarly. Popular information resources are intended for mass audiences and by their nature have a lower level of scholarliness or may not be scholarly at all.
The last criterion in ACCUSSO is objectivity. Objectivity is at the top of the pyramid for the most reliable type of information. The flip side of objectivity is subjectivity and bias. Some subjectivity and bias is obvious because of the stated position or agenda of the author. The point-of-view of the information may be one-sided or dogmatic. That's not so bad if you are aware of it at the outset (such as a site that is openly a mouthpiece for or against certain issues and topics, e.g., abortion, UFOs, environmentalism). At other times subjectivity and bias may be hidden or subtle, conscious or unconscious, on the part of the author, editor, or publisher of the information.
Here are some red flags indicating a poor resource of information: anonymity of author, no indication of credentials, no indication of editor or publisher, sweeping statements, exaggerations, appeals to emotions, use of pejorative terms, obvious or hidden conflicts of interest, bad grammar and spelling errors.
Keep in mind the pyramid when you mine the Internet for information. Start at the top of the pyramid for your information needs, and work your way down if you have to.
For periodical articles, at the top of the pyramid are the databases provided by libraries for their patrons. With your library card from your local college and public library, you have access to reliable periodical resources such as InfoTrac, ProQuest, OCLC FirstSearch, EBSCOhost, and Newsbank, all accessible from the library Web pages with your special passwords or library card number.
Sometimes information is elusive because it is obscure or so very new, and you may need to use Web resources other than periodical databases. These other resources include news sites, newsgroups, invisible sites, electronic journals, and search tools.
If you use search tools, use subject directories or search engines judiciously. An advantage of subject directories (Yahoo! and the Open Directory in Google are the best-known ones) is that, although smaller in total listings than search engines, they are more selective with a higher ratio of quality sites, pre-screened by editors and annotated. There are two types of subject directories, popular and academic. Yahoo! and Google are the best of the broader, popular, commercial kind of directory. For a higher level of scholarly information, start with academic subject directories, which are put together by librarians. Among the best are Infomine (http://infomine.ucr.edu), Librarians' Index to the Internet (http://lii.org), and The Internet Public Library (www.ipl.org).
If you use the search engines, Google (www.google.com) is considered the best general search engine (as opposed to the many subject-specific search engines). Because search engines do not use human editors, but instead rely on robotic "spiders" that roam the Internet and collect sites, there is a great amount of junk that is found by search engines, which calls for more caution on the part of the searcher.
Here are some tips for getting more out of Google and other search engines for scholarly searching. Restrict your search parameters to .edu and .gov sites (these type of sites are at the top of the pyramid). To do this in Google, in the search text box, in addition to your search keywords, type the following: site:.edu or site:.gov. Also, when typing in your keywords, add the words study and journal, and you will pull up information that references studies and that has been cited in journals.
