5.2. What Sources Should I Use?

In your literature review, you should include all the relevant published works written by researchers in your field that document a scholarly conversation on the specific phenomenon, issue, or problem you’re studying. There is no hard and fast rule about what is or isn’t part of the literature on a given topic, but you will usually want your literature review to prioritize studies from your own field (i.e., sociology) or subfield (e.g., economic sociology). Scientists in a particular field or subfield tend to talk to each other, and the “literature” is essentially a collection of these scholarly debates. That said, some topics can’t help but be multidisciplinary—take, for instance, public health issues—and if another discipline has studied a particular phenomenon much more than your own, it behooves you to become familiar with at least some of that work, so you’re not reinventing the wheel or ignoring excellent research.

Another important criterion for deciding what to include in your literature review is whether the work is peer-reviewed. Peer review refers to a formal process in which other researchers ensure that a work of scholarship meets the standards and expectations of the professional field. While mainstream news outlets and book publishers do have capable editors, the review process looks much different at academic journals and academic presses. Typically, peer review involves multiple reviewers who are experts on the topic at hand, and the identities of both the reviewers and authors are kept hidden, encouraging candid and unbiased reviews. In these and other ways, peer review acts as a gatekeeper (hopefully, in a positive sense), ensuring that only rigorous research is published—specifically, research that builds thoughtfully on previous work, rests on sensible strategies of data collection and analysis, and draws reasonable conclusions from the data. Although peer review is far from perfect, the process provides for stricter scrutiny of scientific publications.

In scientific fields, papers published in peer-reviewed journals account for all or most of the sources in an academic literature review. These papers fall into a few different categories. Empirical papers involve the collection and/or analysis of data. This can be original data that the authors gathered by conducting their own interviews, fielding their own surveys, and so on. An empirical paper could also be based on data that someone else collected: in this case, researchers obtain access to that existing dataset but then conduct their own analysis of it. (We call the previously collected dataset secondary data and the original analysis of it a secondary data analysis). Empirical papers contain a discussion of theory, particularly in sections devoted to reviewing the literature and interpreting the study’s empirical results, but the amount of theorizing can vary greatly. In contrast, theoretical papers (or theory papers) focus entirely on building, refining, or challenging theories. The authors will likely comment on empirical work—perhaps to show support for one of their theories or to challenge the validity of previous theories—but there is no original data collection or analysis.

Review articles are another notable kind of paper that you will come across in academic journals, especially in journals like the Annual Review of Sociology and Sociology Compass that specialize in publishing these articles. They are essentially long-form literature reviews not tied to a single empirical study. In addition to surveying the literature on a broad topic, they typically make some overarching arguments about how previous empirical and theoretical work fits together, how best to understand a phenomenon or issue based on existing scientific evidence, and what important omissions exist in that body of research. To the extent they make such arguments, review articles are a kind of theoretical paper, though they can vary considerably in how much they fashion new theories as opposed to describing and categorizing past work. In empirical papers, the literature review often ends with authors making the case for the importance of their own study (building on the previous scholarship just discussed), whereas review articles do not have a specific study to justify because they don’t contain original data collection or analysis. They do, however, frequently put forward a research agenda—that is, a list of unresolved research questions and suggestions for the types of studies that could address critical gaps in the literature.

In your search of the literature, you may also come across systematic reviews. Like review articles, systematic reviews synthesize past research, but they tend to be focused on a narrow empirical question. More common in medical and policy fields, systematic reviews typically attempt to make precise conclusions about the effect of a specific intervention. They usually draw on the existing quantitative literature, although some cover qualitative work. A common type of systematic review, a meta-analysis (also called a meta-synthesis) uses statistical procedures to synthesize the results of many quantitative studies. It is basically a quantitative empirical study that treats past findings as secondary data, calculating the average effect of an independent variable on a dependent variable across studies that measure the same or similar relationships.

When writing a research proposal for your own empirical study, you should incorporate a mix of theoretical and empirical papers in your literature review. Review articles and meta-analyses can be helpful to you in finding or framing the relevant literature, but they won’t show up in every literature review. Adequate coverage of the empirical work on your topic is especially important, given that your goal is to extend that prior work with new data collection and analysis.

So how do you distinguish between these different types of papers? As we discuss in Appendix A: Presenting, Writing, and Publishing, empirical papers (which include meta-analyses) follow a similar structure, with a methods section and a results or findings section somewhere in that mix. While the exact headings may differ slightly from publication to publication, the presence of these two sections will signal to you that you’re reading an empirical paper. Theoretical papers and review articles will not include methods or findings sections. In fact, these papers do not follow a set structure. Authors can more or less organize their arguments in whatever way is convenient. We should also emphasize that just because a paper mentions quantitative or qualitative data does not mean it is an empirical paper. Most papers contain literature reviews, and most of those reviews discuss empirical findings of some sort. For a paper to actually be an empirical study, it needs to report findings from the author’s own analysis.

Let’s talk more about the types of sources that might be included in your literature review. We’ll talk first about papers published as articles in peer-reviewed journals, which will typically be the majority of sources that you cite. In sociology, academic books are also commonly included in literature reviews. Other sources can be helpful, too, but as we will describe, you should be more cautious about relying on them.

Journals and Journal Articles

Papers are typically published in academic journals, periodical publications that contain a number of papers that reflect the journal’s focus. Academic journals can be generalist journals that publish any noteworthy paper in sociology or another discipline, or they can be specialist journals dedicated to a subfield within the discipline (like the sociology of education or organizational sociology). The mission of any journal is to publish work that advances knowledge about the field or subfield.

Journal articles are written by scholars with the intended audience of other scholars interested in the subject matter. They contain extensive references to other studies in the relevant literature, so that readers can better judge the arguments and data and so that the authors connect their research to the work of other scholars in the same area. Professional sociologists often read one or more generalist sociology journals regularly, along with one or more specialty journals in their subfield. As a result, publishing your paper in a sociology journal is often the best way to reach other scholars who are interested in the topics you study.

Scientific journals almost always vet the papers they publish through peer review. Their editors recruit experts in each paper’s area of research who confidentially assess its merits and advise the journal on whether to publish it. For their paper to be accepted, authors usually must revise their work substantially in response to any reviewer feedback, which can go on for multiple rounds. (For more details, see the sidebar Peer Review: What Makes a Science a Science.) Because of this rigorous process of evaluation and revision, journal articles are the main source of scientific information that most researchers rely upon, and they dominate any literature reviews written on a particular scientific topic.

You can often tell whether a journal is peer-reviewed by going to its website. Usually, the website will describe its procedures for peer review. You can also search on a university library catalog; peer-reviewed journals are typically listed as such in their catalog entries. Finally, you can turn to periodical databases such as Ulrichsweb that record whether a particular publication is peer-reviewed.

Note that journals have different reputations for the rigor of their peer-reviewing process and the quality of the work they publish. The “top” journals in the field—for U.S. sociologists publishing empirical papers, this would include the American Sociological Review, American Journal of Sociology, and Social Forces—are able to tap nationally or internationally recognized experts as reviewers on virtually any substantive or methodological topic. These publications also reject the vast majority of submissions. Other journals do not have such strict standards. That does not mean they publish shoddy research, but you want to be more cautious with less selective journals, knowing the review process may not have been as intensive. You should also be aware of so-called predatory journals, which are meant to generate money for the publishing company by charging scholars to have their work published. (Given the difficulty of finding reviewers, some prestigious journals have submission fees, but those fees tend to be small—serving to discourage less serious submissions.) Predatory journals may claim to be peer-reviewed, but in reality, they publish any article—no matter how poor it is—as long as the author pays them.

How do you know which journals are reputable? Perhaps the best way is to read a lot of the literature in your research area and see which journals keep cropping up in the reference lists. These are likely to be reputable journals (since scholars are citing them) and may even specialize in topics like yours, meaning you’ll want to be a regular reader of that journal. You can also get a sense of a journal’s reputation in the field by looking up its impact factor. Databases like Journal Citation Reports rank academic journals based on how many citations their articles received a year or two after publication—a rough measure of how influential those articles are, and therefore how respected the journal that publishes them is. One of the world’s most prestigious academic journals, Nature (which publishes articles across scientific disciplines), had an impact factor of 64.8 in 2022, according to Journal Citations Reports. This means that the papers that were published in Nature in 2020 received, on average, 64.8 citations in 2022. Journals in sociology—a smaller scientific field without the sizable practitioner communities of the hard sciences—tend to have much lower impact factors. And just because a journal does not have a high impact factor does not necessarily mean it is a disreputable journal: journals that specialize in niche subjects may have particularly low impact factors, and yet the most prominent scholars in a subfield may be contributing to them. All that said, journal impact factors provide another way that you can check the trustworthiness of a source you’re considering using.

Part of the tuition your university or college charges you goes to paying the publishers of academic journals for the privilege of accessing their articles. Because access fees are incredibly expensive, your school likely does not pay for access to all the journals in the world. Nevertheless, you should never have to pay for access to an academic journal article while you are in school. If your library does not subscribe to a journal you need to read, you can usually go through its website to order the article through interlibrary loan (you can follow the same procedure to get a copy of an academic book, too). If you are not currently affiliated with a university, getting academic sources is more challenging, but you should feel comfortable asking an author for a copy of their article. Academics are not paid any royalties for journal articles (and sometimes they don’t even receive them for academic books), and they will usually be happy to share their papers with anyone interested in their work. Most academic journals allow their authors to share their articles privately with interested readers, so the authors won’t get in copyright trouble, either.

You should keep an eye out for an important string of numbers (and sometimes letters) labelled “DOI” or “doi” somewhere on the first page of a journal article. This stands for digital object identifier. Based on an international standard, DOIs are unique identifiers for scholarly papers, government reports, and other documents. Most recently published scholarly materials (including this textbook) have their own DOIs. Although you should not assume by the presence of a DOI alone that an article is scholarly—much less that it is peer-reviewed—seeing one listed is a hint that you’re on the right track. Note that lesser-known journals may not have assigned DOIs to their articles. Likewise, even though DOIs have been added to some older sources, for the most part articles published before 2000—when the nonprofit DOI Foundation introduced the standard—will not have them. One useful feature of DOIs is that you can enter them into a web browser to be sent directly to the official webpage for that document, such as the article’s page on its journal’s website. Simply copy and paste the DOI after https://doi.org/ in your browser. For instance, if the article’s DOI is 10.1177/15365042221131082, then https://doi.org/10.1177/15365042221131082 will take you to that article’s webpage.

Academic Books: Monographs and Edited Volumes

In some fields (including sociology), the academic literature also includes monographs, scholarly books that focus on a particular topic. Sometimes, chapters of a monograph will have been previously published as journal articles (which means you don’t necessarily have to buy or borrow the whole book to learn the gist of the arguments being made). Monographs will often take on the structure of a journal article, too, with a chapter devoted to a literature review, a chapter devoted to describing the study’s methods, and separate chapters on key findings.

Monographs are often published by university presses, nonprofit publishers affiliated with universities that typically have the words “university press” in their names. University presses and other scholarly presses routinely peer-review their book manuscripts. The process is different from what takes place with refereed journals, however. For one thing, when a book author is pitching their book to a university press, that press may recruit peer reviewers to take a look at the submitted book proposal—which typically includes a few sample chapters. Those reviewers will give the press and the author their feedback on the quality of the proposal and whether the book should go forward. Later, once the book project is completed, a peer review will also be conducted on the entire manuscript (possibly by an entirely new group of reviewers). Another difference between journal and book peer-reviewing is that the reviews for book manuscripts are single-blinded: the reviewers know who the author or authors are (it is just too impractical to mask their identities in such a long work), although the names of the reviewers are still kept secret.

Sometimes research written by different scholars is collected together in what is called an edited volume—a book whose chapters are individual papers. These papers may not have made it into academic journals for whatever reason, or they may be seminal articles that are being republished in the volume. Edited volumes are considered less reputable than journal articles, as the papers collected within them are often reviewed in a less rigorous fashion (e.g., without blinding)—if they are peer-reviewed at all. That said, chapters in edited volumes are routinely cited and are considered a legitimate source to use in your literature review.

Other Scholarly Sources of Information

When writing a literature review, you should prioritize peer-reviewed journal articles and monographs as sources. Other types of material can be used, but they are generally places to start—not end—your search for sources. Consider academic encyclopedias and textbooks. When you are first getting to know your research area, you might want to consult an academic encyclopedia like the Blackwell Encyclopedia of Sociology, which covers a wide range of sociological topics and theorists, or the Stanford Encyclopedia of Philosophy, which has briefs on a number of social theorists. Some textbooks (including this one) are peer-reviewed by experts, and they can include useful discussions of, and citations for, literature relevant to your study. However, professional sociologists rarely mention these works in their papers. Instead, they will go straight to the sources that these sources cite and branch out from there. Review articles do tend to be cited in professional scientific writings—especially if they have a particular interesting theoretical take on the literature, or if the person citing the review article wants to briefly refer to a body of literature that is not central to their research question (e.g., “for more on this topic, see this review article”). But again, it is usually best to go straight to the original sources that the article mentions.

Another kind of scholarly source is gray literature, which is research and information produced by nonacademics. These can be reports by government agencies like the Centers for Disease Control and Prevention or U.S. Census Bureau, write-ups of survey results by polling outfits like Gallup, or policy briefs by think tanks like the (progressive) Center on Budget and Policy Priorities and (conservative) Manhattan Institute. As we mentioned in Chapter 2: Using Sociology in Everyday Life, government agencies, pollsters, and think tanks have researchers on staff who are trained social scientists. As a result, you may find yourself drawing upon their analyses and interpretations of data for your literature review, particularly when you want to describe a particular context. (The official statistics that government agencies provide are invaluable in this regard.) That said, this kind of research is not peer-reviewed and rarely engages with theory, which makes it less useful to sociologists. It also frequently espouses a specific political viewpoint in line with the organization’s mission, which may make any findings suspect.

At times, you will come across writings by scholars that are published in nonacademic venues. For instance, academics occasionally write books for mainstream commercial presses (called trade presses) rather than university presses. Similarly, scholars can write about their research or related issues in articles published by news outlets. The writings they produce for these outlets frequently take the form of commentary pieces, or op-eds (we discuss how to pitch and write op-eds in Appendix A: Presenting, Writing, and Publishing). Sociologists may also write about their research on personal or organizational blogs; various sections of the American Sociological Association host their own blogs, and some blogs written by social scientists have become influential in wider policy debates.

Trade books and op-eds are not peer-reviewed. That said, they may be subject to intensive editing and even fact-checking, depending on the publisher. They also may engage in scholarly debates and discuss theories and empirical findings that might be useful for literature reviews. As a result, these works are sometimes heavily cited in academic circles, although really their audience is the general public (or whatever audience the publication or publisher caters to). You can use them as sources, but do not rely on them too heavily; instead, go to the journal articles or monographs that are usually the basis for the scholar’s popular writings.

You may come across a variety of academic “works in progress” that can be included in your literature review, but you generally will not want to use them if more polished work is available. For instance, dissertations and theses—which often go on to be published as journal articles or monographs—can be rich sources of information and feature extensive reference lists to scan for resources. As part of the dissertation and thesis defense, a committee of professors did review the work, and it needed to gain their approval to be accepted. This is not the same thing as a peer review, however, and you may want to wait until the work is published in a journal or as a book before citing its findings. Regardless, you should definitely read the manuscript’s literature review and plumb the sources the author uses for your own review.

Talks at academic conferences (which we describe in Appendix A: Presenting, Publishing, and Writing) are sometimes cited as academic sources. At conferences in some disciplines, the papers presented will be published in a peer-reviewed volume of “conference proceedings”—when they reach that stage, they are not works in progress but rather publications, some of which go on to be widely cited in their field. In sociology, however, conferences do not tend to impose a rigorous peer-review process for papers—and many don’t even require scholars to submit completed manuscripts in order to present their work. As a result, conference papers and presentations should be used less as sources for your literature review and more as guides leading you to other good research—for instance, other work by the same scholars on that topic, or the final version of the presented paper that is later published in a journal. If you do find a relevant conference presentation, you should ask the presenter to pass along the corresponding paper manuscript. As we noted earlier, most scholars are happy to share their work with you—whether it’s a conference paper or even a published journal article.

A final suggestion about evaluating scholarly sources: when academic databases list a particular paper or book in their search results, they often include the number of other academic sources that have cited that paper or book. Like impact factors for journals, a citation count for papers or books is a blunt indicator of how influential that work of scholarship is within the research community (note, of course, that sometimes an article is cited a lot because it is controversial!). When reviewing the scholarly literature, you may want to start with well-cited books and articles so that you have covered any “must-read” studies in your research area—but remember that sociology articles in general tend to have lower citation counts than those in larger scientific fields.

Nonacademic Sources

When you start learning about your research topic, you will often come across two types of nonacademic sources. First, there may be news articles discussing the issue. These pieces are intended for a general audience and are published in periodicals (magazines and newspapers) and online news outlets. With the exception of scholarly op-eds and the like, they are usually written by journalists. Second, you may encounter trade publications—magazines, newsletters, and (nonacademic) journals geared toward people who work in certain industries or professions. Unlike magazines and newspapers, trade publications may take some specialized knowledge to understand. They often have information about industry trends and practical information for people working in the field.

Trade publications may be somewhat more reputable than general-interest periodicals because the authors tend to be specialists in their field. Although the authors’ writings for a trade publication will not have gone through peer review, their take on a topic may be relevant to your literature review, especially if your research delves into practical applications. Generally speaking, however, neither news articles nor trade publications belong in your literature review. As we have emphasized, your proposed study needs to be connected to the scholarly conversation in the scientific literature, and reading too many nonacademic sources can be distracting and divert you from that conversation.

Nonacademic sources do have some uses in academic writing. For one thing, news articles are frequently used as data in a content analysis, which is a scholarly analysis of existing text and media often intended to explore popular views of an issue within a particular context (we will cover content analysis in Chapter 15: Materials-Based Methods). For a literature review, nonacademic publications can provide supplementary context or help with brainstorming. There may be pertinent facts or ideas in these sources you should include, especially when you need to offer some general background on a topic (which typically goes in the introduction to a research proposal or research paper). In particular, academics rely on journalistic pieces for details about specific events or organizations they are discussing. In this case, they are drawing on the news articles as primary sources—first-hand accounts of an event or an organization, which they can use as data or cite in their literature reviews to back up a claim. Although these sources are not peer-reviewed, more reputable news organizations have high standards for their reporting and may utilize professional fact-checking—which sometimes means their factual accounts may be more reliable than those in academic writing. Beyond their regular reporting, news organizations at times will gather statistics that can also be of use to social scientists. For instance, news organizations like the New York Times and CNN regularly commission political polls. In the absence of government reporting, the Washington Post and other news outlets have compiled historical databases tracking police shootings.

Nonacademic sources are also a good place to start when you are doing your initial brainstorming. They can spark ideas and give you relevant and precise keywords to use in searching the academic literature, the real target of your review. For instance, news articles often summarize the results of noteworthy scientific studies. In these cases, the articles are secondary sources, reporting the results of original research.

In a scholarly journal article, you will usually be given a great deal of information about the researchers’ method of data collection, their sample, and the process by which they recruited their research participants. All of these details provide important contextual information that can help you assess the researchers’ claims. But when you read a journalist’s write-up of social scientific research in a news article, you usually won’t get the same level of detail about the study. You can use these secondary accounts of research to give you a quick sense of what social scientists have found and to help you locate the relevant academic literature. However, you should not rely solely on the summaries of research in secondary sources: you must read the original source. Besides for the lack of detail they provide in their write-ups of studies, journalists are not scientists. You have undoubtedly seen news articles making claims about how a particular food prevents cancer or how a new treatment can extend your life, which later turn out to be wildly inaccurate or premature. Clearly, journalists can exaggerate or misinterpret results. Careful scholars will critically examine the primary source rather than rely on someone else’s summary. Likewise, many news outlets also produce opinion pieces, which are not always labeled as commentary rather than news. These articles may present facts that support the author’s viewpoint while excluding or downplaying facts that contradict it, and they are best avoided.

What about an organization’s website, press releases, and social media feed? If an organization is a noteworthy player in the field you’re studying, then looking at their posts, reports, announcements, and other online content can clue you into important topics to study. Many advocacy websites will provide references for the facts they cite, providing you with the primary source of the information. Again, though, you generally want to turn to the original source rather than citing the webpage that mentions it.

Another type of source that students go to almost immediately is general-interest encyclopedias like Wikipedia. (These are tertiary sources, which summarize secondary sources.) For its part, Wikipedia is a marvel of human knowledge. For the most part anyone can contribute to this digital platform, but the entries are overseen by skilled and specialized editors who volunteer their time and knowledge to ensure that their articles are correct and up to date. Nevertheless, you will almost never want to cite Wikipedia or similar general sources in a literature review. Instead, use it as a resource to get a general sense of the research on a topic and look to any secondary sources it cites.

Just like academic sources, nonacademic sources vary greatly in terms of how carefully they are researched, written, edited, and reviewed for accuracy. Common sense will help you identify obviously questionable sources, such as highly political partisan news outlets or a random person’s blog. Sometimes, however, a source’s reliability is not so obvious. You will need to consider criteria such as the type of source, its intended purpose and audience, the author’s qualifications, the publication’s reputation, any indications of bias or hidden agendas, how current the source is, and the overall quality of the writing, thinking, and design.

Nonacademic sources are excellent places to start your journey into the literature, because they do not require specialized knowledge to understand the content and may inspire deeper inquiry. Nevertheless, you generally don’t need to discuss or cite them unless you need attribution for a specific fact—for instance, the results of a political poll mentioned in a news article. Although nonacademic sources can be an important part of how we learn about a topic, your literature review should instead focus on finding peer-reviewed articles and books. These publications are really the basis of scientific conversations among researchers.