5.4. How Do I Read and Evaluate Sources?

By now, you’ve hopefully collected a number of academic sources relevant to your topic area. It’s now time to evaluate the information you’ve found. Using inaccurate, disreputable, and carelessly researched information will undermine the perceived quality of your own work. You not only want to be sure of the credibility and quality of each of the sources you use, but you also want to determine whether it is an appropriate fit for your literature review. You should always have a good reason to mention a work; if it does not help you make the case for your research problem and the study that addresses it, then cut it.

When evaluating whether or not to include a source in your literature review, ask yourself a series of questions:

1. Does the author have the credentials to write on the topic? Search the author’s name in a general search engine like Google. What are the researcher’s academic credentials? What else has this author written? Search by the author’s name in the databases you are using (in Google Scholar, prominent authors will have their own pages) and see how much they have published on any given subject. If you are writing a sociological literature review, you typically want to prioritize authors who are sociologists, because then other sociologists who read your work will know whom you are talking about and will be able to clearly connect the studies mentioned to one another.
2. Who published the source? You will want to make sure that any journal articles you cite have been published by a reputable journal with high standards of peer review. Avoid predatory journals that publish anything, or journals that other scholars have not heard of—which typically means that well-respected scholars are not providing peer reviews for these journals. Likewise, you may want to prioritize books published by reputable publishers. Remember that university presses conduct peer reviews of the books they publish, whereas trade presses typically do not. For gray literature produced by think tanks, government agencies, or other organizations, look them up online to learn more about what potential biases they might have, including from their funding sources. Again, you generally do not want to include nonacademic sources in a literature review, but if you choose to do so, definitely avoid articles from obscure publications. A site with numerous typos, outdated design, dubious advertising, and other evidence of unprofessionalism is unlikely to provide trustworthy content.
3. Is the source reliable and objective? What point of view does the author represent? Are they upfront about that point of view? Is the article trying to argue a position, and if so, is it doing that in a way that adequately considers opposing sides? Is the article in a publication with a particular editorial position? Does it fairly present the findings or views of other authors with different perspectives on your topic? Do the researchers have any discernible conflicts of interest? On this last point, funders usually require that recipients acknowledge them in academic publications. But publications in the popular press routinely leave out funding sources, and the researcher may fail to disclose a personal stake they have in the research they conduct. In these cases, do a quick search on the web to see if you can learn more about a researcher’s funding, affiliations, and other potential biases. You will probably want to put less weight in your literature review on research that appears motivated to support a particular agenda. At the same time, the fact that a study received funding from a particular organization, even a political organization, doesn’t necessarily discount its findings. And researchers may not be able to disclose certain details of their study even if they wanted to: as we will discuss in Chapter 8: Ethics, they are ethically bound to protect the identities of participants unless those people explicitly say otherwise. Nevertheless, getting as much context about a study as you can allows you to better assess what you can legitimately take away from a given set of findings.
4. Is the information up to date? Prioritize sources that are more current—published, say, within the past five or 10 years. It’s fine to cite earlier work, especially since the more recent work builds upon that past work, but you want to make sure that the literature you discuss also covers what we currently know about the topic—not just what we used to know.
5. How accurate is the source’s information, and how strong is its evidence? Check the facts in the article. Can the statistics be verified through other sources? Does this information seem to fit with what you have read in other sources? When sources mention existing research, do they accurately summarize the results of studies that you can read for yourself? For a research study, what research methods were used? How credible are its findings, given its research design? (On this last point, see the sidebar A Hierarchy of Evidence?)
6. How important is this source in the literature? As noted earlier, Google Scholar and other databases list the number of citations a particular source has within the scholarly literature—a rough measure of that study’s impact within the research community. Your literature review might want to cover any papers that score high on these or similar indicators, since they are likely to be seminal within your field.
7. Is the source relevant to your topic? This is perhaps the most important question to ask. Some sources may be included in your literature review even if you have problems with their methods or findings—you can always explicitly discuss those problems, after all—but you don’t want to waste your time reading and summarizing irrelevant literature. Don’t try to read everything you dredge up during your searches, and don’t include a source in your literature review unless you have a good reason to do so. To suss out a source’s relevance, consider these questions: How does the article fit into the scope of the literature on this topic? Is it a general work that provides an overview, or is it focused on only one aspect of your topic? Does the information in it help you answer your research question, or is it difficult to make some sort of connection? Does the source present one or more points of view in a debate, so that you can show that you have addressed all sides of the argument in your paper?

Breaking Down an Empirical Paper

While you should get into the habit of familiarizing yourself with each part of the papers you wish to cite, there are techniques for reading academic sources that can make them a little easier to digest quickly. Read the sidebar How to Read Social Science Writing Strategically for some general tips. During your initial searching of the literature, you should just be reading abstracts and perhaps skimming for some additional key details. Once you have identified an empirical paper that you’d like to read more carefully, skim through its introduction, literature review, discussion, and conclusion sections. (As you know, theoretical articles and books have no set structure, but reading the introduction and conclusion should cover much the same ground.) The paper’s literature review will tell you what previous empirical and theoretical work the author considers to be important within this research area. The discussion and conclusion sections will help you understand what the author views as their study’s major findings and limitations and how the author perceives their work to relate to other research.

Ask yourself the following questions about each section of an article—from the abstract to the acknowledgments—as you read through it:

1. Title: What is the focus of this study? The key concepts in the study’s research question (or the research question itself) will usually be included in its title. (Sometimes, the title is a sexy title meant to grab people’s attention, while the subtitle contains the actual concepts of interest.)
2. Author(s): Who conducted the study, and what institutions are they affiliated with? What biases might have affected this research because of who the authors are or which organization they work for?
3. Abstract: What are the key findings? How were those findings reached? What theoretical framework does the researcher employ? Note that sometimes a journal article or book chapter will have keywords listed below the abstract that are used to help with database searching. These keywords will likely include the most important concepts in the study.
4. Introduction: How does the author frame their research focus? What other possible ways of framing the problem exist? Why might the author have chosen this particular way of framing the problem?
5. Literature review: How selective does the researcher appear to have been in identifying relevant literature to discuss? Does the review of literature appear appropriately extensive? Does the researcher provide a critical review? Carefully go through the paper’s literature review as well as its reference list to see which other scholars the study’s authors are in conversation with—whom they are relying on to support their arguments, and whose arguments they are countering, contradicting, or expanding upon.
6. Methods (sample, case selection): Where was the data collected? Did the researcher collect their own data or use someone else’s data? What population is the study trying to make claims about, and does the sample represent that population well? What are the sample’s major strengths and major weaknesses?
7. Methods (data collection): How was the data collected? What do you know about the relative strengths and weaknesses of the method employed? What other methods of data collection might have been employed, and why was this particular method employed? What do you know about the data collection strategy and instruments (e.g., questions asked, locations observed)? What don’t you know about the data collection strategy and instruments?
8. Methods (data analysis): How was the data analyzed? Is there enough information provided for you to feel confident that the proper analytic procedures were employed?
9. Findings or results: What are the study’s major findings? Are the findings linked to previously described research questions, objectives, hypotheses, and literature? Are sufficient amounts of data (e.g., quotes and observations in qualitative work, statistics in quantitative work) provided to support the conclusions drawn? Are the tables readable?
10. Discussion, implications, or conclusion: Does the author generalize the study’s findings to some population beyond the sample used in the study? How are these claims presented? Are the claims made supported by data provided in the results section (e.g., supporting quotes, statistical significance)? Have any limitations of the study been fully disclosed and adequately addressed? Are the study’s theoretical and policy implications sufficiently explored?
11. Acknowledgments and declaration of conflicts of interest: Who provided feedback on this research? Who provided support in the form of funding or other resources? What potential conflicts of interest did the researcher disclose? The declaration of conflicts of interest does not appear in every journal article, but if something is listed there, you should definitely consider the potential for bias in the research. While the acknowledgments may seem like a personal note, they can also be useful to help situate the work. Specifically, they help identify the study’s major stakeholders. For instance, scholars will thank the foundations, government agencies, and other organizations that funded their research, which may say something about the scholarly or political perspective the authors are taking. Scholars often show their work to mentors or peers before submitting their work for publication, and they will thank these individuals in their acknowledgments. Whom they thank may give you clues about where they fall in the debates on a particular scholarly topic, especially when the authors are more junior scholars whose perspective may have been shaped by an influential mentor.

(Note: For a more detailed discussion of each part of an empirical journal article, refer to Appendix A: Presenting, Writing, and Publishing.)

As you read through the literature, take detailed notes and use a consistent format to record key information about each source. You will want to get through each paper quickly but also get everything you need. One strategy is to break down articles into their constituent parts, taking notes on the following details:

1. Research question: Read through the abstract, introduction, and literature review to identify the study’s research question or questions. Note that the question may be phrased in multiple ways. In fact, you may wish to rephrase it for your own sake using the templates we described in Chapter 4: Research Questions, which may clarify for you the variables and relationships that are central to the study. For a qualitative study, the research question could take the form: “How does the independent variable affect the dependent variable?” For a quantitative study, it could take the form: “What effect does the independent variable have on the dependent variable?” In any case, make sure to identify all the relevant variables as well as how they are measured—not just those stated explicitly in the research question, but also (if applicable) moderating variables, control variables, and causal mechanisms.
2. Research problem: Look to the abstract, introduction, and literature review to find the study’s research problem. If the paper was clearly written, it will talk about the specific gaps in the literature that the study intended to address. You may come across phrasings that signal a research problem: “Past research assumed,” “Scholars have not yet looked at,” “What has been less studied is,” and so on. (See the University of Manchester’s useful Academic Phrasebank for other telltale language that scholars use to highlight the “inadequacies of previous studies,” the “paucity or lack of previous research,” or a “knowledge gap in the field of study”—that is, how they flag potential research problems.) Watch for these turns of phrase in the articles you read, which will help you spot the research problem that each study is seeking to resolve.
3. Hypotheses: These hypotheses should be the expected answers to the research questions, derived from the study’s literature review. They might be listed formally or described in paragraph form, and sometimes (particularly for qualitative work) they are implied rather than explicitly stated.
4. Methods: The paper’s methods section (which may go under different names and may encompass multiple sections) will tell you about the type of data collected (units of analysis), the sample size, the sample selection procedures, data collection and data analysis methods used, the measures used (i.e., how the study’s key concepts were operationalized), the control variables used, and so on. If the study is particularly important to your own study—perhaps because you are improving upon its methods, or studying the same question in a different context—you may want to write extensive notes about the methods it used; otherwise, you can quickly summarize the highlights.
5. Findings: Based on summaries throughout the article (including the abstract, introduction, and conclusion), you should be able to boil down the study’s results to several key takeaways. These should be answers to (1), the research question, and may or may not coincide with (3), the hypotheses (you should note whether the findings confirm or disconfirm those hypotheses). In addition to summarizing the study’s overarching arguments, it’s a good idea to include specific examples—of key statistics from a quantitative analysis, or representative quotes or vignettes from qualitative analysis—to flesh out those points in your notes. Quantitative studies may concisely capture some of these takeaways in their tables and charts.
6. Theoretical implications: The abstract, introduction, discussion, and conclusion should make a case for how the study addressed (2), the research problem. Circle back to the research problem and see how exactly the study built on, contradicted, or otherwise engaged with past studies. Also, see if the authors discuss directions for future research (e.g., “more research is needed in this area”). They usually do this at the end of their paper, sometimes in a separate section called “Future Research.” Record any promising ideas, which may flag a research problem that your own study can address.
7. Limitations: The authors should have discussed the limitations of their work in the paper’s conclusion or methods section or in a separate limitations section. If possible, you should go beyond what limitations the authors divulged and see if you can identify any strengths and weaknesses in the methodological or theoretical approaches they pursued for their study. Are there any scope conditions for the study—that is, contexts where its findings are likely to hold up or not hold up? Were there any shortcomings in their data collection or analysis? Every social scientific study has weaknesses, and you should be able to identify one or more just by the methods they chose—all of which have trade-offs, as we have discussed.
8. Other important details: For articles that are highly relevant to your study, you may want to record other specifics not captured in these categories—for instance, the definitions of key terms, or any statistics or trends mentioned that might provide helpful context.
9. Relevance to your research question: Finally, note how the source relates to other studies, and how it is relevant to your study. (The two considerations are intertwined, as your study may want to address disagreements between this study and others, or build on that previous body of work.) You may want to rank each source as “high,” “medium,” or “low” relevance. For papers that you decide not to include in your literature review, you may want to note your reasoning for exclusion, such as “small sample size,” “local case study,” or “lacks evidence to support conclusions.”

Depending on your personal preference, you might want to store your notes on each source in a grid, table, or an outline. This breakdown (sometimes called a literature review matrix or summary table) will help you compare and contrast sources, see the relationships among them, and identify common themes as well as areas where you may need to do more searching. You can organize this information within a document or a spreadsheet—perhaps ordering it by the author’s last name, with each source on a separate page in a document or a separate row in a spreadsheet. The spreadsheet version of this literature review matrix is depicted in Figure 5.2. Whichever method you choose, this type of organization will help you to both understand the information you find and structure the writing of your review. Remember that the goal is always to relate whatever you read specifically to your research question.

As you read through your expanding matrix of sources, look for common themes, as they may provide the structure for your literature review. And, remember, research is an iterative process. You may find yourself going back and searching reference lists and databases for more sources as you process the articles you’ve already collected.

Identifying a Research Problem

By now, we have hammered into your head that your literature review must identify a research problem and must make a strong case that your study will fill that gap in the literature. But how exactly does it fill the gap? In a nutshell, a study addresses a research problem by agreeing or disagreeing with past research.

Wayne Booth and his collaborators (2016) describe these two stances as creative agreement and creative disagreement. In the first case, a study builds on existing work—for instance, by providing new evidence to confirm its findings, extending its theories to new contexts, or fleshing out particular aspects of its explanations. In the second case, a study contradicts past research, such as by finding alternative explanations for a phenomenon or challenging the evidence or logic of previous studies. This agreement or disagreement generally takes one of three forms. First, the new study can improve theoretically on previous scholarship, bringing new or refined theories to bear on the phenomenon of interest. It can improve empirically on past work, introducing novel or superior data. And it can improve on the literature methodologically, utilizing better methods of data collection and analysis.

There are a wide variety of strategies for creatively agreeing or disagreeing with other scholars, and you can look at Booth and his collaborators’ textbook (2016) for a fuller accounting. We want to flag two relatively straightforward intellectual moves, however, that novice researchers can follow to identify a research problem and justify their proposed study. First, you can creatively agree with an existing theory by extending it to new contexts. For instance, previous research found that ride-sharing apps operate in an exploitative fashion in rich countries; does the same dynamic hold up in poor countries, where different institutions of government regulation and worker organizing are operating? Second, you can creatively disagree with previous work by applying a different theoretical perspective. Psychologists have looked at unemployment in terms of the ways it challenges a person’s identity; can we look at this identity loss from a sociological perspective, in terms of how the breaking of ties to organizations and social networks weakens those identities?

The articles you read as part of your literature review will help you uncover one or more research problems, though you will also have to draw on your own creativity and critical thinking as a sociologist. Please be aware that the research problem that your study ultimately settles on may not be the one you started out with. Through the process of data collection and analysis, you may realize that your findings speak to a more important or pertinent research problem than what you initially identified in your literature review. That’s a good thing. Skilled researchers adapt to the facts on the ground, and they inevitably come up with more innovative and complex ideas as they dive more deeply into the data.