11 3.1 Reading an empirical journal article
- Identify the key components of empirical journal articles
- Define the basic elements of the results section in a journal article
- Describe statistical significance and confidence intervals
Reading scholarly articles can be a more challenging than reading a book, magazine, news article—or even some textbooks. Theoretical and practical articles are, generally speaking, easier to understand. Empirical articles, because they add new knowledge, must go through great detail to demonstrate that the information they offer is based on solid science. Empirical articles can be challenging to read, and this section is designed to make that process easier for you.
Nearly all articles will have an abstract, the short paragraph at the beginning of an article that summarizes the author’s research question, methods used to answer the question, and key findings. The abstract may also give you some idea about the theoretical perspective of the author. So, reading the abstract gives you both a framework for understanding the rest of the article and its punch line–what the author(s) found and whether the article is relevant to your area of inquiry. For this reason, I suggest skimming abstracts as part of the literature search process.
As you will recall from Chapter 2, theoretical articles have no set structure and will look similar to reading a chapter of a book. Empirical articles contain the following sections (although exact section names vary): introduction, methods, results, and discussion. The introduction contains the literature review for the article and is an excellent source of information as you build your own literature review. The methods section reviews how the author gathered their sample, how they measured their variables, and how the data were analyzed. The results section provides an in-depth discussion of the findings of the study. The discussion section reviews the main findings and addresses how those findings fit in with the existing literature. Of course, there will also be a list of references (which you should read!) and there may be a few tables, figures, or appendices at the end of the article as well.
While you should get into the habit of familiarizing yourself with each part of the articles you wish to cite, there are strategic ways to read journal articles that can make them a little easier to digest. Once you have read the abstract for an article and determined it is one you’d like to read in full, read through the introduction and discussion sections next. Because your own review of literature is likely to emphasize findings from previous literature, you should mine the article you’re reading for what’s important to know about your topic. Reading the introduction helps you see the findings and articles the author considers to be significant in the topic area. Reading an article’s discussion section helps you understand what the author views as their study’s major findings and how the author perceives those findings to relate to other research.
As you progress through your research methods course, you will pick up additional research elements that are important to understand. You will learn how to identify qualitative and quantitative methods, the criteria for establishing causality, different types of causality, as well as exploratory, explanatory, and descriptive research. Subsequent chapters of this textbook will address other elements of journal articles, including choices about measurement, sampling, and design. As you learn about these additional items, you will find that the methods and results sections begin to make more sense and you will understand how the authors reached their conclusions.
As you read a research report, there are several questions you can ask yourself about each section, from abstract to conclusion. Those questions are summarized in Table 3.1. Keep in mind that the questions covered here are designed to help you, the reader, to think critically about the research you come across and to get a general understanding of the strengths, weaknesses, and key takeaways from a given study. I hope that by considering how you might respond to the following questions while reading research reports, you’ll gain confidence in describing the report to others and discussing its meaning and impact with them.
|Report section||Questions worth asking|
|Abstract||What are the key findings? How were those findings reached? What framework does the researcher employ?|
|Acknowledgments||Who are this study’s major stakeholders? Who provided feedback? Who provided support in the form of funding or other resources?|
|Problem statement (introduction)||How does the author frame their research focus? What other possible ways of framing the problem exist? Why might the author have chosen this particular way of framing the problem?|
|How selective does the researcher appear to have been in identifying relevant literature to discuss? Does the review of literature appear appropriately extensive? Does the researcher provide a critical review?|
|Sample (methods)||Where was the data collected? Did the researcher collect their own data or use someone else’s data? What population is the study trying to make claims about, and does the sample represent that population well? What are the sample’s major strengths and major weaknesses?|
|Data collection (methods)||How were the data collected? What do you know about the relative strengths and weaknesses of the method employed? What other methods of data collection might have been employed, and why was this particular method employed? What do you know about the data collection strategy and instruments (e.g., questions asked, locations observed)? What don’t you know about the data collection strategy and instruments?|
|Data analysis (methods)||How were the data analyzed? Is there enough information provided for you to feel confident that the proper analytic procedures were employed accurately?|
|Results||What are the study’s major findings? Are findings linked back to previously described research questions, objectives, hypotheses, and literature? Are sufficient amounts of data (e.g., quotes and observations in qualitative work, statistics in quantitative work) provided in order to support conclusions drawn? Are tables readable?|
|Discussion/conclusion||Does the author generalize to some population beyond her/his/their sample? How are these claims presented? Are claims made supported by data provided in the results section (e.g., supporting quotes, statistical significance)? Have limitations of the study been fully disclosed and adequately addressed? Are implications sufficiently explored?|
Understanding the results section
As mentioned previously in this chapter, reading the abstract that appears in most reports of scholarly research will provide you with an excellent, easily digestible review of a study’s major findings and of the framework the author is using to position their findings. Abstracts typically contain just a few hundred words, so reading them is a nice way to quickly familiarize yourself with a study. If the study seems relevant to your paper, it’s probably worth reading more. If it’s not, then you have only spent a minute or so reading the abstract. Another way to get a snapshot of the article is to scan the headings, tables, and figures throughout the report (Green & Simon, 2012). 
At this point, I have read hundreds of literature reviews written by students. One of the challenges I have noted is that students will report the summarized results from the abstract, rather than the detailed findings in the results section of the article. This is a problem when you are writing a literature review because you need to provide specific and clear facts that support your reading of the literature. The abstract may say something like: “we found that poverty is associated with mental health status.” For your literature review, you want the details, not the summary. In the results section of the article, you may find a sentence that states: “for households in poverty, children are three times more likely to have a mental health diagnosis.” This more detailed information provides a stronger basis on which to build a literature review.
Using the summarized results in an abstract is an understandable mistake to make. The results section often contains diagrams and symbols that are challenging to understand. Often, without having completed more advanced coursework on statistical or qualitative analysis, some of the terminology, symbols, or diagrams may be difficult to comprehend. To that end, the purpose of this section is to improve reading comprehension by providing an introduction to the basic components of a results section.
Journal articles often contain tables, and scanning them is a good way to begin reading an article. A table provides a quick, condensed summary of the report’s key findings. The use of tables is not limited to one form or type of data, though they are used most commonly in quantitative research. Tables are a concise way to report large amounts of data. Some tables present descriptive information about a researcher’s sample, which is often the first table in a results section. These tables will likely contain frequencies (N) and percentages (%). For example, if gender happened to be an important variable for the researcher’s analysis, a descriptive table would show how many and what percent of all study participants are women, men, or other genders. Frequencies or “how many” will probably be listed as N, while the percent symbol (%) might be used to indicate percentages.
In a table presenting a causal relationship, two sets of variables are represented. The independent variable, or cause, and the dependent variable, the effect. We’ll go into more detail on variables in Chapter 6. The independent variable attributes are typically presented in the table’s columns, while dependent variable attributes are presented in rows. This allows the reader to scan across a table’s rows to see how values on the dependent variable attributes change as the independent variable attribute values change. Tables displaying results of quantitative analysis will also likely include some information about the strength and statistical significance of the relationships presented in the table. These details tell the reader how likely it is that the relationships presented will have occurred simply by chance.
Let’s look at a specific example. Table 3.2, which is based on data from a study of older workers conducted by Dr. Blackstone, an original author of this textbook. It presents the causal relationship between gender and experiencing harassing behaviors at work. In this example, gender is the independent variable (the cause) and the harassing behaviors listed are the dependent variables (the effects).  Therefore, we place gender in the table’s columns and harassing behaviors in the table’s rows.
Reading across the table’s top row, we see that 2.9% of women in the sample reported experiencing subtle or obvious threats to their safety at work, while 4.7% of men in the sample reported the same. We can read across each of the rows of the table in this way. Reading across the bottom row, we see that 9.4% of women in the sample reported experiencing staring or invasion of their personal space at work while just 2.3% of men in the sample reported having the same experience. We’ll discuss p value later in this section.
|Behavior Experienced at work||Women||Men||p value|
|Subtle or obvious threats to your safety||2.9%||4.7%||0.623|
|Being hit, pushed, or grabbed||2.2%||4.7%||0.480|
|Comments or behaviors that demean your gender||6.5%||2.3%||0.184|
|Comments or behaviors that demean your age||13.8%||9.3%||0.407|
|Staring or invasion of your personal space||9.4%||2.3%||0.039|
|Note: Sample size was 138 for women and 43 for men.|
These statistics represent what the researchers found in their sample, and they are using their sample to make conclusions about the true population of all employees in the real world. Because the methods we use in social science are never perfect, there is some amount of error in that value. The researchers in this study estimated the true value we would get if we asked every employee in the world the same questions on our survey. Researchers will often provide a confidence interval, or a range of values in which the true value is likely to be, to provide a more accurate description of their data. For example, at the time I’m writing this, my wife and I are expecting our first child next month. The doctor told us our due date was August 15th. But the doctor also told us that August 15th was only their best estimate. They were actually 95% sure our baby might be born any time between August 1st and September 1st. Confidence intervals are often listed with a percentage, like 90% or 95%, and a range of values, such as between August 1st and Setptember 1st. You can read that as: we are 95% sure your baby will be born between August 1st and September 1st. So, while we get a due date of August 15th, the uncertainty about the exact date is reflected in the confidence interval provided by our doctor.
Of course, we cannot assume that these patterns didn’t simply occur by chance. How confident can we be that the findings presented in the table did not occur by chance? This is where tests of statistical significance come in handy. Statistical significance tells us the likelihood that the relationships we observe could be caused by something other than chance. While your statistics class will give you more specific details on tests of statistical significance and reading quantitative tables, the important thing to be aware of as a non-expert reader of tables is that some of the relationships presented will be statistically significant and others may not be. Tables should provide information about the statistical significance of the relationships presented. When reading a researcher’s conclusions, pay attention to which relationships are statistically significant and which are not.
In Table 3.2, you may have noticed that a p value is noted in the very last column of the table. A p value is a statistical measure of the probability that there is no relationship between the variables under study. Another way of putting this is that the p value provides guidance on whether or not we should reject the null hypothesis. The null hypothesis is simply the assumption that no relationship exists between the variables in question. In Table 3.2, we see that for the first behavior listed, the p value is 0.623. This means that there is a 62.3% chance that the null hypothesis is correct in this case. In other words, it seems likely that any relationship between observed gender and experiencing threats to safety at work in this sample is simply due to chance.
In the final row of the table, however, we see that the p value is 0.039. In other words, there is a 3.9% chance that the null hypothesis is correct. Thus, we can be somewhat more confident than in the preceding example that there may be some relationship between a person’s gender and their experiencing the behavior noted in this row. Statistical significance is reported in reference to a value, usually 0.05 in the social science. This means that the probability that the relationship between gender and experiencing staring or invasion of personal space at work is due to random chance is less than 5 in 100. Social science often uses 0.05, but other values are used. Studies using 0.1 are using a more forgiving standard of significance, and therefore, have a higher likelihood of error (10%). Studies using 0.01 are using a more stringent standard of significance, and therefore, have a lower likelihood of error (1%).
Notice that I’m hedging my bets here by using words like somewhat and may be. When testing hypotheses, social scientists generally phrase their findings in terms of rejecting the null hypothesis rather than making bold statements about the relationships observed in their tables. You can learn more about creating tables, reading tables, and tests of statistical significance in a class focused exclusively on statistical analysis. For now, I hope this brief introduction to reading tables will improve your confidence in reading and understanding the quantitative tables you encounter while reading reports of social science research.
A final caveat is worth noting here. The previous discussion of tables and reading the results section is applicable to quantitative articles. Quantitative articles will contain a lot of numbers and the results of statistical tests demonstrating association between those numbers. Qualitative articles, on the other hand, will consist mostly of quotations from participants. For most qualitative articles, the authors want to put their results in the words of their participants, as they are the experts. The results section may be organized by theme, with each paragraph or subsection illustrating through quotes how the authors interpret what people in their study said.\
- Reading a research article requires reading beyond the abstract.
- In tables presenting causal relationships, the independent variable is typically presented in the table’s columns while the dependent variables are presented in the table’s rows.
- When reading a research report, there are several key questions you should ask yourself for each section of the report.
- Abstract- the short paragraph at the beginning of an article that summarizes the its main point
- Confidence interval- a range of values in which the true value is likely to be
- Null hypothesis- the assumption that no relationship exists between the variables in question
- P-value- a statistical measure of the probability that there is no relationship between the variables under study
- Statistical significance- the likelihood that the relationships that are observed could be caused by something other than chance
- Table- a quick, condensed summary of the report’s key findings
CSAF releases 2009 reading list by Master Sgt. Steven Goetsch public domain
- Green, W. & Simon, B. L. (2012). The Columbia guide to social work writing. New York, NY: Columbia University Press. ↵
- It wouldn’t make any sense to say that people’s workplace experiences cause their gender, so in this example, the question of which is the independent variable and which are the dependent variables has a pretty obvious answer. ↵