.
Teaching and Learning Resource (TLR)
1. Title
A Guide to the Critical Reading of Scientific Research Papers
2. Keywords
Abstract, peer review, control, replication, referee, scepticism, interpretation
3. Introduction
Many students (and tutors) find primary research papers difficult to read, and there is often a tendency amongst students to accept uncritically the conclusions of published research. A mature scepticism takes time and practice to develop, but is essential given the provisional nature of scientific knowledge and the frequency of even quite basic errors, along with questionable assumptions and conclusions only partly supported by evidence, in the environmental science literature. However, it can be difficult for students to acquire practice in critically reading primary research papers for at least two reasons. First, many papers are forbiddingly long or complex, and often students and staff rely on review papers (or populist literature such as New Scientist) which provide a carefully 'sanitised' view of research. Second, students defer to the mysterious authority of the journal's selection process, and assume that they could not challenge the decisions of the experts involved. This teaching package aims to address these problems by providing a short, accessible (and fictional) research paper, in applied experimental botany, and asking students to consider the best process by which to assess it, illustrating the basic concepts of controlled experimentation using an example from drug education.
4. Aim
To provide an introduction to reading primary research and to the peer review process
5. Learning outcomes
After using this package, students should be able to:
None required
A summary flowchart of how this TLR is intended for use is given below.:
A Guide to Reading Scientific Research Papers
The times given to each stage depends largely on how the tutor wants to run the TLR, but some suggested times are given here.
Activities 1 and 2 are intended for students to complete on their own, before any seminar or tutorial sessions. Students should take around an hour on this. The subject area of this paper has been chosen as relatively simple and hopefully accessible to readers from other disciplines. If the tutor feels that it is not appropriate, then of course another paper could be chosen and/or written for this exercise. However, the main points which this TLR aims to demonstrate are relavant to all experimental science. Students with no ecological background might benefit from seeing the glossary of terms included with the abstract provided.
Activity 3 could be done as a whole-class activity, perhaps with the tutor asking a few individuals to read out their abstracts, and extracting the essential points (or keywords) from each to compile a list on the board. This can then be compared with the abstract provided. Activity 4 is best done as a group exercise, in groups of perhaps 3-6. The aim is to stimulate discussion of how best to ensure quality control of published information. Groups should produce a list of the main requirements of a fair review process. Tutors might wish to compare the students' lists with usual practice (for example, students might be surprised to learn that papers are usually not refereed anonymously, there are often only 2 referees, there is little training and no payment for referees etc.). Activities 3 and 4 could together form the basis of a 40 minute tutorial.
Activities 5 and 6 could be completed either individually or in groups (5 will take 20-40 minutes, or longer if more criteria are introduced. It could be run as part of the tutorial including stages 3 and 4. Activity 6 needs perhaps another 30 minutes). The material provided discusses controls, replicates and independence of data. Tutors might wish to include other important considerations as additional material depending on the level of the class - some suggestions are given in the box. There are numerous flaws in the manuscript provided. Whilst this is entirely fictional, it is based on a genuine problem in applied ecology which has resulted in much research, and some of the errors in the paper are not uncommon in the published literature. The most egregious flaws include:
Introduction
Methods
Statistics and Results
Discussion
Students' comments should be made on the referees' report forms for the journal Eclectic Ecology, which are enclosed.
Activity 7 is best done as a group exercise, asking who has accepted the paper, and who has recommended its' rejection; it should take at least 30 minutes, and might be done in small groups before bringing the whole class together. Students and tutors could consider the following issues:
What are the main flaws identified by the students? Are there any good points in the design? (such as the long time period considered, which is unusual in such studies, the relatively large scale of the study and the thorough measurements taken). What situations are there when it is acceptable to deviate from the normal experimental procedure? For example, it is often impossible to find an appropriate control when studying a natural system, such as a whole river affected by a pollution discharge (or, for that matter, the whole planet affected by greenhouse gases). There might be ethical reasons not to use controls in medical trials (e.g. treating patients with HIV). There could be dispute over what constitutes a genuinely independent treatment - in the example given, just how far away from each other do the replicate exclosures need to be? If they are too close, then they might be non-independent (perhaps underground rhizomes from some of the plants extend from one replicate to the next?). If they are too far away, they might be very different from each other at the start of the experiment.
Follow the guidelines in the enclosed materials. Activities required
(other than reading the main sheet) are marked with an
...
The tasks required are listed below:
9. Stimulus Material
All necessary materials are provided. They consist of a bogus, shortened research paper (The Effects of Grazing on Woodland Regeneration in the Highlands), including 3 figures and 2 tables, the guidelines for referees and the report forms from the (imaginary) Journal of Eclectic Ecology, to which the manuscript has been submitted, and the Introduction to Reading Scientific Papers.
10. Degree stage
This TLR could be used at second or third levels, depending on the scientific background of the students.
11. Resource requirements
None.
12. Preparation
Students need to read the enclosed research paper, and prepare their abstracts
The aims and/or learning outcomes of this TLR are related to those of other TLRs listed in the following 'thematic clusters':
14. Follow-up activities
A similar exercise using an actual published paper, with less obvious flaws.
Barnard, C., Gilbert, F. and McGregor, P. (1993). Asking Questions in Biology -Design, Analysis and Presentation in Practical Work. Longman, Harlow.
16. Users' comments
“[The TLR’s aims and learning outcomes were] highly appropriate since most students would tend to take at face value what they read in academic journals ... We are also sometimes in danger of assuming too much of our students in relation to their development of critical reading skills i.e. we somehow expect them to pick it up almost by ‘osmosis’.”
“The text information was very helpful, particularly Appendix 3 on reading scientific papers. However, I found it necessary to summarise this information in ‘lecture’ form prior to the suggested activities.”
“I do wonder if the TLR is a little ambitious in the amount of ground it aims to cover by also including concepts related to experimental design ... our own students would have benefited from ... a separate TLR ... devoted to such an important area and it might have given the ‘critical reading’ emphasis a sharper focus.”
“Whilst I found all the resources included in the TLR very helpful I would in future consider using an alternative text for students to evaluate (or perhaps a range of texts of their own choice) that were directly related to projects in which they were currently involved ... I would also consider asking students to ‘peer review’ some of their own assessed work as a follow up activity.”
APPENDIX 1: Research paper
The Effects of Grazing on Woodland Regeneration in the Highlands
Introduction
Grazing by large herbivores and smaller herbivorous rodents is one of the most significant factors affecting the vegetation dynamics of many British woodlands (Pigott, 1985; Mitchell and Kirby, 1991). Continuous grazing can change the species composition of a site, by eliminating sensitive species and allowing the competitive release of resistant species, which may be undesirable from a conservation or management perspective. It can also alter the physical structure of the vegetation, causing selection for prostrate rather than erect growth form amongst the ground flora and often removing or reducing the shrub layer up to a distinct browse line of 1.5-2m. Impacts can extend to ecosytem-level effects, such as changes in nutrient cycling, productivity, susceptibility to fire and water retention (Crawley, 1983).
The prescription for suitable management measures for particular sites requires that the effects of the different species of grazers on vegetation dynamics be understood. Although many studies have considered grazing in woodlands, few have separated the impacts of different species of herbivore. This study uses an experimental exclosure approach to investigate how vegetation changes after it is relieved from grazing pressure by large herbivores and by rodents.
Caledonian woodland, dominated by Scots pine, Pinus sylvestris, once covered much of Scotland, but is now reduced to only 1% of Scotland's land area (Forestry Commission, 1983). One of the most important remaining fragments (around 858 hectares) occurs in Glen Affric, Invernesshire. In common with most of the native Scottish woodlands, the Glen Affric woods are subjected to grazing from a wide range of animals, both wild (deer, hares, voles) and domestic (sheep). Excessive grazing has been a particular problem in the Highlands since the 18th century. In unenclosed sites at Glen Affric this grazing prevents the regeneration of sensitive species, so that many woodlands have a curiously impoverished age structure with most trees being well over 50 years old. The area thus provides an ideal study site to examine the effects of relief of grazing in an area markedly affected by a long history of heavy animal usage.
Materials and Methods
In 1970, two adjacent areas of mixed woodland (total area 14 hectares) were enclosed with deer fencing by local conservation managers to create two separate plots of approximately equal area (Fig.1). We used these plots as our large vertebrate exclosures. Six smaller exclosures, each 3x3 meters length and 0.5m tall, with chickenwire lids, were established at random points within the larger exclosures in 1976. Each consisted of 1cm diameter chicken-wire mesh stretched around 4 corner posts, with the mesh dug 40cm into the soil. These provided total exclosures of all herbivores, including voles, mice and rabbits. Three nearby woodlands of similar size were chosen as control sites (Fig. 1).
The plots were surveyed six years after enclosure in 1976 to establish a baseline and then surveyed in July 1984 and July/August 1992. In each case vegetational differences were recorded between the large herbivore exclosures, small herbivore exclosures and grazed (control) plots, and changes over time within each plot assessed.
At each sample date, a 50m grid was established within the large exclosure and control plots by recognised survey techniques. All vegetation measurements were replicated within 20 randomly chosen meter square quadrats within the grid square, providing 20 replicates in each plot. Where measurements taken did not extend to the full 2500 m2 square but represented a sample only, the positions of the samples were located randomly within the grid square. Sample locations were not constant from survey to survey. At each survey data, a randomly chosen 1 m2 quadrat within each of the small herbivore exclosures was also surveyed. The following parameters were recorded during each survey:
Trees
Tree numbers and species composition were recorded in each of the quadrats grids. Basal diameters of all trees encountered in each grid square were also recorded. Numbers, species composition and basal diameters were taken for all the trees within the whole area of each small herbivore exclosure.
Shrub and herb layer
Quadrats (1m2) were randomly located in each 50 x 50m grid square in each survey, giving on each occasion a total of 20 quadrats within each plot, and a single randomly placed quadrat in each of the small herbivore exclosures. All species rooted within each quadrat were scored on a presence/absence basis and percentage cover of each estimated. Data could then be pooled to provide estimates of frequency of occurrence (no. of quadrats in which a species was recorded out of a total possible 20 for large herbivore exclosures and controls, and 6 for small herbivore exclosures) and mean percentage cover. Above-ground biomass of herbaceous vegetation was estimated for each quadrat by clipping all plant material at ground level and fresh-weighing the harvested sample.
Statistical tests
Chi-squared tests were used to investigate differences in the size-frequency distributions of trees in large herbivore exclosure, small herbivore exclosure and control plots.
Differences in the mean species richness of trees and shrubs, and the mean biomass of ground vegetation between treatments, and within treatments between sampling times, were tested using analysis of variance.
Results
Trees
In 1976, there were signs of regeneration within the exclosed plots, with a number of trees measured of < 10cm basal diameter . If these are ignored, and only the larger trees are considered, the survey provides a reasonable picture of the vegetation structure when the exclosures were established. The most common of the larger trees were Scots pine, birch (Betula pubescens) and Larch (Larix decidua). It is clear from this data that the woodlands must have had very similar structure before the exclosures were erected.
By 1984 many new trees, mostly birch, had established in the large and small herbivore exclosures; the mean density of saplings within the plots was 340 ha-1. The mean density of saplings in control plots was 12 ha-1.
In 1992 there was little change in the control plots, but further regeneration and growth had occured in the exclosures (Fig. 2). There was a highly significant difference between the size frequency distributions of the three commonest tree species in the three plot types (P. sylvestris: X2 = 52, d.f. = 6, P< 0.001; B. pubescens: X2 = 56, d.f. = 2, P< 0.001; Larix decidua: X2 = 47, d.f. = 2, P< 0.001), although the chi-squared cells showed little difference between the distrubutions in the large herbivore and small herbivore exclosures. Small numbers of other species of tree (principally rowan, Sorbus aucuparia, alder Alnus glutinosa, and silver birch, Betula pendula) had established in the exclosure plots, although the mean species richness was higher in the large than small herbivore exclosures (Table 1). Species richness of trees increased significantly in both types of exclosure between 1976 and 1992 (Table 1).
Herb and Shrub Layer
Although no detailed survey of the vegetation in the exclosure plots was carried out when they were first established in 1970, we can gain some impression of what the shrub and herb layers must have been like at this time by reference to the control sites. These are heavily browsed and typically consist of mostly heather (Calluna vulgaris), the grasses Molinia caerulea and Nardus stricta and patches of bracken (Pteridium aquilinum). By 1992 mean species richness and biomass were significantly higher in the large exclosures than in the small herbivore exclosures and the controls (Table 2, Fig. 3).
Discussion
Previous studies have shown that grazing on the Scottish highlands has a major impact on the plant community. For example, King (1960) demonstrated that grazing by sheep and deer allowed the invasive grass Nardus a competitive advantage over heather, and resulted in increasing areas of hillside being covered by this undesirable species. Throughout the highlands, high stocking levels of sheep and red deer are held responsible for the prevention of the regeneration of natural woodlands. This paper supports this view. The exclosures which prevented grazing by large herbivores allowed seedlings to establish, possibly for the first time in decades. The exclosures also increased the species richness of trees, herbs and shrubs, allowing less common species, such as alder, a chance to establish.
Few studies have considered the impacts of smaller herbivores on highland vegetation. The lack of any significant difference between size-frequency distributions of trees in the large and small exclosures show that small herbivores were not having a negative impact on tree growth. Surprisingly, these results show that species richness of both trees and herbs was lower in the small herbivore exclosures, suggesting that small herbivores actually encourage the growth of many species. This might be due to these herbivores reducing the sward height, thus allowing light to penetrate and seedlings to establish. This area deserves further study.
This paper has demonstrated the effectiveness of large herbivore exclosures in encouraging the growth of trees and herbs . Other means of reducing large herbivore numbers, such as importing wolves, changing the structure of agricultural subsidies or large scale culls, have proved effective elsewhere, but are impossible in the highlands. We therefore recommend widespread use of large herbivore exclosures, and efforts to encourage grazing by small herbivores.
References
Crawley, M.J. (1983). Herbivory: The Dynamics of Animal-Plant Interactions. Blackwell Scientific Publications, Oxford.
Edwards, P.J. & Gillman, M. (1987). Herbivores and plant succession. In Colonization: Succession and Stability, ed. by A.J. Gray, M.J. Crawley & P.J. Edwards. Blackwell Scientific Publications, Oxford, pp.295-314.
Forestry Commission (1983). Census of Woodlands and Trees 1979-82. HMSO, Edinburgh.
King, J. (1960). Observations on the seedling establishment and growth of Nardus stricta in burned Callunetum, Journal of Ecology, 48, pp.667-77.
Pigott, C.D. (1985). Selective damage to tree seedlings by voles Clethrion glareolus. Oecologia Berl., 67, pp.367-71.
APPENDIX 2: Guidelines for referees, the report forms and abstract of the article "The Effects of Grazing on Woodland Regeneration in the Highlands"
Journal of Eclectic Ecology
Edinburgh, London and New York
Guidelines for Referees
The Journal of Eclectic Ecology publishes original contributions in any area of research in applied ecology and conservation. The journal has a particular interest in work which is applied and practical in nature and the perspective is both national and international. The journal also publishes 'Points of View' contributions where authors are encouraged to generate debate and discussion on any issue of current interest to ecologists.
The Journal of Eclectic Ecology aims to publish material of high academic standards. All articles are routinely reviewed by two referees. The referees' task is to a) advise on the academic standards of the paper and b) suggest how weaknesses may be resolved, if possible. Whilst referees should be as constructive as possible, they are reminded that The Journal of Eclectic Ecology can only accept around 75% of the manuscripts submitted. Specific strengths and weaknesses should be outlined. No 'checklist' of points to cover is provided as articles vary considerably in their approach, but among the issues that should be mentioned are significance and originality of the work, the appropriateness for applied ecological research, methodological strengths and weaknesses, the specific adequacy/inadequacy of each section of the paper, the quality of writing (organisation, clarity, style, etc.) and the quality and relevance of the discussion.
Your comments would be appreciated on the referee's report form provided. Please use continuation sheets if necessary. In addition to general comments, please indicate if you think the paper should be accepted by completing the summary rating and recommendation sections.
Abstract of "The Effects of Grazing on Woodland Regeneration in the Highlands"
Two 7 hectare deer exclosures were established in 1970 within Pinus sylvestris woodland in Glen Affric, Invernesshire. In 1976, small herbivore (rabbits, voles and mice) exclosures were established within the large herbivore exclosures, and the vegetation of these exclosures, and of three control sites, was surveyed in 1976, 1985 and 1992.
Changes over time in species composition and age structure of trees in the three areas are discussed. Changes in species compostion and biomass of the ground flora are also considered. Highly significant differences are recorded between the grazed sites and the large grazer exclosures. Regeneration of trees, particularly Pinus sylvestris, Betula pubescens and Larix decidua, occurred in both large and small herbivore exclosures, and species richness of all trees increased, leading to significant changes in these plots over the course of the experiment. No regeneration was recorded in control sites.
Species richness and total biomass of ground flora also increased in the exclosures compared with the controls. However, richness and biomass were not as high in the small herbivore exclosures as in the large herbivore exclosures, suggesting that whilst large herbivores suppress regeneration and diversity of plants, small herbivores may play a role in increasing rates of recovery from intensive grazing.
Grazing, regeneration, exclosure, woodland, herbivore, highlands
Analysis of variance - a statistical test for the differences between two or more means.
Biomass - the total weight of all the living material in a given sample.
Chi-squared test - a statistical test for differences between proportions (for example, between two size-frequency distributions).
Competitive release - the growth of a previously supressed species once one or more of its' competitors is removed or reduced in numbers, often by a predator.
Exclosure - an area, often within a cage or fence, from which an experimental organism or organisms are excluded.
Size-frequency distribution - the numbers or proportions of a given sample that occur within each of a number of size categories (for example, 10% are less than 1m, 80% are 1-2m and 10% are more than 2m).
Species richness - the number of different species in a given sample or area.
Sward - the ground vegetation, especially grasses and herbs.
APPENDIX 3: An Introduction to Reading Scientific Papers
Student
Action - read the manuscript The Effects of
Grazing on Woodland Regeneration in the Highlands submitted for
publication to The Journal of Eclectic Ecology, and write a 200
word abstract summarising the paper.
Scientific journals publish a number of different types of papers. For example, review papers give an overview of the results of research in a particular area, whilst opinion and commentary pieces reflect the author's view of a subject. However, the most important category of scientific paper is that which reports the results of original research work, often performed as a manipulative experiment. It is this category of paper which is considered here.
By the time a paper is published, it has usually been through a process of quality control, called peer review (see below). However, this does not mean it will be perfect. Many published papers contain factual and statistical errors, and statements which are really assertions disguised as objective fact. Given this, and the fact that scientific knowledge is often provisional and changing, the proper attitude when reading scientific papers is one of scepticism; it is the task of the reader to judge whether the authors' conclusions are justified. This package describes the main components of a scientific paper, and provides an outline of how to act as a critical reader of science.
What's in a paper?
In order to make them instantly recognised and easily understood, research papers always follow a standard format, each part of which has a specific function. The main parts are described below:
The Abstract - This should summarise, usually in less than 300 words, the area of interest, methodology and principal results of the work. It should let the reader get a good impression of what the paper contains.
The Introduction - This reflects the planning of the research project. It describes the state of knowledge in the relevant area, with reference to work already published, and shows why the work in the paper was necessary. It should clearly state what hypotheses are being tested in the paper, and why the chosen research method is appropriate.
The Methods - This describes how the research was carried out. It should cover everything relevant to the actual experimental procedure, and also how the data collected was analysed. An important criterion when assessing the methods section is to ask, 'does the author(s) give enough information to allow me to repeat the experiment?' If the answer to this is no, then the methods section is not detailed enough.
The Results - This represents a summary and analysis of the data. Usually graphs and tables will be included here, if they make presentation of the data clearer. Note that the results section should simply present the results of the work described, without discussing them.
The Discussion - This is the most 'open' section of the paper, where the author(s) draws conclusions from the work described. Have the initial aims of the investigation been achieved? Have the hypotheses of interest been tested? How do the results fit-in with other people's work, and what further work needs to be done? Discussion sections often contain speculations and generalizations. This is fine, as long as the author(s) is careful to point out what conclusions his research can definitely support, and what is more speculative.
Peer Review
Before a paper is accepted, it must be submitted for peer review. This involves a small number of people (the referees) not involved in the research project reading the paper and deciding if it is suitable for publication. The editors of the journal are usually also involved in the reviewing process.
The peer review provides the crucial quality control for scientific papers, and has important implications. If the referees are too harsh, and reject a paper which has merit, years of work might go unreported, and the scientific community might be denied useful information. The career of the researcher might also be damaged. On the other hand, if the editors accept a flawed paper, this can reduce the reputation of the journal and lead to embarrassing revelations (such as the recent case of a physicist fabricating a deliberately irrational sociology paper, which was accepted by a leading journal). The review process must be above any suspicion of bias (for example, that well known researchers always have their work accepted because of their reputations, or that referees have rejected a paper because it happens to contradict their own beliefs about a subject). So what is the best way for peer reviewing to operate?
Design
the ideal peer review procedure
Produce a list of the requirements for an ideal procedure. You might want to consider the following questions:
The Essentials
Although the methods and technology used in scientific experiments might vary tremendously, from particle accelerators in physics to chainsaws in ecology, the basic design of an experiment should, where possible, follow a few simple rules. Here is a description of 'the essentials':
Drugs Education - does it work? Many millions of pounds are spent on health education programmes every year. But do these campaigns actually work? The only way to find out is to adopt an experimental approach. Suppose you were asked to determine if a new way of teaching about the dangers of drug abuse to teenagers had an effect. You could set up an experiment as follows:
What is wrong with this design? Consider how you would interpret the results you might get:
| Result | Interpretation |
| Percentage using drugs goes down | 1. The education has been effective. 2. Something else (e.g. a highly publicised death, like that of Leah Betts) has changed behaviour. 3. There is bias e.g. students are lying. |
| Percentage using drugs goes up | 1. Education has encouraged drug use. 2. Education has discouraged drug use, without it, even more students would be using drugs. |
So the experiment cannot tell you anything useful, because you could interpret the results in many ways. The first thing you need to do to improve the design is to include a control as well as a treatment group, that is a group which does not receive the treatment, to act as a comparison.
Write
down now four characteristics of a good control group for this experiment
Now consider your interpretation of the possible outcomes:
| Result | Interpretation |
| Drug use in the treatment group goes down, drug use in the control group stays the same | 1. The education has been effective. 2. Some other factor has acted to reduce drug use in the treatment, but not control group (e.g. perhaps one member of the treatment group was arrested, thus scaring his friends off drugs). |
| Drug use in treatment and control groups stays the same | 1. The education has been ineffective. 2. Some factor would have caused the treatment group to increase their consumption of drugs, but they did not do so because of the education. |
Although the design is now better, it still cannot answer your question. This is partly because the question itself is too broad. It is much harder to answer the question "Does this intervention have an effect on illegal drug use?" than to address the more focused question, "Does this intervention reduce illegal drug use?" In general, it is best to make sure that the hypothesis you are testing (in this case, that this education reduces drug use in teenagers) is as focused as possible.
The other main flaw is the lack of replication, both of treatments and controls. How do we know that the groups we have chosen are representative? If the treatment group happened to have a charismatic teacher, drug abuse might reduce because of this, and not because of the particular educational technique of interest. However, if we had ten treatment and control groups, chosen randomly, it is most unlikely that the treatment groups will all happen to have particularly charismatic teachers, and the control group particularly poor ones. So this would be a much better design:
Increasing the number of replicates will make the experiment more convincing, but will increase the costs and work involved. One way round this might be simply to subdivide the treatment and control groups. For example, if all the students in a class (rather than the class itself) were regarded as replicates, a single class would provide perhaps 20 replicates.
Write
down what is wrong with this approach.
Finally, a good experimental design will ensure that the replicates (of treatments and controls) are independent. Consider running your experiment in a single large school, with five classes as treatments and five as controls. It is likely that students in the control groups will have sisters, brothers and friends in the treatment groups, and will therefore hear from them details of the education that the treatment groups are receiving. This could affect the behaviour of students in the control groups, making the experiment invalid (for example, if the control-group students reduced their drug use because what they heard from their friends was so effective, the experimental results would suggest that the education was not effective, since there would be no, or little, difference between control and treatment groups).
