OTseeker - Occupational Therapy Systematic Evaluation of Evidence

Tutorial

Introduction to Critical Appraisal

What is a randomised controlled trial?

Is the therapy clinically useful?

Systematic Reviews

Introduction to Critical Appraisal

Unfortunately not all published research is of good quality. Conducting a well-designed research study is difficult—there are many issues to consider during the design phase. Often, even when researchers take a lot of care in designing a study there are many practical and ethical considerations that can make it difficult to conduct a study in a way avoids introducing bias. As well as this often the way in which some research is reported may make it hard for readers to understand what was done.1 Critical appraisal is a process that a reader undertakes to determine the  trustworthiness, impact (or importance) and relevance of a research paper. When critically appraising a study, there are three things to think about: 1,2,3

  1. Internal validity. Internal validity refers to whether the research evidence is trustworthy. According to Guyatt, Sackett & Cook4 it “considers whether the treatment effect reported in the article represents the true direction and magnitude of the treatment effect”. When considering internal validity essentially you want to know to what extent the results were likely to be due to the factor you are interested in rather than some other explanation. Some common alternative explanations that might contribute to the association or effect that is found in a study are: 1) chance, 2) confounding and 3) bias.1,5
    1. One possible explanation for a study’s results is that any findings of differences between two groups are due to random variation (chance). This random variation is less likely when there is an adequate sample size.1,5
    2. Confounding occurs when the factor of interest becomes confused with a confounding factor. A red herring if you like. Confounding variables are usually causally associated with the outcome variable under investigation and non-causally associated with the explanatory variable of interest. They lead to error in the interpretation of what may be an accurate measurement.1,5
    3. Bias is a systematic error in the way that participants are selected for a study, outcomes are measured or data are analysed which, in turn, leads to inaccurate results. Biases can operate in either direction leading to an underestimation or overestimation of the effect reported in a study.1,5,6 Later on in this tutorial a detailed explanation of the types of factors that might introduce bias into randomised controlled trials is discussed.
  2. Impact. Once you have decided that the internal validity of the study is sufficient you need to consider the size of the results of the study. The main thing that you need to determine is the impact or clinical importance of the results. This is different to establishing the statistical significance of results (establishing that a result is unlikely to be attributable to chance).1,7 Clinical significance asks the question: is it worth it? It is sometimes described as the minimum difference that would be important to clients, and therefore takes into account client’s values.
  3. Applicability. This involves evaluating whether the results of the study might apply or be relevant to your client and has to do with the idea of external validity or generalisability of the results. That is, to what extent can we apply the results of the study to people other than the participants of the study?1,4  Another way to approach this question is to ask yourself: Is there a reason why the results wouldn’t apply to my client?4

References:

  1. Hoffmann, T., Bennett, S & Del Mar, C (2013). Introduction to evidence-based practice. In Hoffmann, T., Bennett, S & Del Mar, C (Eds.). Evidence Based Practice across the Health Professions. Sydney: Churchill Livingston.
  2. Sackett DL, Richardson WS, Rosenberg WM, Haynes RB. (1997). Evidence-Based Medicine: How to Practice and Teach EBM. 1st ed. New York: Churchill Livingstone.
  3. Critical Appraisal Skills Programme http://www.casp-uk.net/
  4. Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: A. Are the results of this study valid? JAMA, 270, 2598-2601.
  5. Levin, K (2005). Study design II. Issues of chance, bias, confounding and contamination. Evidence-Based Dentistry, 6, 102–103. doi:10.1038/sj.ebd.6400356
  6. Higgins JPT, Altman DG, Sterne JAC, editors. Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011. Online. Available: http://handbook.cochrane.org/
  7. Fethney, F. (2010). Statistical and clinical significance, and how to use confidence intervals to help interpret both, Australian Critical Care, 23(2), 93-97, http://dx.doi.org/10.1016/j.aucc.2010.03.001.

Critical Appraisal of Randomised Controlled Trials and Systematic Reviews

Randomised controlled trials have the capacity to provide strong evidence about the effectiveness or ineffectiveness of interventions. This is because the methods that they use have the potential to minimise bias for the purposes of establishing treatment effectiveness.1 Systematic reviews of randomised controlled trials synthesise the results of multiple trials providing a stronger sense of the effects of interventions. Unfortunately the literature contains both well performed trials and reviews which draw valid conclusions and trials and reviews that have sufficient bias so as to impact on their conclusions. This tutorial will briefly recap on what randomised controlled trials are, consider in some detail methods to minimise the risk of bias in randomised trials, and then briefly discuss systematic reviews.

What is a randomised controlled trial?

This is a study in which a group of clients is randomly allocated into either an experimental group or a control group. These groups are followed up for the variables / outcomes of interest. Randomised controlled trials potentially offer less bias and more certainty than other study designs that the outcomes being measured are actually due to the experimental treatment condition, rather than other factors.2 More detailed information on randomised controlled trials is provided later in this tutorial.

Are the findings of this trial likely to be valid?

This tutorial is designed to help readers of randomised controlled trials differentiate those trials which are likely to be valid from those that might not be. It also looks briefly at how therapists might use the findings of properly performed studies to make clinical decisions. The approach used here borrows heavily from the "Readers' Guides" first produced by the Department of Clinical Epidemiology and Biostatistics at McMaster University and published in the Canadian Medical Association Journal. The Guides were subsequently revised by the Evidence-Based Medicine Working Group as "Users' Guides" and published in the Journal of the American Medical Association (Guyatt, G.H., & Rennie, D. (1993). JAMA, 270, 2096-2097). The Users Guides are highly recommended as a more detailed source of information on clinical trials and evidence-based practice in general. Citations are given below.

Rigorous answers to questions about treatment effectiveness can be provided by properly designed, properly implemented clinical trials. Unfortunately the literature contains both well performed trials which draw valid conclusions and badly performed trials which draw invalid conclusions; the reader must be able to distinguish between the two. This tutorial describes key features of clinical trials (or "methodological filters") which confer validity.

Some studies that purport to determine the effectiveness of occupational therapy treatments simply assemble a group of participants with a particular condition and take measures of the severity of the condition before and after treatment. If participants improve over the period of treatment, the treatment is said to have been effective. Studies which employ these methods rarely provide satisfactory evidence of treatment effectiveness because it isn’t certain that the observed improvements were due to the treatment, and not to extraneous variables such as natural recovery, statistical regression (a statistical phenomena whereby people become less "extreme" over time simply as a result of the variability in their condition), placebo effects, or the "Hawthorne" effect (where participants report improvements because they think this is what the investigator wants to hear). The only satisfactory way to deal with these threats to the validity of a study is to have a control group. Then a comparison is made between the outcomes of participants who received the treatment and participants who did not receive the treatment.

The logic of controlled studies is that, on average, extraneous variables should act to the same degree on both treatment and control groups, so that any difference between groups at the end of the experiment should be due to treatment. By way of example, it is widely known that most cases of acute low back pain resolve spontaneously and rapidly, even in the absence of any treatment, so simply showing that participants improved with a course of a treatment would not constitute evidence of treatment effectiveness. A controlled trial which showed that treated participants fared better than control participants would constitute stronger evidence that the improvement was due to treatment, because natural recovery should have occurred in both treatment and control groups. The observation that treated participants fared better than control participants suggests that something more than natural recovery was making participants better. Note that, in a controlled study, the "control" group need not receive no treatment. Often, in controlled trials, the comparison is between a control group which receives conventional therapy and an experimental group which receives conventional therapy plus treatment. Alternatively, some trials compare a control group which receives conventional treatment with an experimental group that receives a new therapy.

Five features affecting the internal validity of trials will now be considered:

Random allocation

Importantly, control groups only provide protection against the confounding effects of extraneous variables in so far as treatment and control groups are alike. Only when treatment and control groups are the same in every respect that determines outcome (other than whether or not they get treated) can the experimenter be certain that differences between groups at the end of the trial are due to treatment. In practice this is achieved by randomly allocating the pool of available participants to treatment and control groups. This ensures that extraneous factors such as the extent of natural recovery have about the same effect in treatment and control groups. In fact, when participants are randomly allocated to groups, differences between treatment and control groups can only be due to treatment or chance, and it is possible to rule out chance if the differences are large enough - this is what statistical tests do. Note that this is the only way to ensure the comparability of treatment and control groups. There is no truly satisfactory alternative to random allocation.

Concealment of allocation

The benefits of random allocation may be undone if the implementation of the allocation sequence is poorly handled. If the person who determines whether the participant is eligible for a trial can influence what treatment the participant receives, this can disrupt the randomisation process. Allocation concealment seeks to eliminate selection bias (who gets into the trial and the group they are assigned to). Allocation sequence can be concealed by ensuring the person who generates the allocation sequence is not the person who determines eligibility and entry of participants, and by not using people involved in running the trial to handle the mechanism for treatment allocation. This may be done by using a central telephone randomisation system or by using opaque, sealed envelopes for concealing allocation. For more information on allocation concealment see:
Altman, D. & Schulz, K. (2001). Statistics Notes: Concealing treatment allocation in randomised trials. BMJ, 323, 446-447.

Blinding

Even when participants are randomly allocated to groups, it is necessary to ensure that the effect (or lack of effect) of treatment is not distorted by "observer bias". This refers to the possibility that the investigator’s belief in the effectiveness of a treatment may subconsciously distort the measurement of treatment outcome. The best protection is provided by "blinding" the observer - making sure that the person who measures outcomes does not know if the participant did or did not receive the treatment. It is generally desirable that participants and therapists are also blinded. When participants have been blinded, you can know that the apparent effect of therapy was not produced by placebo or Hawthorne effects. Blinding therapists to the therapy they are applying is often difficult or impossible, but in those studies where therapists are blind to the therapy, you can know that the effects of therapy were not produced by the therapist's enthusiasm with the therapy, rather by the therapy itself.

Follow-up

It is also important that few participants discontinue participation ("drop-out") during the course of the trial. This is because dropouts can seriously distort the study’s findings. A true treatment effect might be disguised if control participants whose condition worsened over the period of the study left the study to seek treatment, as this would make the control group’s average outcome look better than it actually was. Conversely, if treatment caused some participants' condition to worsen and those participants left the study, the treatment would look more effective than it actually was. For this reason dropouts always introduce uncertainty into the validity of a clinical trial. Of course the more dropouts, the greater the uncertainty - a rough rule of thumb is that if more than 15% of participants drop out of a study, the study is potentially seriously flawed. Some authors simply do not report the number of dropouts. In keeping with the established scientific principal of guilty till proven innocent, these studies ought to be considered to be potentially invalid.

Intention to treat analysis

Intention to treat analysis in randomised controlled trials means that each participant’s data are analysed in the groups to which he or she were originally randomly assigned regardless of whether he or she ends up receiving that treatment. Overestimation of clinical effectiveness may occur when an intention to treat analysis isn’t done. The intention to treat analysis maintains the benefits of randomisation. A full application of the intention to treat approach is possible only when complete outcome data are available for all randomised participants. Hollis and Campbell (1999) found a major problem in the application of intention to treat is the inappropriate handling of missing responses producing misleading conclusions. To fully appreciate the potential influence of missing responses, some form of sensitivity analysis is recommended, examining the effect of different strategies on the conclusions.

Further reading on intention to treat analysis can be found:
Hollis, S. & Campbell, F. (1999). What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ, 319, 670-674.

To summarise, the more that clinical trials have the following features, the more certain you can be that the results found are reliable and accurate:

  • Random allocation of participants to treatment and control groups
  • Concealed allocation
  • Blind observers, and preferably participants and therapists as well
  • Few dropouts
  • Intention to treat analysis

The next time you read a clinical trial of an occupational therapy treatment, ask yourself if the trial has these features. As a general rule, those trials which do not satisfy some or all of these criteria don’t constitute strong evidence of treatment effectiveness (or ineffectiveness).
If you want to read further about assessing trial validity, try:

Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: A. Are the results of this study valid? JAMA, 270, 2598-2601.

Links to articles about randomised controlled trials

Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: A. Are the results of this study valid? JAMA, 270, 2598-2601.

Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: B. What were the results and will they help me in caring for my patients? JAMA, 271, 59-6

Is the therapy clinically useful?

How can therapists interpret those trials which appear to be methodologically sound? The message is that it is not sufficient to look simply for evidence of a statistically significant effect of the therapy. You need to be satisfied that the trial measures outcomes that are meaningful, and that the positive effects of the therapy are big enough to make the therapy worthwhile. The harmful effects of the therapy must be infrequent or small so that the therapy does more good than harm. Lastly, the therapy must be cost-effective.

Of course, for a trial to be useful it must investigate meaningful effects of treatment. This means that the outcomes must be measured in a valid way. In general, because we usually judge the primary worth of a treatment by whether it satisfies clients’ needs, measurement outcomes should be meaningful to our clients. Thus a trial which shows that motor training reduces spasticity is less useful than one which shows it enhances functional independence.

Size of the therapy's effect

The size of the therapy's effect is obviously important, but often overlooked. Perhaps this is because many readers of clinical trials do not appreciate the distinction between "statistical significance" and "clinical significance". Or perhaps it reflects the preoccupation of many authors of clinical trials with whether "p < 0.05" or not. Statistical significance ("p < 0.05") refers to whether the effect of the therapy is bigger than can reasonably be attributed to chance alone. That is important (we need to know that the observed effects of therapy were not just a chance finding) but on its own tells us nothing about how big the effect actually was.

The best estimate of the size of the effect of a therapy is the average difference between groups. Thus, if a hypothetical trial on the effects of relaxation reports that back pain, as measured on a 10 cm visual analogue scale (VAS), was reduced by a mean of 4 cm in the treatment group and 1 cm in the control group, our best estimate of the mean effect of treatment is a 3 cm reduction in VAS (as 4 cm minus 1 cm is 3 cm). Another hypothetical trial on home modification advice to prevent falls might report that 10% of clients in the home modification group subsequently had falls, compared to 20% in the control group. In that case our best evidence is that home modification advice reduced the risk of fall by 10% (as 20% minus 10% is 10%). Readers of clinical trials need to look at the size of the reported effect to decide if the effect is big enough to be clinically worthwhile. Remember clients may not be interested in therapies that have only small effects.

Dichotomous outcomes

There is an important subtlety in looking at the size of a therapy's effects. It applies to studies whose outcomes are measured with dichotomous outcomes (dichotomous outcomes can have one of two values, such as dead or alive, injured or not injured, admitted to nursing home or not admitted; this contrasts with variables such as VAS measures of pain, which can have any value between and including 0 and 10). Many studies that measure dichotomous outcomes will report the effect of therapy in terms of ratios, rather than in terms of differences. (The ratio is sometimes called a "relative risk" or "odds ratio" or "hazard ratio", but it comes by other names as well). Expressed in this way, the findings of our hypothetical home modification advice study would be reported as a 50% relative reduction in injury risk (as 10% is half of 20%).

Usually the effect of expressing treatment effects as ratios is to make the effect of the therapy appear large. The better measure is the difference between the two groups. (In fact, the most useful measure may well be the inverse of the difference. This is sometimes called the "number needed to treat" (NNT) because it tell us, on average, how many participants we need to treat to prevent one adverse event - in the home modification example the NNT is 1/0.10 = 10, so one fall may be prevented for every 10 participants who have received home modification advice).

For more information on this two useful papers are:

Herbert, R.D. (2000). Critical appraisal of clinical trials. I: estimating the magnitude of treatment effects when outcomes are measured on a continuous scale. Australian Journal of Physiotherapy, 46, 229-235.
Herbert, R.D. (2000). Critical appraisal of clinical trials. II: estimating the magnitude of treatment effects when outcomes are measured on a dichotomous scale. Australian Journal of Physiotherapy, 46, 309-313.

Confidence intervals

An extra level of sophistication in critical appraisal involves consideration of the degree of imprecision of estimates of effect size offered by clinical trials. Trials are performed on samples of participants that are expected to be representative of certain populations. This means that the best a trial can provide is an (imperfectly precise) estimate of the size of the treatment effect. Clinical trials on large numbers of participants provide better (more precise) estimates of the size of treatment effects than trials on small number of participants. Ideally readers should consider the degree of imprecision of the estimate when deciding what a clinical trials means, because this will often affect the degree of certainty that can be attached to the conclusions drawn from a particular trial. The best way to do this is to calculate confidence intervals about the estimate of the treatment effect size, if these are not explicitly supplied in the trial report.

Interested readers could consult Sim, J. & Reid, N. (1999). Statistical inference by confidence intervals: issues of interpretation and utilization. Physical Therapy, 79, 186-195. A confidence interval calculator is available by clicking here (URL: http://www.graphpad.com/quickcalcs/index.cfm)

Cost-effectiveness

The last part of deciding the usefulness of a therapy involves deciding if the therapy is cost-effective. This is particularly important when health care is paid for, or subsidised, by the public purse. There will never be enough resources to fund all innovations in health care (probably not even all good innovations). Thus the cost of any therapy is that money spent on it cannot be spent on other forms of health care. Sensible allocation of finite funds involves spending money where the effect per dollar is greatest. Of course a therapy cannot be cost-effective if it is not effective. But effective therapies can be cost-ineffective. For more information read:
Drummond, M.F., Richardson, W.S., O'Brien, B.J., Levine, M., & Heyland, D. (1997). User's guide to the medical literature: XIII. How to use an article on economic analysis of clinical practice: A. Are the results of the study valid? JAMA, 277, 1552-1557.

O'Brien, B.J., Heyland, D., Richardson, W.S., Levine, M., & Drummond, M.F. (1997). User's guide to the medical literature: XIII. How to use an article on economic analysis of clinical practice: B. What are the results and will they help me in caring for my patients? JAMA, 277, 1802-1806.

To summarise this section:
Statistical significance does not equate to clinical usefulness. To be clinically useful, a therapy must:

  • affect outcomes that clients are interested in
  • have big enough effects to be worthwhile
  • do more good than harm
  • be cost-effective

If you want to read further on assessing effect size, you could consult:
Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: B. What were the results and will they help me in caring for my patients? JAMA, 271, 59-63.

Other References

  • Cook, D., Guyatt, G., Laupacis, A., Sackett, D., & Goldberg, R. (1995). Clinical recommendations using levels of evidence for antithrombotic agents. Chest, 108 (4Suppl), 2275-305.
  • Fletcher, R. (2002). Evaluation of interventions. Journal of Clinical Epidemiology, 55 (12), 1183-1190.
  • Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: A. Are the results of this study valid? JAMA, 270, 2598-2601.
  • Altman, D. & Schulz, K. (2001). Statistics Notes: Concealing treatment allocation in randomised trials. BMJ, 323, 446-447.
  • Hollis, S. & Campbell, F. (1999). What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ, 319, 670-674.
  • Herbert, R.D. (2000). Critical appraisal of clinical trials. I: estimating the magnitude of treatment effects when outcomes are measured on a continuous scale. Australian Journal of Physiotherapy, 46, 229-235.
    Herbert, R.D. (2000). Critical appraisal of clinical trials. II: estimating the magnitude of treatment effects when outcomes are measured on a dichotomous scale. Australian Journal of Physiotherapy, 46, 309-313.
  • Sim, J. & Reid, N. (1999). Statistical inference by confidence intervals: issues of interpretation and utilization. Physical Therapy, 79, 186-195.
  • Drummond, M.F., Richardson, W.S., O'Brien, B.J., Levine, M., & Heyland, D. (1997). User's guide to the medical literature: XIII. How to use an article on economic analysis of clinical practice: A. Are the results of the study valid? JAMA, 277, 1552-1557.
  • O'Brien, B.J., Heyland, D., Richardson, W.S., Levine, M., & Drummond, M.F. (1997). User's guide to the medical literature: XIII. How to use an article on economic analysis of clinical practice: B. What are the results and will they help me in caring for my patients? JAMA, 277, 1802-1806.
  • Guyatt, G.H., Sackett, D.L., & Cook, D.J. (1993). User's guide to the medical literature: II. How to use an article about therapy or prevention: B. What were the results and will they help me in caring for my patients? JAMA, 271, 59-63.

We gratefully acknowledge that the majority of the content about randomised controlled trials in this tutorial was put together by members of the Centre for Evidence-Based Physiotherapy (http://www.pedro.org.au) with some changes and additions made by the OTseeker team.

Systematic Reviews

What is a systematic review?

Systematic reviews use rigorous methods to locate, assess, and summarise the results of many individual studies in a way that limits bias. In many cases the review summarises primary studies, but does not statistically combine the results. This is sometimes called a qualitative systematic review (not to be confused with qualitative research). A review that statistically combines results of a number of primary studies is often referred to as a meta-analysis.1 Systematic reviews use explicit methods to limit bias in identifying and rejecting studies and therefore their conclusions are usually more reliable and accurate than a narrative or literature review. In brief, systematic reviews:

  • involve a clear definition of eligibility criteria
  • incorporate a comprehensive search to identify all potentially relevant studies (although there are qualitative reviewers that opt for a purposeful sample of papers deemed relevant for generating theory)
  • use explicit, reproducible and uniformly applied criteria in the selection of studies for the review
  • rigorously appraise the risk of bias within individual studies, and
  • systematically synthesise the results of included studies.2

Systematic reviews can be used to synthesise different study methodologies depending on the research question of interest.3  Systematic review about the effects of an intervention most frequently synthesises studies using either only randomised controlled trials, or a mixture of randomised and non-randomised studies. When a review includes nonrandomised studies, it is particularly important to read the full review to identify which types of studies the authors have drawn their conclusions from.4 However systematic reviews may address clinical questions other than effectiveness of interventions such as those about diagnosis, long-term outcomes, or economic factors.3 Additionally, systematic reviews of qualitative research have been undertaken5 with some reviews now also integrating qualitative and quantitative research within the one review.6

Critical appraisal of systematic reviews

How much you can rely on the results of a systematic review will depend on the rigour of the review. There is potential for bias in the methods used in systematic reviews just as there is in other research (although the types of bias differ). It is therefore important that readers are aware of issues of bias for systematic reviews and that they remain aware of the possibility of unjustified conclusions or recommendations.7   The Critically Appraised Skills Program8 has a checklist adapted from a pivotal article on this topic9 that can guide the reader in thinking about the key issues of concern.
Some of the key questions concerning validity of a systematic review include the following questions:

  • Did the review address a clearly focused question?
  • Were important, relevant studies included-that is, was it unlikely that studies were missed?
  • Did the review authors do enough to assess the quality of the included studies?
  • If the results of the review have been combined was it reasonable to do so?

Of course it then important to consider what the results are and how precise they are.
Finally you need to consider the applicability of the results by determining if the results be applied to the local population or to your client.

A link to the full CASP checklist for systematic reviews can be found here. 

A much more detailed checklist has been developed called AMSTAR The Assessment of Multiple Systematic Reviews (AMSTAR) is an 11-item tool with good reported face and content validity that can be used to help readers consider bias within a review.10

The Cochrane Library

One of the key resources for systematic reviews of quantitative studies answering clinical questions about intervention effectiveness is contained in the Cochrane Library and is called the Cochrane Database of Systematic Reviews. Review Groups that are coordinated by The Cochrane Collaboration (www.cochrane.org ) oversee the writing of systematic review by volunteer researchers.1 Each review group has an editorial team which oversees the preparation and maintenance of the reviews, as well as the application of rigorous quality standards. The full text of systematic reviews in the Cochrane Database of Systematic Reviews is available for free in many countries that have access to the Cochrane Library. Another database contained in the Cochrane Library is the Database of Abstracts of Reviews of Effectiveness or DARE. This contains abstracts of systematic reviews that have been quality-assessed. Each abstract includes a summary of the review together with a critical commentary about the overall quality.

Links to articles about systematic reviews

Greenhalgh, T. (1997). How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). BMJ, 315, 672-675.

Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C.,  . . Bouter, L. M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, 10. http://dx.doi.org/10.1186/1471-2288-7-10.

References

  1. Bennett, S., O’Connor, D., Hannes, K. & Doyle, S. (2013). Appraising and understanding systematic reviews of and qualitative evidence. In Hoffmann, T., Bennett, S & Del Mar, C (Eds.). Evidence Based Practice across the Health Professions (2nd ed). Sydney: Churchill Livingston.
  1. Cook, D.J, Mulrow, C.D. & Haynes, R.B. (1997). Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions. Annals of Internal Medicine,126 (5), 376- 80.
  1.  Gough, D., Oliver, J., & Thomas, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1(1), 28. http://dx.doi.org/10.1186/2046-4053-1-28
  1. Bennett, S., Hoffmann, T., McCluskey, A., Coghlan, N., & Tooth, L. (2013). Systematic reviews informing occupational therapy. American Journal of Occupational Therapy, 67, 1–10. http://dx.doi.org/10.5014/ajot.2013.005819
  1. Barnett-Page E & Thomas J (2009). Methods for the synthesis of qualitative research: A critical review. BMC Medical Research Methodology, 5:59.
  1. Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K.,Rees, R., . . . Kavanagh, J. (2004). Integrating qualitative research with trials in systematic reviews. British Medical Journal, 328,1010–1012. http://dx.doi.org/10.1136/bmj.328.7446.1010
  1. Scott, I., Greenberg, P., Poole, P., & Campbell, D. (2006).Cautionary tales in the interpretation of systematic reviews. Internal Medicine Journal, 36, 587–599. http://dx.doi.org/10.1111/j.1445-5994.2006.01140.x

  2. Public Health Resource Unit, England. (2006). Critical Appraisal Skills Programme (CASP): Ten questions to help you make sense of reviews. Retrieved from http://www.casp-uk.net/wp-content/uploads/2011/11/CASP-Systematic-Review-Checklist-31.05.13.pdf

  3. Oxman, A. D., Cook, D. J., & Guyatt, G. H.; Evidence-Based Medicine Working Group. (1994). Users’ guides to the medical literature: VI. How to use an overview. JAMA, 272, 1367–1371. http://dx.doi.org/10.1001/jama.1994.03520170077040

  4. Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C.,  . . Bouter, L. M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, 10. http://dx.doi.org/10.1186/1471-2288-7-10.