What is a Meta-Analysis? What are the benefits and weaknesses of using it?

Question

What will be an ideal response?

Answer 1

Meta-analysis is a quantitative method for identifying patterns in findings across multiple studies of the same research question (Cooper & Hedges, 1994). Unlike a traditional literature review, which describes previous research studies verbally, meta-analyses treat previous studies as cases whose features are measured as variables and are then analyzed statistically. It is similar to conducting a survey in which the respondents are previous studies. Meta-analysis shows how evidence about interventions varies across research studies. If the methods used in these studies varied, then meta-analysis can be used to describe how this variation affected study findings. If social contexts or demographic characteristics varied across the studies, then a meta-analysis can indicate how social context or demographic characteristics affected study findings. Meta-analysis often accompanies systematic reviews that summarize what we know about the effectiveness of a particular intervention. By integrating different study samples and controlling for social context and demographic characteristics, meta-analysis enhances the generalizability of the findings. Meta-analysis is often used to provide systematic reviews to be used for evidence-based policies.
Meta-analysis The quantitative analysis of findings from multiple studies.
Meta-analysis can be used when a number of studies have attempted to answer the same research question with similar quantitative methods. It is not typically used for evaluating results from multiple studies that used different methods or measured different dependent variables. It is also not very sensible to use meta-analysis to combine study results when the original case data from these studies are available and can actually be combined and analyzed together (Lipsey & Wilson, 2001). Rather, meta-analysis is a technique to combine and statistically analyze the statistical findings in published research reports.
After a research problem is formulated about prior research, the literature must be searched systematically to identify the entire population of relevant studies. Typically, multiple bibliographic databases are used; some researchers also search for related dissertations and conference papers. Eligibility criteria must be specified carefully to determine which studies to include and which to omit as too different. Mark Lipsey and David Wilson (2001) suggest that eligibility criteria include the following:
• Distinguishing features. These include the specific intervention tested and perhaps the groups compared.
• Research respondents. These specify the population to which generalization is sought.
• Key variables. These must be sufficient to allow tests of the hypotheses of concern and controls for likely additional influences.
• Research methods. Apples and oranges cannot be directly compared, so some trade-off must be made between including the range of studies about a research question and excluding those that are so different in their methods as not to yield comparable data.
• Cultural and linguistic range. If the study population is going to be limited to English-language publications or in some other way, this must be acknowledged, and the size of the population of relevant studies in other languages should be estimated.
• Time frame. Social processes relevant to the research question may have changed for such reasons as historical events or new technologies, so temporal boundaries around the study population must be considered.
• Publication type. Will the analysis focus only on published reports in professional journals, or will it include dissertations and/or unpublished reports?
Once the studies are identified, their findings, methods, and other features are coded (e.g., sample size, location of sample, strength of the association between the independent and dependent variables). Statistics are then calculated to identify the average effect of the independent variable on the dependent variable as well as the effect of methodological features and other characteristics of the studies (Cooper & Hedges, 1994). The effect size statistic is the key to capturing the association between the independent and dependent variables across multiple studies. The effect size statistic is a standardized measure of association—often the difference between the mean of the experimental group and the mean of the control group on the dependent variable, adjusted for the average variability in the two groups (Lipsey & Wilson, 2001).
Effect size A measure of association—often the difference between the mean of the experimental group and the mean of the control group on the dependent variable, adjusted for the average variability in the two groups.
The meta-analytic approach to synthesizing research results can produce more generalizable findings than those obtained with just one study. Methodological weaknesses in the studies included in the meta-analysis are still a problem; only when other studies without particular methodological weaknesses are included can we estimate effects with some confidence. In addition, before we can place any confidence in the results of a meta-analysis, we must be confident that all (or almost all) relevant studies were included and that the information we need to analyze was included in all (or most) of the studies (Matt & Cook, 1994).
One of the challenges of meta-analysis is that the authors of the articles to be reviewed may not always report sufficient information. For example, the study reports (whether a journal article or unpublished report) may not contain information about participant characteristics, an especially important variable if we are to consider the generalizability of the results to different population groups.