1. Define the domain of research
- By independent variable
- By commonly researched variables.
- By causes and consequences of important variables.
2. Establish criteria for including studies in the review
- Published vs. unpublished study.
- The time period covered in the review.
- Operational definitions of the variables.
- The quality of a study.
3. Determine type of effect size to use.
- Cohen's d
- Pearson's Product Moment or Point Biserial Correlation.
4. Search for relevant studies.
- Computer search.
- Manual search.
- Conference and Technical Symposium Presentations
- Letters to researchers in the area to be studied.
5. Select the final set of studies.
- Do individually.
- Do by more than one individual.
Note inter rater reliabilities will need to be reported.
6. Extract data on variables of interest, sample sizes, effect sizes, reliability
of measurement and other noteworthy characteristics of each study.
when gathering reliability and range departure information, you do
not need to restrict the search to the studies used in the meta-analysis.
- Use all the data when multiple measures are reported.
- Use a subset of the data.
- Average multiple study measures to one outcome measure.
7. Code each study for characteristics that might be related to the effect
size reported in the study.
- Research design factors.
- Sample Characteristics.
- Type of dependent variable.
8. Conduct Reliability checks on the coding procedures.
- With a subset of the data, using 1 to 4 other coders.
- With all the data, using 1 to 4 other coders.
9. When there are multiple measures of independent &/or;dependent variables,
decide whether to group them a priori or not.
- Theoretical diversity among variables.
- Operational measurement diversity among variables.
10. Determine the mean and variance of effect sizes across studies.
- Mean effect size weighted by sample size.
- Calculate Chi Square test for homogeneity.
- Calculate Fail Safe N.
- Between-studies variance in effect size for determining moderator variables.
- Estimation of artifactual sources of between studies variance (sampling
error, attenuation due to measurement error, and/or range restriction)
- Estimation of true between-studies variance.
- Estimation of true mean effect size corrected for measurement and sampling
error, and range restriction.
11. Decide whether to search for moderator variables.
- Significance Test (Chi Square test)
- Amount of between-studies variation that is artifactual.
- Rule of thumb: if the variance accounted
for by the error variance is less than 75% of the variance of the sample
weighted correlations than there may be a moderator variable otherwise
the variation is mainly due to random error (e.g., range restriction,
sampling error, or measurement error.
12. Select Potential Moderators (if warranted).
- Theoretical considerations.
- Operational measurement considerations.
13. Determine the mean and variance of effect sizes within moderator subgroups
--Procedure similar to Step 10.