Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review
Background Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.Methods We searched for potentially releva...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMJ Publishing Group
2018-03-01
|
| Series: | BMJ Open |
| Online Access: | https://bmjopen.bmj.com/content/8/3/e019703.full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832542495468158976 |
|---|---|
| author | Julian P T Higgins Joanne E McKenzie Matthew J Page |
| author_facet | Julian P T Higgins Joanne E McKenzie Matthew J Page |
| author_sort | Julian P T Higgins |
| collection | DOAJ |
| description | Background Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.Methods We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.Results We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.Conclusion There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations. |
| format | Article |
| id | doaj-art-1d55fb002bc64959bbb387152b877909 |
| institution | Kabale University |
| issn | 2044-6055 |
| language | English |
| publishDate | 2018-03-01 |
| publisher | BMJ Publishing Group |
| record_format | Article |
| series | BMJ Open |
| spelling | doaj-art-1d55fb002bc64959bbb387152b8779092025-02-04T03:05:09ZengBMJ Publishing GroupBMJ Open2044-60552018-03-018310.1136/bmjopen-2017-019703Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic reviewJulian P T Higgins0Joanne E McKenzie1Matthew J Page2National Institute for Health and Care Research Bristol Biomedical Research Centre, Bristol, UK1 School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australiasenior research fellowBackground Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.Methods We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.Results We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.Conclusion There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.https://bmjopen.bmj.com/content/8/3/e019703.full |
| spellingShingle | Julian P T Higgins Joanne E McKenzie Matthew J Page Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review BMJ Open |
| title | Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review |
| title_full | Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review |
| title_fullStr | Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review |
| title_full_unstemmed | Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review |
| title_short | Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review |
| title_sort | tools for assessing risk of reporting biases in studies and syntheses of studies a systematic review |
| url | https://bmjopen.bmj.com/content/8/3/e019703.full |
| work_keys_str_mv | AT julianpthiggins toolsforassessingriskofreportingbiasesinstudiesandsynthesesofstudiesasystematicreview AT joanneemckenzie toolsforassessingriskofreportingbiasesinstudiesandsynthesesofstudiesasystematicreview AT matthewjpage toolsforassessingriskofreportingbiasesinstudiesandsynthesesofstudiesasystematicreview |