Citation: Kara Schick-Makaroff, Marjorie MacDonald, Marilyn Plummer, Judy Burgess, Wendy Neander. What Synthesis Methodology Should I Use? A Review and Analysis of Approaches to Research Synthesis.[J]. AIMS Public Health, 2016, 3(1): 172-215. doi: 10.3934/publichealth.2016.1.172
[1] | David V. McQueen, Erma Manoncourt, Yuri N. Cartier, Irina Dinca, Ülla-Karin Nurm . The Transferability of Health Promotion and Education Approaches Between Non-communicable Diseases and Communicable Diseases—an Analysis of Evidence. AIMS Public Health, 2014, 1(4): 182-198. doi: 10.3934/publichealth.2014.4.182 |
[2] | Chan-Young Kwon, Boram Lee . The efficacy of acupuncture on suicidal behavior: A protocol for systematic review and meta-analysis. AIMS Public Health, 2022, 9(4): 651-660. doi: 10.3934/publichealth.2022046 |
[3] | Jessie-Lee McIsaac, Grace Warner, Logan Lawrence, Robin Urquhart, Sheri Price, Jacqueline Gahagan, Mary McNally, Lois A Jackson . The application of implementation science theories for population health: A critical interpretive synthesis. AIMS Public Health, 2018, 5(1): 13-30. doi: 10.3934/publichealth.2018.1.13 |
[4] | Afeez Abiola Hazzan, Joyce O Hazzan, Mark Oremus . Measuring quality and level of care provided by family caregivers of persons with dementia: protocol for a systematic review of validated instruments. AIMS Public Health, 2021, 8(2): 206-212. doi: 10.3934/publichealth.2021016 |
[5] | Wanda Martin, Bernie Pauly, Marjorie MacDonald . Situational Analysis for Complex Systems: Methodological Development in Public Health Research. AIMS Public Health, 2016, 3(1): 94-109. doi: 10.3934/publichealth.2016.1.94 |
[6] | Mehreen Tariq, Margaret Haworth-Brockman, Seyed M Moghadas . Ten years of Pan-InfORM: modelling research for public health in Canada. AIMS Public Health, 2021, 8(2): 265-274. doi: 10.3934/publichealth.2021020 |
[7] | Karen E. Lamb, Lukar E. Thornton, Ester Cerin, Kylie Ball . Statistical Approaches Used to Assess the Equity of Access to Food Outlets: A Systematic Review. AIMS Public Health, 2015, 2(3): 358-401. doi: 10.3934/publichealth.2015.3.358 |
[8] | Siti Roshaidai Mohd Arifin, Helen Cheyne, Margaret Maxwell . Review of the prevalence of postnatal depression across cultures. AIMS Public Health, 2018, 5(3): 260-295. doi: 10.3934/publichealth.2018.3.260 |
[9] | Kai Cheng, Jiangtao Wang, Jian Liu, Xiangsheng Zhang, Yuanyuan Shen, Hang Su . Public health implications of computer-aided diagnosis and treatment technologies in breast cancer care. AIMS Public Health, 2023, 10(4): 867-895. doi: 10.3934/publichealth.2023057 |
[10] | Joyce Lo, Sharan Jaswal, Matthew Yeung, Vijay Kumar Chattu, Ali Bani-Fatemi, Aaron Howe, Amin Yazdani, Basem Gohar, Douglas P. Gross, Behdin Nowrouzi-Kia . A systematic review of the literature: Gender-based violence in the construction and natural resources industry. AIMS Public Health, 2024, 11(2): 654-666. doi: 10.3934/publichealth.2024033 |
List of Abbreviation (in Additional File 1)
CIS: Critical Interpretive Synthesis
GFT: Grounded Formal Theory
M-A: Meta-Analysis
MNS: Meta-Narrative Synthesis
SR: Systematic review
Since the turn of the century, public health emergencies have been identified worldwide, particularly related to infectious diseases. For example, the Severe Acute Respiratory Syndrome (SARS) epidemic in Canada in 2002-2003, the recent Ebola epidemic in Africa, and the ongoing HIV/AIDs pandemic are global health concerns. There have also been dramatic increases in the prevalence of chronic diseases around the world [1,2,3]. These epidemiological challenges have raised concerns about the ability of health systems worldwide to address these crises. As a result, public health systems reform has been initiated in a number of countries. In Canada, as in other countries, the role of evidence to support public health reform and improve population health has been given high priority. Yet, there continues to be a significant gap between the production of evidence through research and its application in practice [4,5]. One strategy to address this gap has been the development of new research synthesis methodologies to deal with the time-sensitive and wide ranging evidence needs of policy makers and practitioners in all areas of health care, including public health.
As doctoral nursing students facing a review of the literature for our dissertations, and as a faculty member teaching a research methods course, we encountered several ways of conducting a research synthesis but found no comprehensive resources that discussed, compared, and contrasted various synthesis methodologies on their purposes, processes, strengths and limitations. To complicate matters, writers use terms interchangeably or use different terms to mean the same thing, and the literature is often contradictory about various approaches. Some texts [6,7,8,9] did provide a preliminary understanding about how research synthesis had been taken up in nursing, but these did not meet our requirements. Thus, in this article we address the need for a comprehensive overview of research synthesis methodologies to guide public health, health care, and social science researchers and practitioners.
Research synthesis is relatively new in public health but has a long history in other fields dating back to the late 1800s. Research synthesis, a research process in its own right [10], has become more prominent in the wake of the evidence-based movement of the 1990s. Research syntheses have found their advocates and detractors in all disciplines, with challenges to the processes of systematic review and meta-analysis, in particular, being raised by critics of evidence-based healthcare [11,12,13].
Our purpose was to conduct an integrative review of the literature to explore the historical, contextual, and evolving nature of research synthesis [14,15]. We synthesize and critique the main approaches to research synthesis that are relevant for public health, health care, and social scientists. Research synthesis is the overarching term we use to describe approaches to combining, aggregating, integrating, and synthesizing primary research findings. Each synthesis methodology draws on different types of findings depending on the purpose and product of the chosen synthesis (see Additional File 1).
Based on our current knowledge of the literature, we identified these approaches to include in our review: systematic review, meta-analysis, qualitative meta-synthesis, meta-narrative synthesis, scoping review, rapid review, realist synthesis, concept analysis, literature review, and integrative review. Our first step was to divide the synthesis types among the research team. Each member did a preliminary search to identify key texts. The team then met to develop search terms and a framework to guide the review.
Over the period of 2008 to 2012 we extensively searched the literature, updating our search at several time points, not restricting our search by date. The dates of texts reviewed range from 1967 to 2015. We used the terms above combined with the term “method* (e.g., “realist synthesis” and “ethod*) in the database Health Source: Academic Edition (includes Medline and CINAHL). This search yielded very few texts on some methodologies and many on others. We realized that many documents on research synthesis had not been picked up in the search. Therefore, we also searched Google Scholar, PubMed, ERIC, and Social Science Index, as well as the websites of key organizations such as the Joanna Briggs Institute, the University of York Centre for Evidence-Based Nursing, and the Cochrane Collaboration database. We hand searched several nursing, social science, public health and health policy journals. Finally, we traced relevant documents from the references in obtained texts.
We included works that met the following inclusion criteria: (1) published in English; (2) discussed the history of research synthesis; (3) explicitly described the approach and specific methods; or (4) identified issues, challenges, strengths and limitations of the particular methodology. We excluded research reports that resulted from the use of particular synthesis methodologies unless they also included criteria 2, 3, or 4 above.
Based on our search, we identified additional types of research synthesis (e.g., meta-interpretation, best evidence synthesis, critical interpretive synthesis, meta-summary, grounded formal theory). Still, we missed some important developments in meta-analysis, for example, identified by the journal's reviewers that have now been discussed briefly in the paper. The final set of 197 texts included in our review comprised theoretical, empirical, and conceptual papers, books, editorials and commentaries, and policy documents.
In our preliminary review of key texts, the team inductively developed a framework of the important elements of each method for comparison. In the next phase, each text was read carefully, and data for these elements were extracted into a table for comparison on the points of: key characteristics, purpose, methods, and product; see Additional File 1). Once the data were grouped and extracted, we synthesized across categories based on the following additional points of comparison: complexity of the process, degree of systematization, consideration of context, underlying assumptions, unit of analysis, and when to use each approach. In our results, we discuss our comparison of the various synthesis approaches on the elements above. Drawing only on documents for the review, ethics approval was not required.
We identified four broad categories of research synthesis methodology: Conventional, quantitative, qualitative, and emerging syntheses. From our dataset of 197 texts, we had 14 texts on conventional synthesis, 64 on quantitative synthesis, 78 on qualitative synthesis, and 41 on emerging syntheses. Table 1 provides an overview of the four types of research synthesis, definitions, types of data used, products, and examples of the methodology.
Types of Research Synthesis | Definition | Data Types Used | Products | Examples |
1. Conventional Synthesis | Older forms of review with less-systematic examination, critique, and synthesis of the literature on a mature topic for re-conceptulization or on a new topic for preliminary conceptualization | · Quantitative studies | · Narrative expression and summary | · Integrative review [14,18,19,20,21] |
· Qualitative studies | · Tables, charts, graphical displays, diagrams and maps | · Narrative synthesis [16] | ||
· Other types of data e.g., theoretical literature, policy | · Theory, theoretical/conceptual frameworks, or conceptual maps | · Conventional literature review[17] | ||
2. Quantitative Synthesis | Combining, aggregating, or integrating quantitative empirical research with data expressed in numeric form | · Quantitative studies | · Narrative expression and summary | · Systematic review [13,26,27] |
· Mathematical scores | · Meta-analysis [22,23,24,25] | |||
· Statements of generalizability | · Best evidence synthesis [28,29,30] | |||
3. Qualitative Synthesis | Combining, aggregating, or integrating qualitative empirical research and/or theoretical work expressed in narrative form | · Qualitative studies | · Narrative expression and summary | · Meta-synthesis [31,32,33,34,35,36,37,38,39,40,41,42,43,44] |
· Other types of data e.g., theoretical literature | · Theory, theoretical/conceptual frameworks, or conceptual maps | · Concept analysis [45,46,47] | ||
· A definition | · Grounded formal theory [37,48,49,50,51,52] | |||
· Meta-study [31,37,48,49,50,53,54] | ||||
· Meta-analysis [37,38,55,56,57,58] | ||||
· Meta-interpretation [59] | ||||
· Meta-ethnography [49,50,60,61,62] | ||||
4. Emerging Synthesis | Newer syntheses that provide a systematic approach to synthesizing varied literature in a topic area that includes diverse data types | · Quantitative studies | · Narrative expression and summary | · Scoping review [63] |
· Qualitative studies | · Tables, charts, graphical displays, diagrams and maps | · Rapid review [64,65] | ||
· Other types of data e.g., theoretical work, grey literature, editorials, commentaries, policy, evaluations | · Mathematical scores | · Rapid realist review [66] | ||
· Theory, theoretical/conceptual frameworks, or conceptual maps | · Meta-narrative synthesis [67,68] | |||
· A report written for decision-makers | · Realist synthesis [68,69] | |||
· Meta-summary [70] | ||||
· Critical interpretive synthesis [71,72,73,74] | ||||
· Other types of mixed-research synthesis [49,72,75,76,77,78,79] |
Although we group these types of synthesis into four broad categories on the basis of similarities, each type within a category has unique characteristics, which may differ from the overall group similarities. Each could be explored in greater depth to tease out their unique characteristics, but detailed comparison is beyond the scope of this article.
Additional File 1 presents one or more selected types of synthesis that represent the broad category but is not an exhaustive presentation of all types within each category. It provides more depth for specific examples from each category of synthesis on the characteristics, purpose, methods, and products than is found in Table 1.
Here we draw on two types of categorization. First, we utilize Dixon Woods et al.'s [49] classification of research syntheses as being either integrative or interpretive. (Please note that integrative syntheses are not the same as an integrative review as defined in Additional File 1.) Second, we use Popay's [80] enhancement and epistemological models.
The defining characteristics of integrative syntheses are that they involve summarizing the data achieved by pooling data [49]. Integrative syntheses include systematic reviews, meta-analyses, as well as scoping and rapid reviews because each of these focus on summarizing data. They also define concepts from the outset (although this may not always be true in scoping or rapid reviews) and deal with a well-specified phenomenon of interest.
Interpretive syntheses are primarily concerned with the development of concepts and theories that integrate concepts [49]. The analysis in interpretive synthesis is conceptual both in process and outcome, and “the product is not aggregations of data, but theory? [49, p.12]. Interpretive syntheses involve induction and interpretation, and are primarily conceptual in process and outcome. Examples include integrative reviews, some systematic reviews, all of the qualitative syntheses, meta-narrative, realist and critical interpretive syntheses. Of note, both quantitative and qualitative studies can be either integrative or interpretive?
The second categorization, enhancement versus epistemological, applies to those approaches that use multiple data types and sources [80]. Popay's [80] classification reflects the ways that qualitative data are valued in relation to quantitative data.
In the enhancement model, qualitative data adds something to quantitative analysis. The enhancement model is reflected in systematic reviews and meta-analyses that use some qualitative data to enhance interpretation and explanation. It may also be reflected in some rapid reviews that draw on quantitative data but use some qualitative data.
The epistemological model assumes that quantitative and qualitative data are equal and each has something unique to contribute. All of the other review approaches, except pure quantitative or qualitative syntheses, reflect the epistemological model because they value all data types equally but see them as contributing different understandings.
By and large, the quantitative approaches (quantitative systematic review and meta-analysis) have typically used purely quantitative data (i.e., expressed in numeric form). More recently, both Cochrane [81] and Campbell [82] collaborations are grappling with the need to, and the process of, integrating qualitative research into a systematic review. The qualitative approaches use qualitative data (i.e., expressed in words). All of the emerging synthesis types, as well as the conventional integrative review, incorporate qualitative and quantitative study designs and data.
Four types of research questions direct inquiry across the different types of syntheses. The first is a well-developed research question that gives direction to the synthesis (e.g., meta-analysis, systematic review, meta-study, concept analysis, rapid review, realist synthesis). The second begins as a broad general question that evolves and becomes more refined over the course of the synthesis (e.g., meta-ethnography, scoping review, meta-narrative, critical interpretive synthesis). In the third type, the synthesis begins with a phenomenon of interest and the question emerges in the analytic process (e.g., grounded formal theory). Lastly, there is no clear question, but rather a general review purpose (e.g., integrative review). Thus, the requirement for a well-defined question cuts across at least three of the synthesis types (e.g., quantitative, qualitative, and emerging).
This is a contested issue within and between the four synthesis categories. There are strong proponents of quality appraisal in the quantitative traditions of systematic review and meta-analysis based on the need for strong studies that will not jeopardize validity of the overall findings. Nonetheless, there is no consensus on pre-defined criteria; many scales exist that vary dramatically in composition. This has methodological implications for the credibility of findings [83].
Specific methodologies from the conventional, qualitative, and emerging categories support quality appraisal but do so with caveats. In conventional integrative reviews appraisal is recommended, but depends on the sampling frame used in the study [18]. In meta-study, appraisal criteria are explicit but quality criteria are used in different ways depending on the specific requirements of the inquiry [54]. Among the emerging syntheses, meta-narrative review developers support appraisal of a study based on criteria from the research tradition of the primary study [67,84,85]. Realist synthesis similarly supports the use of high quality evidence, but appraisal checklists are viewed with scepticism and evidence is judged based on relevance to the research question and whether a credible inference may be drawn [69]. Like realist, critical interpretive syntheses do not judge quality using standardized appraisal instruments. They will exclude fatally flawed studies, but there is no consensus on what 'fatally flawed' means [49,71]. Appraisal is based on relevance to the inquiry, not rigor of the study.
There is no agreement on quality appraisal among qualitative meta-ethnographers with some supporting and others refuting the need for appraisal. [60,62]. Opponents of quality appraisal are found among authors of qualitative (grounded formal theory and concept analysis) and emerging syntheses (scoping and rapid reviews) because quality is not deemed relevant to the intention of the synthesis; the studies being reviewed are not effectiveness studies where quality is extremely important. These qualitative synthesis are often reviews of theoretical developments where the concept itself is what is important, or reviews that provide quotations from the raw data so readers can make their own judgements about the relevance and utility of the data. For example, in formal grounded theory, the purpose of theory generation and authenticity of data used to generate the theory is not as important as the conceptual category. Inaccuracies may be corrected in other ways, such as using the constant comparative method, which facilitates development of theoretical concepts that are repeatedly found in the data [86,87]. For pragmatic reasons, evidence is not assessed in rapid and scoping reviews, in part to produce a timely product. The issue of quality appraisal is unresolved across the terrain of research synthesis and we consider this further in our discussion.
All research syntheses share a common purpose -- to summarize, synthesize, or integrate research findings from diverse studies. This helps readers stay abreast of the burgeoning literature in a field. Our discussion here is at the level of the four categories of synthesis. Beginning with conventional literature syntheses, the overall purpose is to attend to mature topics for the purpose of re-conceptualization or to new topics requiring preliminary conceptualization [14]. Such syntheses may be helpful to consider contradictory evidence, map shifting trends in the study of a phenomenon, and describe the emergence of research in diverse fields [14]. The purpose here is to set the stage for a study by identifying what has been done, gaps in the literature, important research questions, or to develop a conceptual framework to guide data collection and analysis.
The purpose of quantitative systematic reviews is to combine, aggregate, or integrate empirical research to be able to generalize from a group of studies and determine the limits of generalization [27]. The focus of quantitative systematic reviews has been primarily on aggregating the results of studies evaluating the effectiveness of interventions using experimental, quasi-experimental, and more recently, observational designs. Systematic reviews can be done with or without quantitative meta-analysis but a meta-analysis always takes place within the context of a systematic review. Researchers must consider the review's purpose and the nature of their data in undertaking a quantitative synthesis; this will assist in determining the approach.
The purpose of qualitative syntheses is broadly to synthesize complex health experiences, practices, or concepts arising in healthcare environments. There may be various purposes depending on the qualitative methodology. For example, in hermeneutic studies the aim may be holistic explanation or understanding of a phenomenon [42], which is deepened by integrating the findings from multiple studies. In grounded formal theory, the aim is to produce a conceptual framework or theory expected to be applicable beyond the original study. Although not able to generalize from qualitative research in the statistical sense [88], qualitative researchers usually do want to say something about the applicability of their synthesis to other settings or phenomena. This notion of 'theoretical generalization' has been referred to as 'transferability' [89,90] and is an important criterion of rigour in qualitative research. It applies equally to the products of a qualitative synthesis in which the synthesis of multiple studies on the same phenomenon strengthens the ability to draw transferable conclusions.
The overarching purpose of emerging syntheses is challenging the more traditional types of syntheses, in part by using data from both quantitative and qualitative studies with diverse designs for analysis. Beyond this, however, each emerging synthesis methodology has a unique purpose. In meta-narrative review, the purpose is to identify different research traditions in the area, synthesize a complex and diverse body of research. Critical interpretive synthesis shares this characteristic. Although a distinctive approach, critical interpretive synthesis utilizes a modification of the analytic strategies of meta-ethnography [61] (e.g., reciprocal translational analysis, refutational synthesis, and lines of argument synthesis) but goes beyond the use of these to bring a critical perspective to bear in challenging the normative or epistemological assumptions in the primary literature [72,73]. The unique purpose of a realist synthesis is to amalgamate complex empirical evidence and theoretical understandings within a diverse body of literature to uncover the operative mechanisms and contexts that affect the outcomes of social interventions. In a scoping review, the intention is to find key concepts, examine the range of research in an area, and identify gaps in the literature. The purpose of a rapid review is comparable to that of a scoping review, but done quickly to meet the time-sensitive information needs of policy makers.
There are varying degrees of systematization across the categories of research synthesis. The most systematized are quantitative systematic reviews and meta-analyses. There are clear processes in each with judgments to be made at each step, although there are no agreed upon guidelines for this. The process is inherently subjective despite attempts to develop objective and systematic processes [91,92]. Mullen and Ramirez [27] suggest that there is often a false sense of rigour implied by the terms 'systematic review' and 'meta-analysis' because of their clearly defined procedures.
In comparison with some types of qualitative synthesis, concept analysis is quite procedural. Qualitative meta-synthesis also has defined procedures and is systematic, yet perhaps less so than concept analysis. Qualitative meta-synthesis starts in an unsystematic way but becomes more systematic as it unfolds. Procedures and frameworks exist for some of the emerging types of synthesis [e.g., 50, 63, 71, 93] but are not linear, have considerable flexibility, and are often messy with emergent processes [85]. Conventional literature reviews tend not to be as systematic as the other three types. In fact, the lack of systematization in conventional literature synthesis was the reason for the development of more systematic quantitative [17,20] and qualitative [45,46,61] approaches. Some authors in the field [18] have clarified processes for integrative reviews making them more systematic and rigorous, but most conventional syntheses remain relatively unsystematic in comparison with other types.
Some synthesis processes are considerably more complex than others. Methodologies with clearly defined steps are arguably less complex than the more flexible and emergent ones. We know that any study encounters challenges and it is rare that a pre-determined research protocol can be followed exactly as intended. Not even the rigorous methods associated with Cochrane [81] systematic reviews and meta-analyses are always implemented exactly as intended. Even when dealing with numbers rather than words, interpretation is always part of the process. Our collective experience suggests that new methodologies (e.g., meta-narrative synthesis and realist synthesis) that integrate different data types and methods are more complex than conventional reviews or the rapid and scoping reviews.
The products of research syntheses usually take three distinct formats (see Table 1 and Additional File 1 for further details). The first representation is in tables, charts, graphical displays, diagrams and maps as seen in integrative, scoping and rapid reviews, meta-analyses, and critical interpretive syntheses. The second type of synthesis product is the use of mathematical scores. Summary statements of effectiveness are mathematically displayed in meta-analyses (as an effect size), systematic reviews, and rapid reviews (statistical significance).
The third synthesis product may be a theory or theoretical framework. A mid-range theory can be produced from formal grounded theory, meta-study, meta-ethnography, and realist synthesis. Theoretical/conceptual frameworks or conceptual maps may be created in meta-narrative and critical interpretive syntheses, and integrative reviews. Concepts for use within theories are produced in concept analysis. While these three product types span the categories of research synthesis, narrative description and summary is used to present the products resulting from all methodologies.
There are diverse ways that context is considered in the four broad categories of synthesis. Context may be considered to the extent that it features within primary studies for the purpose of the review. Context may also be understood as an integral aspect of both the phenomenon under study and the synthesis methodology (e.g., realist synthesis). Quantitative systematic reviews and meta-analyses have typically been conducted on studies using experimental and quasi-experimental designs and more recently observational studies, which control for contextual features to allow for understanding of the 'true' effect of the intervention [94].
More recently, systematic reviews have included covariates or mediating variables (i.e., contextual factors) to help explain variability in the results across studies [27]. Context, however, is usually handled in the narrative discussion of findings rather than in the synthesis itself. This lack of attention to context has been one criticism leveled against systematic reviews and meta-analyses, which restrict the types of research designs that are considered [e.g., 95].
When conventional literature reviews incorporate studies that deal with context, there is a place for considering contextual influences on the intervention or phenomenon. Reviews of quantitative experimental studies tend to be devoid of contextual considerations since the original studies are similarly devoid, but context might figure prominently in a literature review that incorporates both quantitative and qualitative studies.
Qualitative syntheses have been conducted on the contextual features of a particular phenomenon [33]. Paterson et al. [54] advise researchers to attend to how context may have influenced the findings of particular primary studies. In qualitative analysis, contextual features may form categories by which the data can be compared and contrasted to facilitate interpretation. Because qualitative research is often conducted to understand a phenomenon as a whole, context may be a focus, although this varies with the qualitative methodology. At the same time, the findings in a qualitative synthesis are abstracted from the original reports and taken to a higher level of conceptualization, thus removing them from the original context.
Meta-narrative synthesis [67,84], because it draws on diverse research traditions and methodologies, may incorporate context into the analysis and findings. There is not, however, an explicit step in the process that directs the analyst to consider context. Generally, the research question guiding the synthesis is an important factor in whether context will be a focus.
More recent iterations of concept analysis [47,96,97] explicitly consider context reflecting the assumption that a concept's meaning is determined by its context. Morse [47] points out, however, that Wilson's [98] approach to concept analysis, and those based on Wilson [e.g., 45], identify attributes that are devoid of context, while Rodgers' [96,99] evolutionary method considers context (e.g., antecedents, consequences, and relationships to other concepts) in concept development.
Realist synthesis [69] considers context as integral to the study. It draws on a critical realist logic of inquiry grounded in the work of Bhaskar [100], who argues that empirical co-occurrence of events is insufficient for inferring causation. One must identify generative mechanisms whose properties are causal and, depending on the situation, may nor may not be activated [94]. Context interacts with program/intervention elements and thus cannot be differentiated from the phenomenon [69]. This approach synthesizes evidence on generative mechanisms and analyzes contextual features that activate them; the result feeds back into the context. The focus is on what works, for whom, under what conditions, why and how [68].
When we began our review, we 'assumed' that the assumptions underlying synthesis methodologies would be a distinguishing characteristic of synthesis types, and that we could compare the various types on their assumptions, explicit or implicit. We found, however, that many authors did not explicate the underlying assumptions of their methodologies, and it was difficult to infer them. Kirkevold [101] has argued that integrative reviews need to be carried out from an explicit philosophical or theoretical perspective. We argue this should be true for all types of synthesis.
Authors of some emerging synthesis approaches have been very explicit about their assumptions and philosophical underpinnings. An implicit assumption of most emerging synthesis methodologies is that quantitative systematic reviews and meta-analyses have limited utility in some fields [e.g., in public health -13, 102] and for some kinds of review questions like those about feasibility and appropriateness versus effectiveness [103,104]. They also assume that ontologically and epistemologically, both kinds of data can be combined. This is a significant debate in the literature because it is about the commensurability of overarching paradigms [105] but this is beyond the scope of this review.
Realist synthesis is philosophically grounded in critical realism or, as noted above, a realist logic of inquiry [93,99,106,107]. Key assumptions regarding the nature of interventions that inform critical realism have been described above in the section on context. See Pawson et al. [106] for more information on critical realism, the philosophical basis of realist synthesis.
Meta-narrative synthesis is explicitly rooted in a constructivist philosophy of science [108] in which knowledge is socially constructed rather than discovered, and what we take to be 'truth' is a matter of perspective. Reality has a pluralistic and plastic character, and there is no pre-existing 'real world' independent of human construction and language [109]. See Greenhalgh et al. [67,85] and Greenhalgh & Wong [97] for more discussion of the constructivist basis of meta-narrative synthesis.
In the case of purely quantitative or qualitative syntheses, it may be an easier matter to uncover unstated assumptions because they are likely to be shared with those of the primary studies in the genre. For example, grounded formal theory shares the philosophical and theoretical underpinnings of grounded theory, rooted in the theoretical perspective of symbolic interactionism [110,111] and the philosophy of pragmatism [87,112,113,114].
As with meta-narrative synthesis, meta-study developers identify constructivism as their interpretive philosophical foundation [54,88]. Epistemologically, constructivism focuses on how people construct and re-construct knowledge about a specific phenomenon, and has three main assumptions: (1) reality is seen as multiple, at times even incompatible with the phenomenon under consideration; (2) just as primary researchers construct interpretations from participants' data, meta-study researchers also construct understandings about the primary researchers' original findings. Thus, meta-synthesis is a construction of a construction, or a meta-construction; and (3) all constructions are shaped by the historical, social and ideological context in which they originated [54]. The key message here is that reports of any synthesis would benefit from an explicit identification of the underlying philosophical perspectives to facilitate a better understanding of the results, how they were derived, and how they are being interpreted.
The unit of analysis for each category of review is generally distinct. For the emerging synthesis approaches, the unit of analysis is specific to the intention. In meta-narrative synthesis it is the storyline in diverse research traditions; in rapid review or scoping review, it depends on the focus but could be a concept; and in realist synthesis, it is the theories rather than programs that are the units of analysis. The elements of theory that are important in the analysis are mechanisms of action, the context, and the outcome [107].
For qualitative synthesis, the units of analysis are generally themes, concepts or theories, although in meta-study, the units of analysis can be research findings (“meta-data-analysis”), research methods (“meta-method”) or philosophical/theoretical perspectives (“meta-theory”) [54]. In quantitative synthesis, the units of analysis range from specific statistics for systematic reviews to effect size of the intervention for meta-analysis. More recently, some systematic reviews focus on theories [115,116], therefore it depends on the research question. Similarly, within conventional literature synthesis the units of analysis also depend on the research purpose, focus and question as well as on the type of research methods incorporated into the review. What is important in all research syntheses, however, is that the unit of analysis needs to be made explicit. Unfortunately, this is not always the case.
In this section, we discuss the overarching strengths and limitations of synthesis methodologies as a whole and then highlight strengths and weaknesses across each of our four categories of synthesis.
With the vast proliferation of research reports and the increased ease of retrieval, research synthesis has become more accessible providing a way of looking broadly at the current state of research. The availability of syntheses helps researchers, practitioners, and policy makers keep up with the burgeoning literature in their fields without which evidence-informed policy or practice would be difficult. Syntheses explain variation and difference in the data helping us identify the relevance for our own situations; they identify gaps in the literature leading to new research questions and study designs. They help us to know when to replicate a study and when to avoid excessively duplicating research. Syntheses can inform policy and practice in a way that well-designed single studies cannot; they provide building blocks for theory that helps us to understand and explain our phenomena of interest.
The process of selecting, combining, integrating, and synthesizing across diverse study designs and data types can be complex and potentially rife with bias, even with those methodologies that have clearly defined steps. Just because a rigorous and standardized approach has been used does not mean that implicit judgements will not influence the interpretations and choices made at different stages.
In all types of synthesis, the quantity of data can be considerable, requiring difficult decisions about scope, which may affect relevance. The quantity of available data also has implications for the size of the research team. Few reviews these days can be done independently, in particular because decisions about inclusion and exclusion may require the involvement of more than one person to ensure reliability.
For all types of synthesis, it is likely that in areas with large, amorphous, and diverse bodies of literature, even the most sophisticated search strategies will not turn up all the relevant and important texts. This may be more important in some synthesis methodologies than in others, but the omission of key documents can influence the results of all syntheses. This issue can be addressed, at least in part, by including a library scientist on the research team as required by some funding agencies. Even then, it is possible to miss key texts. In this review, for example, because none of us are trained in or conduct meta-analyses, we were not even aware that we had missed some new developments in this field such as meta-regression [117,118], network meta-analysis [119,120,121], and the use of individual patient data in meta-analyses [122,123].
One limitation of systematic reviews and meta-analyses is that they rapidly go out of date. We thought this might be true for all types of synthesis, although we wondered if those that produce theory might not be somewhat more enduring. We have not answered this question but it is open for debate. For all types of synthesis, the analytic skills and the time required are considerable so it is clear that training is important before embarking on a review, and some types of review may not be appropriate for students or busy practitioners.
Finally, the quality of reporting in primary studies of all genres is variable so it is sometimes difficult to identify aspects of the study essential for the synthesis, or to determine whether the study meets quality criteria. There may be flaws in the original study, or journal page limitations may necessitate omitting important details. Reporting standards have been developed for some types of reviews (e.g., systematic review, meta-analysis, meta-narrative synthesis, realist synthesis); but there are no agreed upon standards for qualitative reviews. This is an important area for development in advancing the science of research synthesis.
The conventional literature review and now the increasingly common integrative review remain important and accessible approaches for students, practitioners, and experienced researchers who want to summarize literature in an area but do not have the expertise to use one of the more complex methodologies. Carefully executed, such reviews are very useful for synthesizing literature in preparation for research grants and practice projects. They can determine the state of knowledge in an area and identify important gaps in the literature to provide a clear rationale or theoretical framework for a study [14,18]. There is a demand, however, for more rigour, with more attention to developing comprehensive search strategies and more systematic approaches to combining, integrating, and synthesizing the findings.
Generally, conventional reviews include diverse study designs and data types that facilitate comprehensiveness, which may be a strength on the one hand, but can also present challenges on the other. The complexity inherent in combining results from studies with diverse methodologies can result in bias and inaccuracies. The absence of clear guidelines about how to synthesize across diverse study types and data [18] has been a challenge for novice reviewers.
Quantitative systematic reviews and meta-analyses have been important in launching the field of evidence-based healthcare. They provide a systematic, orderly and auditable process for conducting a review and drawing conclusions [25]. They are arguably the most powerful approaches to understanding the effectiveness of healthcare interventions, especially when intervention studies on the same topic show very different results. When areas of research are dogged by controversy [25] or when study results go against strongly held beliefs, such approaches can reduce the uncertainty and bring strong evidence to bear on the controversy.
Despite their strengths, they also have limitations. Systematic reviews and meta-analyses do not provide a way of including complex literature comprising various types of evidence including qualitative studies, theoretical work, and epidemiological studies.?Only certain types of design are considered and qualitative data are used in a limited way. This exclusion limits what can be learned in a topic area.
Meta-analyses are often not possible because of wide variability in study design, population, and interventions so they may have a narrow range of utility. New developments in meta-analysis, however, can be used to address some of these limitations. Network meta-analysis is used to explore relative efficacy of multiple interventions, even those that have never been compared in more conventional pairwise meta-analyses [121], allowing for improved clinical decision making [120]. The limitation is that network meta-analysis has only been used in medical/clinical applications [119] and not in public health. It has not yet been widely accepted and many methodological challenges remain [120,121]. Meta-regression is another development that combines meta-analytic and linear regression principles to address the fact that heterogeneity of results may compromise a meta-analysis [117,118]. The disadvantage is that many clinicians are unfamiliar with it and may incorrectly interpret results [117].
Some have accused meta-analysis of combining apples and oranges [124] raising questions in the field about their meaningfulness [25,28]. More recently, the use of individual rather than aggregate data has been useful in facilitating greater comparability among studies [122]. In fact, Tomas et al. [123] argue that meta-analysis using individual data is now the gold standard although access to the raw data from other studies may be a challenge to obtain.
The usefulness of systematic reviews in synthesizing complex health and social interventions has also been challenged [102]. It is often difficult to synthesize their findings because such studies are “epistemologically diverse and methodologically complex”[[69], p.21]. Rigid inclusion/exclusion criteria may allow only experimental or quasi-experimental designs into consideration resulting in lost information that may well be useful to policy makers for tailoring an intervention to the context or understanding its acceptance by recipients.
Qualitative syntheses may be the type of review most fraught with controversy and challenge, while also bringing distinct strengths to the enterprise. Although these methodologies provide a comprehensive and systematic review approach, they do not generally provide definitive statements about intervention effectiveness. They do, however, address important questions about the development of theoretical concepts, patient experiences, acceptability of interventions, and an understanding about why interventions might work.
Most qualitative syntheses aim to produce a theoretically generalizable mid-range theory that explains variation across studies. This makes them more useful than single primary studies, which may not be applicable beyond the immediate setting or population. All provide a contextual richness that enhances relevance and understanding. Another benefit of some types of qualitative synthesis (e.g., grounded formal theory) is that the concept of saturation provides a sound rationale for limiting the number of texts to be included thus making reviews potentially more manageable. This contrasts with the requirements of systematic reviews and meta-analyses that require an exhaustive search.
Qualitative researchers debate about whether the findings of ontologically and epistemological diverse qualitative studies can actually be combined or synthesized [125] because methodological diversity raises many challenges for synthesizing findings. The products of different types of qualitative syntheses range from theory and conceptual frameworks, to themes and rich descriptive narratives. Can one combine the findings from a phenomenological study with the theory produced in a grounded theory study? Many argue yes, but many also argue no.
Emerging synthesis methodologies were developed to address some limitations inherent in other types of synthesis but also have their own issues. Because each type is so unique, it is difficult to identify overarching strengths of the entire category. An important strength, however, is that these newer forms of synthesis provide a systematic and rigorous approach to synthesizing a diverse literature base in a topic area that includes a range of data types such as: both quantitative and qualitative studies, theoretical work, case studies, evaluations, epidemiological studies, trials, and policy documents. More than conventional literature reviews and systematic reviews, these approaches provide explicit guidance on analytic methods for integrating different types of data. The assumption is that all forms of data have something to contribute to knowledge and theory in a topic area. All have a defined but flexible process in recognition that the methods may need to shift as knowledge develops through the process.
Many emerging synthesis types are helpful to policy makers and practitioners because they are usually involved as team members in the process to define the research questions, and interpret and disseminate the findings. In fact, engagement of stakeholders is built into the procedures of the methods. This is true for rapid reviews, meta-narrative syntheses, and realist syntheses. It is less likely to be the case for critical interpretive syntheses.
Another strength of some approaches (realist and meta-narrative syntheses) is that quality and publication standards have been developed to guide researchers, reviewers, and funders in judging the quality of the products [108,126,127]. Training materials and online communities of practice have also been developed to guide users of realist and meta-narrative review methods [107,128]. A unique strength of critical interpretive synthesis is that it takes a critical perspective on the process that may help reconceptualize the data in a way not considered by the primary researchers [72].
There are also challenges of these new approaches. The methods are new and there may be few published applications by researchers other than the developers of the methods, so new users often struggle with the application. The newness of the approaches means that there may not be mentors available to guide those unfamiliar with the methods. This is changing, however, and the number of applications in the literature is growing with publications by new users helping to develop the science of synthesis [e.g., 129]. However, the evolving nature of the approaches and their developmental stage present challenges for novice researchers.
Choosing an appropriate approach to synthesis will depend on the question you are asking, the purpose of the review, and the outcome or product you want to achieve. In Additional File 1, we discuss each of these to provide guidance to readers on making a choice about review type. If researchers want to know whether a particular type of intervention is effective in achieving its intended outcomes, then they might choose a quantitative systemic review with or without meta-analysis, possibly buttressed with qualitative studies to provide depth and explanation of the results. Alternately, if the concern is about whether an intervention is effective with different populations under diverse conditions in varying contexts, then a realist synthesis might be the most appropriate.
If researchers' concern is to develop theory, they might consider qualitative syntheses or some of the emerging syntheses that produce theory (e.g., critical interpretive synthesis, realist review, grounded formal theory, qualitative meta-synthesis). If the aim is to track the development and evolution of concepts, theories or ideas, or to determine how an issue or question is addressed across diverse research traditions, then meta-narrative synthesis would be most appropriate.
When the purpose is to review the literature in advance of undertaking a new project, particularly by graduate students, then perhaps an integrative review would be appropriate. Such efforts contribute towards the expansion of theory, identify gaps in the research, establish the rationale for studying particular phenomena, and provide a framework for interpreting results in ways that might be useful for influencing policy and practice.
For researchers keen to bring new insights, interpretations, and critical re-conceptualizations to a body of research, then qualitative or critical interpretive syntheses will provide an inductive product that may offer new understandings or challenges to the status quo. These can inform future theory development, or provide guidance for policy and practice.
What is the current state of science regarding research synthesis? Public health, health care, and social science researchers or clinicians have previously used all four categories of research synthesis, and all offer a suitable array of approaches for inquiries. New developments in systematic reviews and meta-analysis are providing ways of addressing methodological challenges [117,118,119,120,121,122,123]. There has also been significant advancement in emerging synthesis methodologies and they are quickly gaining popularity. Qualitative meta-synthesis is still evolving, particularly given how new it is within the terrain of research synthesis. In the midst of this evolution, outstanding issues persist such as grappling with: the quantity of data, quality appraisal, and integration with knowledge translation. These topics have not been thoroughly addressed and need further debate.
We raise the question of whether it is possible or desirable to find all available studies for a synthesis that has this requirement (e.g., meta-analysis, systematic review, scoping, meta-narrative synthesis [25,27,63,67,84,85]). Is the synthesis of all available studies a realistic goal in light of the burgeoning literature? And how can this be sustained in the future, particularly as the emerging methodologies continue to develop and as the internet facilitates endless access? There has been surprisingly little discussion on this topic and the answers will have far-reaching implications for searching, sampling, and team formation.
Researchers and graduate students can no longer rely on their own independent literature search. They will likely need to ask librarians for assistance as they navigate multiple sources of literature and learn new search strategies. Although teams now collaborate with library scientists, syntheses are limited in that researchers must make decisions on the boundaries of the review, in turn influencing the study's significance. The size of a team may also be pragmatically determined to manage the search, extraction, and synthesis of the burgeoning data. There is no single answer to our question about the possibility or necessity of finding all available articles for a review. Multiple strategies that are situation specific are likely to be needed.
While the issue of quality appraisal has received much attention in the synthesis literature, scholars are far from resolution. There may be no agreement about appraisal criteria in a given tradition. For example, the debate rages over the appropriateness of quality appraisal in qualitative synthesis where there are over 100 different sets of criteria and many do not overlap [49]. These differences may reflect disciplinary and methodological orientations, but diverse quality appraisal criteria may privilege particular types of research [49]. The decision to appraise is often grounded in ontological and epistemological assumptions. Nonetheless, diversity within and between categories of synthesis is likely to continue unless debate on the topic of quality appraisal continues and evolves toward consensus.
If research syntheses are to make a difference to practice and ultimately to improve health outcomes, then we need to do a better job of knowledge translation. In the Canadian Institutes of Health Research (CIHR) definition of knowledge translation (KT), research or knowledge synthesis is an integral component [130]. Yet, with few exceptions [131,132], very little of the research synthesis literature even mentions the relationship of synthesis to KT nor does it discuss strategies to facilitate the integration of synthesis findings into policy and practice. The exception is in the emerging synthesis methodologies, some of which (e.g., realist and meta-narrative syntheses, scoping reviews) explicitly involve stakeholders or knowledge users. The argument is that engaging them in this way increases the likelihood that the knowledge generated will be translated into policy and practice. We suggest that a more explicit engagement with knowledge users in all types of synthesis would benefit the uptake of the research findings.
Research synthesis neither makes research more applicable to practice nor ensures implementation. Focus must now turn seriously towards translation of synthesis findings into knowledge products that are useful for health care practitioners in multiple areas of practice and develop appropriate strategies to facilitate their use. The burgeoning field of knowledge translation has, to some extent, taken up this challenge; however, the research-practice gap continues to plague us [133,134]. It is a particular problem for qualitative syntheses [131]. Although such syntheses have an important place in evidence-informed practice, little effort has gone into the challenge of translating the findings into useful products to guide practice [131].
Our study took longer than would normally be expected for an integrative review. Each of us were primarily involved in our own dissertations or teaching/research positions, and so this study was conducted 'off the sides of our desks.' A limitation was that we searched the literature over the course of 4 years (from 2008-2012), necessitating multiple search updates. Further, we did not do a comprehensive search of the literature after 2012, thus the more recent synthesis literature was not systematically explored. We did, however, perform limited database searches from 2012-2015 to keep abreast of the latest methodological developments. Although we missed some new approaches to meta-analysis in our search, we did not find any new features of the synthesis methodologies covered in our review that would change the analysis or findings of this article. Lastly, we struggled with the labels used for the broad categories of research synthesis methodology because of our hesitancy to reinforce the divide between quantitative and qualitative approaches. However, it was very difficult to find alternative language that represented the types of data used in these methodologies. Despite our hesitancy in creating such an obvious divide, we were left with the challenge of trying to find a way of characterizing these broad types of syntheses.
Our findings offer methodological clarity for those wishing to learn about the broad terrain of research synthesis. We believe that our review makes transparent the issues and considerations in choosing from among the four broad categories of research synthesis. In summary, research synthesis has taken its place as a form of research in its own right. The methodological terrain has deep historical roots reaching back over the past 200 years, yet research synthesis remains relatively new to public health, health care, and social sciences in general. This is rapidly changing. New developments in systematic reviews and meta-analysis, and the emergence of new synthesis methodologies provide a vast array of options to review the literature for diverse purposes. New approaches to research synthesis and new analytic methods within existing approaches provide a much broader range of review alternatives for public health, health care, and social science students and researchers.
The authors declare that they have no conflicts of interest in this article.
KSM co-designed the study, collected data, analyzed the data, drafted/revised the manuscript, and managed the project.
MM co-designed the study, collected data, analyzed the data, and co-wrote and revised the manuscript for intellectual content.
MP contributed to searching the literature, developing the analytic framework, and extracting data for the Additional File.
JB contributed to searching the literature, developing the analytic framework, and extracting data for the Additional File.
WN contributed to searching the literature, developing the analytic framework, and extracting data for the Additional File.
All authors read and approved the final manuscript.
KSM is an assistant professor in the Faculty of Nursing at the University of Alberta. Her work on this article was largely conducted as a Postdoctoral Fellow, funded by KRESCENT (Kidney Research Scientist Core Education and National Training Program, reference #KRES110011R1) and the Faculty of Nursing at the University of Alberta.
MM's work on this study over the period of 2008-2014 was supported by a Canadian Institutes of Health Research Applied Public Health Research Chair Award (grant #92365).
We thank Rachel Spanier who provided support with reference formatting.
Additional File 1 - Selected Types of Research Synthesis
This Additional File is our dataset created to organize, analyze and critique the literature that we synthesized in our integrative review. Our results were created based on analysis of this Additional File.
Types of Research Synthesis | Key Characteristics | Purpose | Methods | Product |
CONVEN-TIONAL | What is it? “The integrative literature review is a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated” [[14], p.356]. | Integrative reviews are used to address mature topics in order to re-conceptualize the expanding and diverse literature on the topic. They are also used to comprehensively review new topics in need of preliminary conceptualization [14]. | Integrative reviews generally contain similar steps [14,18], which include the following: | Conclusions are often presented in a table/diagram. Explicit details from primary sources to support conclusions must be provided to demonstrate a logical chain of evidence. |
1) Identify a clear problem. | ||||
Integrative Review | Data type: Integrative literature reviews include studies using diverse methodologies (i.e., experimental and non-experimental research, as well as qualitative research) in order to more fully understand a phenomenon of interest. It may also include theoretical and empirical literature. | Integrative reviews should ultimately present the “state of the art” of knowledge, depict the breadth and depth of the topic, and contribute to greater understanding of the phenomenon [18]. | 2) Determine the variables of interest (e.g., population, concept). | Torraco [14] suggests they can be represented in four forms: |
3) State a specific research purpose. | 1) A research agenda, | |||
Research question: Start by clearly identifying the problem that the review is addressing and the purpose of the review. There usually is not a specific research question, but rather a research purpose. | 4) Define and clearly document a search strategy. Aim to locate as many of the existing studies as possible. Purposive sampling may be used along with a more comprehensive approach. | 2) A taxonomy or conceptual classification of constructs, | ||
5) Critically evaluate the quality of primary reviews depending on the sampling frame used in the integrative review. | 3) Alternative models/conceptual framework, and | |||
Quality appraisal: The quality of primary sources may be appraised using broad criteria. How quality is evaluated will depend upon the sampling frame [18]. | 6) Identify a systematic analytic method. The constant comparative method [86,135] is one overarching approach commonly used. | 4) Metatheory. | ||
7) Keep a record of the process of data analysis (e.g., hunches, decisions, ideas about interpretation). | Results should emphasize implications for policy/practice [18]. | |||
8) State methodological limitations. | ||||
QUANT-ITATIVE | What is it? A SR is a review of literature that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies. Conducting a SR is analogous to conducting a primary study in that there are steps and protocols. It may or may not be done in conjunction with a meta-analysis. | The purpose of a SR is to integrate empirical research for the purpose of generalizing from a group of studies. The reviewer is also seeking to discover the limits of generalization [27]. | A number of authors have provided guidelines for conducting a SR [27] but they generally contain similar steps: | The products of a SR may include: |
1) Specify study aims and define research question. | 1) A statement about the relative “effectiveness” of health care interventions, or about the appropriateness, feasibility, or meaningfulness of findings for particular purposes; | |||
In Cochrane [81], a SR is identified as the highest form of evidence in support of interventions. By contrast, the Joanna Briggs Institute [104] does not define a SR as necessarily the highest form of evidence. | Often, the review focuses on questions of intervention effectiveness. Thus, the intent is to summarize across studies to obtain a summative judgment about the effectiveness of interventions. However, the Joanna Briggs Institute [104] suggests that for nursing, there is a concern not just with effectiveness but also with questions of appropriateness, meaningfulness and feasibility of health practices and delivery methods. Thus, SR’s may have purposes other than to assess the effectiveness of interventions. | 2) Set inclusion criteria for evidence. | 2) A statement about the strength of the relationship between a particular intervention and specific outcomes. | |
Systematic Review (SR) | 3) Design search strategy. | 3) More recently, the product might be a statement about the convergence of theoretical perspectives on a topic. | ||
As noted below, a meta-analysis is always a SR, but a SR is not always a meta-analysis. | 4) Screen potential evidence against criteria for assessing quality. | 4) When done in conjunction with meta-analysis, the product is a mathematic score that represents the statements above. | ||
5) Design data collection protocol. | ||||
Data type: There is nothing that specifies data have to be quantitative, and the definition can apply to qualitative findings. Generally, however, the term has been used most frequently to apply to reviews of quantitative studies - traditional RCTs and experimental or quasi-experimental designs. More recently, both the Campbell and the Cochrane collaborations have been grappling with the need to, and the process of, integrating qualitative research into a SR. A number of studies have been published that do this [13,75,78,135,136,137,138]. | 6) Select appropriate metric to represent the magnitude of findings and assess likelihood they are due to chance. | |||
7) Code the primary studies. | ||||
Research question: A well-defined research question is required. | 8) Analyze and display data using appropriate methods. | |||
9) Draw conclusions based on data. | ||||
Quality appraisal: The Quality Appraisal section under MA above also applies to SR. Some researchers are developing standard reliable and valid quality appraisal tools to judge the quality of primary studies but there remains no consensus on which tools should be used. The Joanna Briggs Institute [104] has developed their own criteria to ensure that only the highest quality studies are included in SRs for nursing, but they hold that studies from any methodological position are relevant. | 10) Discuss alternate interpretations in light of studies’ strengths and limitations. | |||
QUANT-ITATIVE | What is it? M-A is the statistical analysis of a large collection of results from individual studies (usually interventions) for the purposes of integrating the findings, based on conversion to a common metric (effect size) to determine the overall effect and its magnitude. The term was coined by Gene Glass [22,23] but dates back to 1904 [17]. A M-A is always a SR (see above). | Analytic M-As are conducted for the purpose of summarizing and integrating the results of individual primary studies to increase the power for detecting intervention effects, which may be small and insignificant in the individual studies [139,140]. | Specific steps include [25]: | The product for M-A includes a narrative summary of the findings with a conclusion about the effectiveness of interventions. |
1) Define the dependent and independent variables of interest. | 1) Analytic Products: | |||
Meta-Analysis (M-A) | Data type: Data are from quantitative research studies and findings, primarily randomized control trials. Increasingly there is use of experimental, quasi-experimental and some types of observational studies. Each primary study is abstracted and coded into a database. | Exploratory M-As are conducted to resolve controversy in a field or to pose and answer new questions. The main concern is to explain the variation in effect sizes. | 2) Collect the studies in a systematic way attempting to find all published and unpublished studies. | · Graphical displays of the data and a table that displays the key elements of each study. |
3) Read methods carefully and if effect sizes are not reported, identify articles for information to calculate these. | 2) Final product: | |||
Research question: A clear, well-defined research question or hypothesis is required. | 4) Examine variability among the obtained effect sizes informally with graphs and charts, to identify the possibility that moderator variables may account for the variability. | · A mathematic score that represents the strength of the effect of an intervention or the relationships between two variables. | ||
5) Combine effects using several measures of their central tendency and explore reasons for differences if found. | · Identification of variables that moderate or mediate the effects or relationships. | |||
Quality appraisal: Articles are usually appraised according to a set of pre-defined criteria but these criteria vary considerably and there are many methodological limitations [83]. Lower quality studies are not necessarily excluded and there is some debate about whether these should be included [24,29]. When lower quality studies are included, the validity of the findings is often discussed in relation to the study quality. | 6) Examine the significance level of the indices of central tendency, usually employing confidence intervals around unweighted mean effect sizes in a random effects model. | |||
7) Using an examination of the binomial effect size display, evaluate the importance of the obtained effect size. | ||||
QUAL-ITATIVE | What is it? “Meta-study is a research approach involving analysis of the theory, methods, and findings of qualitative research and the synthesis of these insights into new ways of thinking about phenomenon” [[54], p. 1]. | Analysis of research findings, methods, and theory across qualitative studies are compared and contrasted to create a new interpretation [53]. | Paterson et al. [54] propose a clear set of techniques: | Through the three meta-study processes, researchers create a “meta-synthesis” which brings together ideas to develop a mid-range theory as the product. |
1) Choose an analytic approach (e.g. grounded theory, thematic analysis). | ||||
Meta-Study | Data type: Three analytic components are undertaken prior to synthesis. Data includes qualitative findings (meta-data), research methods (meta-method), and/or philosophical/theoretical perspectives (meta-theory). | 2) Use specific sampling techniques according to inclusion/exclusion criteria, including searching for disconfirming cases that challenge the emerging theory. | ||
3) Regardless of approach, group studies according to characteristics (e.g., disease) and treat each group as a case [49]. | ||||
Research question: A relevant, well-defined research question is used. | 4) Engage in three distinct types of analysis, i.e. meta-data, meta-study, meta-theory (may be undertaken concurrently). | |||
5) Synthesize analysis into a theory. | ||||
Critical appraisal: According to Paterson et al. [54], primary articles are appraised according to specific criteria; however the specific appraisal will depend on the requirements of the meta-study. Studies of poor quality will be excluded. Data from included studies may also be excluded if reported themes are not supported by the presented data. | ||||
QUAL-ITATIVE | What is it? Meta-ethnography entails choosing relevant empirical studies to synthesize through repetitive reading while noting metaphors [61,62]. Noblit and Hare explain that “metaphors” refer to “themes, perspectives, organizers, and/or concepts revealed by qualitative studies” [[61], p. 15]. These metaphors are then used as data for the synthesis through (at least) one of three strategies including reciprocal translation, refutational synthesis, and/or line of argument syntheses. A meta-ethnographic synthesis is the creation of interpretive (abstract) explanations that are essentially metaphoric. The goal is to create, in a reduced form, a representation of the abstraction through metaphor, all the while preserving the relationships between concepts [61]. | To synthesize qualitative studies through a building of “comparative understanding” [[61], p. 22] so that the result is greater than the sum of the parts. | Methods used in meta-ethnography generally following the following: | The product of a meta-ethnography is a mid-range theory that has greater explanatory power than could be otherwise achieved in a conventional literature review. |
· Frame the study broadly by an interest, aim or purpose and ultimately, a research question. | ||||
Meta-Ethno-graphy | Data type: Qualitative research studies and findings on a specific topic. | Noblit and Hare summarize that meta-ethnography is “a form of synthesis for ethnographic or other interpretive studies. It enables us to talk to each other about our studies; to communicate to policy makers, concerned citizens, and scholars what interpretive research reveals; and to reflect on our collective craft and the place of our own studies within it” [[61], p.14]. | · Create inclusion/exclusion criteria. | |
· Conduct a review of the literature based on who the audience will be, what is credible to the audience, what accounts are available, and what the researchers’ interests are in the study [61]. | ||||
Research question: An “intellectual interest” [61], p.26] begins the process. Then, a relevant research question, aim, or purpose is developed. | · Identify all the appropriate studies in a field through repeated readings. | |||
Quality appraisal: Researchers are divided on the merits of critical appraisal and whether or not it should be a standard element in meta-ethnography [60]. Some researchers choose to follow pre-determined criteria based on critical appraisal [e.g., 62], whereas others do not critically appraise. | Noblit and Hare [61] identified three possible analysis strategies (all do not have to be completed): | |||
1) Reciprocal translational analysis. Key themes, metaphors, or concepts are identified and translated into each other to create the most representative concept. | ||||
2) Refutational synthesis. Contradictions between key themes, metaphors, or concepts are examined and explained. | ||||
3) Lines of argument synthesis. Interpretation is created from comparison of findings across distinct studies. | ||||
QUAL-ITATIVE | What is it? A grounded formal theory (GFT) is a synthesis of substantive grounded theories (GTs) to produce a higher order, more abstract theory that goes beyond the specifics of the original theories. GFT takes into account the conditions under which the primary study data were collected and analyzed to develop a more generalized and abstract model [31]. | The intent of GFT is to expand the applicability of individual GTs by synthesizing the findings to provide a broad meaning that is based in data and is applicable to people who experience a common phenomenon across populations and context [51]. | GFT uses the same methods that were used to create the original GTs in the synthesis [48,51]. Specific elements of the analytic process include: | A GFT is a mid-range GT that has “fit, work and grab”: that is, it fits the data (concepts and categories from primary studies), works to explain the phenomenon under review, and resonates with the readers’ experiences and understandings. |
1) Theoretical sampling - sample size is determined through purposive and theoretical sampling strategies to answer emerging questions [37,51]. | ||||
Data type: Substantive GTs were originally constructed using the methodology developed by Glaser & Strauss [86]. While some synthesis approaches emphasize including all possible primary GT studies, the concept of saturation in GFT (see Methods column) allows limiting the number of reviewed papers to emphasize robustness rather than completeness [50]. | The focus is on the conditions under which theoretical generalizations apply. GFT aims “to bring cultural and individual differences into dialogue with each other by seeking a metaphor through which those differences can be understood by others” [[31], p.1354]. | 2) Constant comparative analysis -the analyst identifies concepts and their relationship with other data, and compares theoretical ideas to prior and subsequent data. | Thorne et al. suggest that a GFT is “an artistic explanation that works for now, a model created on the basis of limited materials and a specific, situated perspective within known and unconscious limits of representation” [31], p. 1354]. | |
Grounded Formal Theory (GFT) | 3) Memoing - documentation of hunches, decisions, and modifications during analysis. | |||
Research question: GFT begins with a phenomenon of focus [51]. Analytic questions and the overall research question emerge throughout the process. | 4) Saturation - the point at which continued data collection and analysis brings only repeated concepts or ideas. | |||
5) Coding - begins at a descriptive level and progresses towards a more abstract and theoretical level. Findings are synthesized and translated across studies. | ||||
Quality appraisal: There is no discussion in the GFT literature about critically appraising the studies to be included. However, the nature of the analytic process suggests that critical appraisal may not be relevant. The authenticity and accuracy of data in a GFT are not an issue because, for the purposes of generating theory, what is important is the conceptual category and not the accuracy of the evidence. The constant comparative method of GFT will correct for such inaccuracies because each concept must “earn” its way into the theory by repeatedly showing up [67,68]. | ||||
QUAL-ITATIVE | What is it? Concept analysis is a systematic procedure to extract attributes of a concept from literature, definitions and case examples to delineate the meaning of that concept with respect to a certain domain or context. | Concept analysis is used to extend the theoretical meaning of a concept or to understand a conceptual practice problem [142,143]. In this case, concepts are cognitive descriptive meanings utilized for theoretical or practical purposes. | There are varied procedural techniques attributed to various authors such as Wilson [98], Walker & Avant [45], Chinn & (Jacobs) Kramer [145,146], Rodgers & Knafl, [46], Rodgers [99], Schwartz-Barcott & Kim [147], and Morse [47]. | Concept analysis generates a definition of a concept that may be used to operationalize phenomena for further research study [143] or theory development [144]. |
Concept Analysis | Data type: Most writings on concept analysis do not specify the data type. However, our scan of the methodological and empirical literature on concept analysis suggests that although the analytic approach in concept analysis is qualitative, quantitative study designs and data can be used to address the questions related to defining the meaning of a concept [e.g. 99, 141-142]. | Concept analysis is used to identify, clarify, and refine or define the meaning of a concept and can be used as a first step in theory development [47,144]. | Despite varied techniques, steps generally include: | |
1) Determine the purpose and aims. | ||||
Research question: Requires the researcher to isolate or identify a conceptual question or concept of interest. | 2) Delineate domains or boundaries of the concept. | |||
Quality appraisal: Quality appraisal is not typically attended to in concept analyses. Rather, researchers are interested in all instances of actual use of a concept (or surrogate terms) [142]. | 3) Draw on literature, dictionary meanings and/or cases. | |||
4) Analyze data sources to determine qualifying attributes. | ||||
5) Develop a prototype case and compare against contrary or borderline cases. | ||||
6) Test the practical significance. | ||||
7) Formulate defining features. | ||||
8) Relate to theoretical importance or practice application [46,141,148]. | ||||
EMERGING | What is it? Although no universal definition exists, there are some common elements of scoping reviews [129,149]. They are exploratory projects that systematically map the literature on a topic, identifying the key concepts, theories, sources of evidence, and gaps in the research. It involves systematically selecting, collecting and summarizing knowledge in a broad area [130]. | The purpose of a scoping review is to examine the extent, range and nature of research activity in an area. It is done to identify where there is sufficient evidence to conduct a full synthesis or to determine that insufficient evidence exists and additional primary research is needed [130,151]. It may be done for the purpose of disseminating research findings [63] or to clarify working definitions and the conceptual boundaries of a topic area [129]. | Arksey and O’Malley [63] recommend a 5 step process for conducting a scoping review: | The product of a scoping review will depend on the purpose for which it is conducted. In general, however, the narrative report provides an overview of all reviewed material. |
1) Identification of a broad research question. | ||||
Scoping Review | A scoping review is used to address broad topics where many different study designs and methods might be applicable. It may be conducted as part of an ongoing review, or as a stand-alone summary of research. Whereas a systematic review assesses a narrow range of quality-assessed studies to synthesize or aggregate findings, a scoping review assesses a much broader range of literature with a wide focus and does not synthesize or aggregate the findings [59]. | 2) Identification of relevant studies covering a wide breadth of literature and a variety of sources via databases, reference lists, and hand-searching key journals. This process may include consultation with key stakeholders. | The product generally includes: | |
3) Inclusion and exclusion criteria are identified as the review progresses. | 1) Basic numerical or narrative analysis of the extent, nature and distribution of the studies included with tables, graphs, and charts. | |||
Data type: Includes studies using any data type or method. May include empirical, theoretical or conceptual papers. Exclusion and inclusion criteria are inductively derived and based on relevance rather than on the quality of the primary studies or articles [150]. | 4) The data are sifted, sorted, compared and contrasted according to key issues and themes. Data are charted to allow for comparison and to ensure a uniform approach. | 2) Thematic organization of the literature (e.g., by intervention type, or by competing theoretical perspectives). | ||
5) Finally, the information is summarized and reported. Clear documentation of the methodology is important so that the reader can determine any potential reporting bias. | 3) Summary statement about what is known and not known (e.g., in the literature). | |||
Research question: The question is stated broadly and often becomes refined as the study progresses. One or more general questions may guide the review. | More recently, Levac et al. [129] have proposed recommendations to clarify and enhance each stage of the framework described above. | |||
Quality appraisal: The scoping review does not provide an appraisal of the quality of the evidence. It presents the existing literature without weighting the evidence in relation to specific interventions. | ||||
EMERGING | What is it? Rapid review of the literature provides a quick, rather than comprehensive, overview of the literature on a narrowly defined issue. Rapid review evolved out of a need to inform policy makers about issues and interventions in a timely manner [152]. It is often proposed as an intermediary step to be followed by a more comprehensive review. | The purpose is to produce a fast review of the literature, within a defined and usually limited time frame, on a question of immediate importance to a stakeholder group. | There is no standardized methodology as yet, but the depth and breadth of the review depends upon the specific purpose and the allotted time frame. Rapid reviews typically take one to nine months. | Typically a concise report is written for macro-level decision-makers that answer the specific review question. |
1) They begin with a needs assessment followed by formulation of a purpose statement and research question, definition of the context, and review of the literature [152,153,154]. | ||||
Rapid Review | Data type: The literature is often narrowly defined, focusing on a specific issue or a specific local, regional, or federal context [152]. It can include diverse study designs, methods, and data types as well as peer reviewed and gray literature. | 2) A review of the literature is streamlined in numerous ways including: | ||
· Accessing only published or online literature; | ||||
Research question: Rapid reviews require a thorough understanding of the intended audience and a specific, focused research question. | · Limiting by publication date, the number of databases, or language; | |||
·Searching electronic journals only; | ||||
Quality appraisal: Rapid reviews typically do not include an assessment of the quality of the literature, nor do they always include the views of experts and/or reviews by peers [152]. | · Narrowing to specific geographic settings or contexts; | |||
· Restricting the timeframe during which articles are assessed; | ||||
· Limiting contact with authors/industry or key stakeholders for clarification, follow-up, or input [152,153,154]. | ||||
3) References are retrieved, selected, summarized or synthesized, and a report is created. The public may be consulted about the results [152]. | ||||
It is important that those conducting a rapid review describe the methodology in detail to promote transparency, support transferability, and avoid misrepresenting the veracity of the findings [152]. | ||||
EMERGING | What is it? MNS is a new form of systematic review that addresses the issues of synthesizing a large and complex body of data from diverse and heterogeneous sources. At the same time, it is systematic in that it is conducted “according to an explicit, rigorous and transparent method” [[67], p. 418]. | The purpose is to summarize, synthesize and interpret a diverse body of literature from multiple traditions that use different methods, theoretical perspectives, and data types. | The steps to conduct a MNS [67,84,85] include the following: | The product of a MNS is: |
1) Planning Phase: | 1) A set of meta-narratives illustrating the story lines of various research traditions related to a common area or question; | |||
Meta-Narrative Synthesis (MNS) | The approach moves from logico-scientific reasoning (which underlies many approaches to synthesis) to narrative-interpretive reasoning. The unit of analysis for the synthesis is the unfolding “storyline” of a research tradition over time. Five key principles underlie the methodology: pragmatism, pluralism, historicity, contestation, and peer review. | · Assemble a multidisciplinary team, outline an initial broad question, and agree on outputs. | 2) An overarching conceptual framework that explains the phenomenon of interest. | |
Data type: This methodology involves the judicious combination of qualitative and quantitative evidence, and the theoretical and empirical literature. | 2) Search Phase: | |||
· Initially search by intuition, informal networking, browsing to map diversity of perspectives. | ||||
Research question: The original research question is outlined in a broad, open-ended format, and may shift and change through the process. | · Search for seminal papers. | |||
· Search for empirical papers in databases, hand searching key journals, and snowballing. | ||||
Quality appraisal: MNS uses the criteria of the research tradition of the primary study to judge the quality of the research, generally as set out in key sources within that tradition. | 3) Mapping Phase: | |||
· For each research tradition, identify key elements of the research paradigm, key actors and events in unfolding traditions, and prevailing language/imagery. | ||||
4) Appraisal Phase: | ||||
· Evaluate each study for validity/ relevance, extract and collate key results, group comparable studies. | ||||
5) Synthesis Phase: | ||||
· Identify all key dimensions of the problem/issue, provide a narrative account of each contribution, treat conflicting findings as higher order data and explain in terms of contestation between different paradigms from the original data. | ||||
6) Recommendations Phase: | ||||
· Summarize overall messages and relevant evidence; distil and discuss recommendations for policy, practice, and research. | ||||
EMERGING | What is it? A realist synthesis is a review of complex social interventions and programs that seek to unpack the mechanisms by which complex programs produce outcomes, and the context in which the relationship occurs. This is in contrast to systematic reviews, which aim to synthesize studies on whether interventions are effective. Realist synthesis seeks to answer the question: What works for whom, in what ways and under what circumstances? | The purpose of a realist synthesis is to guide program and policy development by providing decision makers with a set of program theories that identify potential policy levers for change. Within its explanatory intent, there are four general purposes: | Pawson et al. [69] identify 5 steps: | Pawson [68] explains that realist synthesis ends up with useful, middle-range theory. However, the product of a realist review combines theoretical understanding with empirical evidence. It focuses on explaining the relationships among the context in which an intervention takes place, the mechanisms by which it works, and the outcomes produced [68,69]. |
1) Reviewing for program theory integrity. | 1) Clarify scope: | Recommendations for dissemination and implementation are explicitly articulated. The result is a series of contextualized decision points that describe the contingencies of effectiveness. That is, a realist review provides an explanatory analysis that answers the original question of “what works for whom, in what circumstances, in what respects, and how” [[69], p. 21]. | ||
Realist Synthesis | This form of synthesis represents a review logic not a review technique [69]. Instead of a replicable method that follows rigid rules, the logic of realist review is based on principles. It reflects a shift away from an ontology of empirical realism to one of critical realism [155]. | 2) Reviewing to adjudicate between rival program theories. | · Identify the review question, nature of the intervention, circumstances for its use, and policy objectives; | |
3) Reviewing the same theory in different settings or with different populations. | · Refine the purpose of the review; | |||
Data type: There is no specific data preference but will include quantitative, qualitative and grey literature. Because the focus is on the mechanisms of action and their context, seemingly disparate bodies of literature and diverse methodologies are included. The focus is upon literature that emphasizes process with detailed descriptions of the interventions and context. | 4) Reviewing official expectations against actual practice [see 69, 107]. | · Make explicit the program theory or theories (e.g., the underlying assumptions about how the intervention is meant to work), synthesize theories, and design a theoretical framework. | ||
2) Search for evidence: | ||||
Research question: The review question is carefully articulated, prioritizing different aspects of an intervention [69]. It can be a broad question. | · Conduct an exploratory search; | |||
· Identify key program theories and refine inclusion criteria; | ||||
Quality appraisal: Realist review supports the principle that high quality evidence should be used but takes a different position than in systematic reviews on how the evidence is to be judged. It rejects a hierarchical approach to quality because multiple methods are needed to identify all aspects of the context, mechanisms and outcomes. Appraisal checklists are viewed skeptically because they cannot be applied evenly across the diverse study types and methods being reviewed. Thus, quality appraisal is seen as occurring in stages with a focus on the relevance of the study or article to the theory under consideration, and the extent to which an inference drawn has sufficient weight to make a credible contribution to the test of a particular intervention theory [69]. | · Purposively sample to test a subset of theories, with additional snowball sampling; | |||
· Search for new studies when review is almost completed. | ||||
3) Appraise primary studies and extract data: | ||||
· Use judgment to supplement critical appraisal checklists; | ||||
· Develop data extraction forms; | ||||
· Extract data. | ||||
4) Synthesize evidence and draw conclusions: | ||||
· Synthesize data to refine program theory; | ||||
· Let the purpose of the review lead the synthesis process; | ||||
· Use contradictory evidence to create insights about the impact of context; | ||||
· Present conclusions as a set of decision points. | ||||
5) Disseminate, implement and evaluate: | ||||
· Draft and test recommendations with key stakeholders focusing on what may influence policy; | ||||
· Work with policy makers and practitioners to apply recommendations; | ||||
· Evaluate the extent to which recommendations lead to program adjustments. | ||||
EMERGING | What is it? CIS is a methodology with an explicit orientation to theory generation, developed to respond to the need identified in the literature for rigorous methods to synthesize diverse types of research evidence generated by diverse methodologies [71] particularly when the body of evidence is very complex [72]. Thus, it was developed to address the limitations of conventional systematic review techniques. It involves an iterative process and recognizes the need for flexibility and reflexivity. It addresses the criticism that many approaches to syntheses are insufficiently critical and do not question the epistemological and normative assumptions reflected in the literature [72]. CIS is “sensitized to the kinds of processes involved in a conventional systematic review while drawing on a distinctively qualitative tradition of inquiry” [[72], p.35]. | The purpose of CIS is to develop an in-depth understanding of an issue/research question “by drawing on broadly relevant literature to develop concepts and theories that integrate those concepts” [[73], p. 71]. The overarching aim is to generate theory. | The developers of CIS explicitly reject a staged approach to the review. Rather, the processes are iterative, interactive, dynamic and recursive. It includes these general categories of activities [71,72]: | The product is a “synthesizing argument” that “links existing constructions from the findings to ‘synthetic constructs’ (new constructs generated through synthesis)” [[73], p. 71]. The synthesizing argument integrates evidence from across the studies in the review into a coherent theoretical framework [71,72]. This may be represented as a “conceptual map” that identifies the main synthetic constructs and illustrates the relationships among them [73]. |
1) Formulate the research question: | ||||
Critical Interpretive Synthesis (CIS) | Data type: CIS utilizes data from quantitative and qualitative empirical studies, conceptual and theoretical papers, reviews and commentaries. | · The question is not formulated in advance because the aim is to allow the definition of the phenomenon of interest to emerge from analysis. | ||
2) Search the literature: | ||||
Research question: It is neither possible nor desirable to specify a precise review question in advance. Rather the process is highly iterative and may not be finalized until the end of the review. | · Involves an organic approach using multiple search strategies (e.g., websites, reference chaining, contacting experts) in addition to a more structured approach; | |||
· Draw on the expertise of the team to identify relevant studies; | ||||
Quality appraisal: There is no hierarchy of designs for determining the quality of qualitative studies and, furthermore, no consensus exists on whether qualitative studies should even be assessed for quality [72]. Studies for inclusion are not selected on the basis of study design or methodological quality. Rather, papers that are relevant are prioritized. However, papers that are determined to be fatally flawed are excluded on the basis of a set of questions for determining quality [see 71]. Often, however, judgments about quality are deferred until the synthesis phase because even methodologically weak papers can provide important theoretical or conceptual insights [73]. | · Identify relevant papers that can form a sampling frame. | |||
3) Sample: | ||||
· May be selective and purposive, with emergent and flexible inclusion criteria; | ||||
· Ongoing selection is guided by theoretical sampling based on the emerging conceptual framework. | ||||
4) Determination of quality: | ||||
· See “quality appraisal” section. | ||||
5) Data extraction: | ||||
· Forms to guide this process can be useful, but with a huge database may be practically impossible; | ||||
· An informal process (highlighting text) can prove helpful. | ||||
6) Interpretive synthesis: | ||||
· Synthesis is based, in part, on the meta-ethnography strategies of reciprocal translational analysis, refutational synthesis, and lines of argument synthesis, but the authors greatly modified these to accommodate the diversity of literature (meta-ethnography used purely qualitative studies); | ||||
· The aim of the analysis is to produce a synthesizing argument, beginning with a detailed inspection of papers, gradually identifying recurring themes and developing a critique, constantly comparing concepts developed against the data and identifying the relationships among them. |
[1] | Vos T, Barber RM, Bell B, et al. (2013). Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013. Lancet 386: 743-800. |
[2] | 6. Bowman KG. (07) A research synthesis overview. Nurs Sci Q 20:171-176. |
[3] | 7. Stevens KR. (2001) Systematic reviews. The heart of evidence-based practice. AACN Clin Issues 12:529-5. |
[4] | 8. Upchurch S, Brosnan CA, Grimes DE. (2002) Teaching research synthesis to advanced practice nurses. J Nurs Educ :222-226. |
[5] | 9. Whittemore R. (200 Combining evidence in nursing research: Methods and implications. Nurs Res 54:56-62. |
[6] | 10. Feldman KA. (1971) Using the work of others: Some observations on reviewing and integrating. Sociol Educ 44:8102. |
[7] | 11. Streiner DL. (2005) I have the answer, now what's the question?: Why meta-analyses do not provide definitive solutions. Can J Psychiat 50:829-831. |
[8] | 12. Chatterji M. (2007) Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. Am J Eval 2 239-255. |
[9] | 13. Jackson N, Waters E. (2005) Criteria for the systematic review of health promotion and public health interventions. Health Promot Int 20: 367-374. |
[10] | 14. Torraco RJ. (2005) Writing integrative literature reviews: Guidelines and examples. Hum Resource Dev Rev 4:356-367. |
[11] | 15. Whittemore R, Knafl K. (2005). The integrative review: Updated methodology. J Adv Nurs 52:546-553. |
[12] | 16. Cooper, H. (2003) Editorial. Psychol Bull :3-9. |
[13] | 17. Chalmers I, Hedges LV, Cooper H. (2002) A brief history of research synthesis. Eval Health Prof 25:12-37. |
[14] | 18. Whittemore R, Knafl K. (2005). The integrative review: Updated methodology. J Adv Nurs 52:546-553. |
[15] | 19. Smith MC, Stullenbarger E. (1991) A prototype for integrative review and meta-analysis of nursing research. J Adv Nurs 16:1272-1283. |
[16] | 20. Cooper H. (1982) The integrative research review: A systematic approach. Beverly Hills: Sage. |
[17] | 21. Jackson GB. (1980) Methods for integrative reviews. Rev Educ Res 50: 438-460. |
[18] | 22. Glass GV. (1976) Primary, secondary, and meta-analysis of research. Educ Res 5:3-8. |
[19] | 23. Glass GV. (8) Integrating findings: The meta-analysis of research. Rev Res Educ 5:351-379. |
[20] | 24. Glass GV. (0) Meta–analysis at 25. Available from: http://www.gvglass.info/papers/meta25.html |
[21] | 25. Rosenthal R, DiMatteo MR. (2001) Meta-analysis: Recent developments in quantitative methods for literature reviews. Annu Rev Psychol 52:59-82. |
[22] | 26. Jenicek M. (1989) Meta-analysis in medicine: Where we are and where we want to go. J Clin Epidemiol 42:35-44. |
[23] | 27. Mullen PD, Ramirez G. (2006) The promise and pitfalls of systematic reviews. Annu Rev Publ Health 27:81-102. |
[24] | 28. Eysenck H. (1995) Meta-analysis or best evidence synthesis? J Eval Clin Prac 1:29-36. |
[25] | 29. Slavin RE. (1986) Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educ Res 15:5-11. |
[26] | 30. Slavin RE. (1995) Best evidence synthesis: An intelligent alternative to meta-analysis. J Clin Epidemiol 48:9-18. |
[27] | 31. Thorne S, Jensen L, Kearney MH, et al. (2004) Qualitative metasynthesis: Reflections on methodological orientation and ideological agenda. Qual Health Res 14:1342-1365. |
[28] | 32. Sandelowski M, Barroso J. (2003) Classifying the findings in qualitative studies. Qual Health Res 13:905-923. |
[29] | 33. Sandelowski M, Barroso J. (2007) Handbook for synthesizing qualitative research. New York: Springer Publishing Company Inc. |
[30] | 34. Annells M. (2005) Guest editorial: A qualitative quandary: Alternative representations and meta-synthesis. J Clin Nurs 14:535-536. |
[31] | 35. Barroso J, Gollop CJ, Sandelowski M, et al. (2003) The challenges of searching for and retrieving qualitative studies. Western J Nurs Res 25:153-178. |
[32] | 36. Estabrooks CA, Field PA, Morse JM. (1994) Aggregating qualitative findings: An approach to theory development. Qual Health Res 4:503-511. |
[33] | 37. Finfgeld DL. Metasynthesis (2003) The state of the art – so far. Qual Health Res 13:893-904. |
[34] | 38. Finlayson K, Dixon A. (2008) Qualitative meta-synthesis: A guide for the novice. Nurs Res 15:59-71. |
[35] | 39. Jensen LA, Allen MN. (1996) Meta-synthesis of qualitative findings. Qual Health Res 6:553-560. |
[36] | 40. Reis S, Hermoni D, Van-Raalte R, et al. (2007) Aggregation of qualitative studies – From theory to practice: Patient priorities and family medicine/general practice evaluations. Patient Educ Couns 65:214-222. |
[37] | 41. Sandelowski, M. (2006) "Meta-jeopardy": The crisis of representation in qualitative metasynthesis. Nurs Outlook 54:10-16. |
[38] | 42. Walsh D, Downe S. (2005) Meta-synthesis method for qualitative research: A literature review. J Adv Nurs 50:204-211. |
[39] | 43. Walsh D, Downe S. (2006) Appraising the quality of qualitative research. Midwifery 22:108-119. |
[40] | 44. Zimmer L. (2006) Qualitative meta-synthesis: A question of dialoguing with texts. J Adv Nurs 53:311-318. |
[41] | 45. Walker L, Avant K. (1983) Strategies for theory construction in nursing. Norwalk, CT: Appleton-Century-Crofts. |
[42] | 46. Rodgers BL, Knafl KA. (2000) Concept development in nursing: Foundations, techniques and applications. Philadelphia: Saunders. |
[43] | 47. Morse J. (1995) Exploring the theoretical basis of nursing using advanced techniques of concept analysis. Adv Nurs Sci 17:31-46. |
[44] | 50. Dixon-Woods M, Agarwal S, Jones D, et al. (2005). Synthesizing qualitative and quantitative evidence: A review of possible methods. J Health Serv Res Po 10:45-53. |
[45] | 51. Kearney MH. (1998). Ready-to-wear: Discovering grounded formal theory. Res Nurs Health 21:179-186. |
[46] | 53. Paterson BL, Dubouloz CJ, Chevrier J, et al. (2009) Conducting qualitative metasynthesis research: Insights from a metasynthesis project. Int J Qual Methods 8, 22-33. |
[47] | 54. Paterson BL, Thorne SE, Canam C, et al. (2001) Meta-study of qualitative health research: A practical guide to meta-analysis and meta-synthesis. Thousand Oaks, CA: Sage Publications Inc. |
[48] | 55. McCormick J, Rodney P, Varcoe C. (2003). Reinterpretations across studies: An approach to meta-analysis. Qual Health Res 13:933-944. |
[49] | 56. Schreiber R, Crooks D, Stern PN. (1997) Qualitative meta-analysis. In Morse JM, editor. Completing a qualitative project: Details and dialogue. Thousand Oaks, CA: Sage, 311-326. |
[50] | 57. Stern PN, Harris CC. (1985) Women's health and the self-care paradox: A model to guide self-care readiness. Health Care Women In 6:151-163. |
[51] | 58. Varcoe C, Rodney P, McCormick J. (2003) Health care relationships in context: An analysis of three ethnographies. Qual Health Res 13:957-973. |
[52] | 59. Weed M. (2008) A potential method for the interpretive synthesis of qualitative research: Issues in the development of `meta-interpretation`. Int J Soc Res Meth 11:13-28. |
[53] | 60. Atkins S, Lewin S, Smith H, et al. Conducting a meta-ethnography of qualitative literature: Lessons learnt. BMC Med Res Methodol 8:1471-2288. |
[54] | 61. Noblit GW, Hare RD. (1988) Meta-ethnography: Synthesizing qualitative studies. Newbury Park, CA: Sage. |
[55] | 62. Campbell R, Pound P, Pope C, et al. (2003) Evaluative meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med 56:671-684. |
[56] | 63. Arksey H, O'Malley L. (2005) Scoping studies: Towards a methodological framework. Int J Soc Res Meth 8:19-32. |
[57] | 64. Best A, Terpstra JL, Moore G, et al. (2009) Building knowledge integration systems for evidence-informed decisions. J Health Organ Manag 23:627-641. |
[58] | 65. Watt A, Cameron A, Sturm L, et al. (2008) Rapid reviews versus full systematic reviews: An inventory of current methods and practice in health technology assessment. Intl J Technol Ass 24:133-139. |
[59] | 66. Saul JE, Willis CD, Bitz J, et al. (2013) A time-responsive tool for informing policy making: Rapid realist review. Implement Sci 8. Available from: http://dx.doi.org/10.1186/1748-8-8-103 |
[60] | 67. Greenhalgh T, Robert G, Macfarlane F, et al. (2005) Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Soc Sci Med 61:417-430. |
[61] | 68. Pawson R. (2002) Evidence-based policy: The promise of 'realist synthesis.' Evaluation 8:340-358. |
[62] | 69. Pawson R, Greenhalgh T, Harvey G, et al. Realist view – A new method of systematic review designed for complex policy interventions. J Health Serv Res Po 10:21-34. |
[63] | 70. Sandelowski M, Barroso J, Voils CI. (2007) Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res Nurs Health 30:99-111. |
[64] | 71. Dixon Woods M, Kirk D, Agarwal S, et al. (2005) Vulnerable groups and access to health care: A critical interpretive review. Available from: http://www.netscc.ac.uk/hsdr/files/project/SDO_FR_08-1210-025_V01.pdf |
[65] | 72. Dixon Woods M, Cavers D, Agarwal S, et al. (2006) Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol doi:10.1186/1471-2288-6-35. |
[66] | 73. Entwistle V, Firnigl D, Ryan M, et al. (2012) Which experiences of health care delivery matter to service users and why? A critical interpretive synthesis and conceptual map. J Health Serv Res Po 17:70-78. |
[67] | 74. Heaton J, Corden A, Parker G. (2012) 'Continuity of care': A critical interpretive synthesis of how the concept was elaborated by a national research programme. Int J Integr Care 12. Available from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429143/ |
[68] | 75. Harden A, Garcia J, Oliver S, et al. (2004) Applying systematic review methods to studies of people's views: An example from public health research. J Epidemiol Commun H 58:794-800. |
[69] | 76. Voils CI, Sandelowski M, Barroso J, et al. (2008) Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Method 20:3-25. |
[70] | 77. Dixon Woods M, Fitzpatrick R. (2001) Qualitative research in systematic reviews. Brit Med J 323:765-766. |
[71] | 78. Goldsmith MR, Bankhead CR, Austoker J. (2007) Synthesising quantitative and qualitative research in evidence-based patient information. J Epidemiol Commun H 61:262-270. |
[72] | 79. Mays N, Pope C, Popay J. (2005) Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Po 10:6-20. |
[73] | 80. Popay J. (2003) Qualitative research and the epidemiological imagination: A vital relationship. Gac Sanit 17 Suppl 3: 58-63. |
[74] | 81. Cochrane. (2015) Available from http://www.cochrane.org/ |
[75] | 82. The Campbell Collaboration. (2015) What helps? What harms? Based on what evidence? Available from http://www.campbellcollaboration.org/ |
[76] | 83. Conn VS, Rantz MJ. (2003) Research methods: Managing primary study quality in meta-analyses. Res Nurs Health 26:322-333. |
[77] | 84. Greenhalgh T, Robert G, Macfarlane F, et al. (2004) Diffusion of innovations in service organizations: Systematic review and recommendations. MILBANK Q 82:581-629. |
[78] | 85. Greenhalgh T, Robert G, Bate P, et al. (2005) Diffusion of innovations in health service organizations: A systematic literature review. Malden, MA: Blackwell Publishing Inc. |
[79] | 86. Glaser BG, Strauss A. (1967) The discovery of grounded theory. Chicago: Aldine. |
[80] | 87. MacDonald M. (2001) Finding a critical perspective in grounded theory. In: Schreiber RS, Stern PN, editors. Using grounded theory in nursing. New York: Springer Publishing Company, 113-158. |
[81] | 89. Guba E, Lincoln Y. (1989) Fourth generation evaluation. Newbury Park: Sage Publications. |
[82] | 90. Sandelowski M. (1989) The problem of rigor in qualitative research. Adv Nurs Sci 8:27-37. |
[83] | 91. Moreira T. (2007) Entangled evidence: Knowledge making in systematic reviews in healthcare. Sociol Health Ill 29:180-197. |
[84] | 92. Voils CI, Barroso J, Hasselblad V, et al. (2007) In or out? Methodological considerations for including and excluding findings from a meta-analysis of predictors of antiretroviral adherence in HIV-positive women. J Adv Nurs 59:163-177. |
[85] | 93. Oliver S, Harden A, Rees R, et al. (2005) An emerging framework for including different types of evidence in systematic reviews for public policy. Evaluation 11:428-446. |
[86] | 94. Poland B, Frohlich KL, Cargo M. (2008) Context as a fundamental dimension of health promotion program evaluation. In: Potvin L, McQueen D, editors. Health promotion evaluation practices in the Americas. New York: Springer Science & Business Media, 299-318. |
[87] | 95. LaRocca R, Yost J, Dobbins M, et al. (2012) The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health 12:751. |
[88] | 97. Risjord M. (2009) Rethinking concept analysis. J Adv Nurs 65:684-691. |
[89] | 98. Wilson J. (1969) Thinking with concepts. Cambridge, England: Cambridge University Press. |
[90] | 99. Rodgers BL. (1993) Concept analysis: An evolutionary view. In: Rodgers BL, Knafl KA, editors. Concept Development in Nursing: Foundations, Techniques, and Applications. Philadelphia: W. B. Saunders Company, 73-92. |
[91] | 100.Bhaskar R. (1979) The possibility of naturalism. Atlantic Heights, NJ: Humanities Press. |
[92] | 101.Kirkevold M. (1997) Integrative nursing research: An important strategy to further the development of nursing science and nursing practice. J Adv Nurs 25:977-984. |
[93] | 102.Kelly M, Morgan A, Ellis S, et al. (2010) Evidence based public health: A review of the experience of the National Institute of Health and Clinical Excellence (NICE) of developing public health guidance in England. Soc Sci Med 71:1056–1062. |
[94] | 103.Droogan J, Cullum N. (1984) Systematic reviews in nursing. Int J Nurs Stud 35:13-22. |
[95] | 104.The Joanna Briggs Institute. (2015) Available from: http://www.joannabriggs.org/index.html |
[96] | 105.Biesta G. (2010) Pragmatism and the philosophical foundations of mixed methods research. In: Tashakkori A, Teddlie C, editors. Mixed methods in social and behavioral research (2nd ed.) Thousand Oaks, CA: Sage, 95-118. |
[97] | 107.Wong G, Westhorp G, Pawson R, et al. (2013) Greenhalgh T. Realist synthesis: RAMESES training materials. Available from: http://www.ramesesproject.org/media/Realist_reviews_training_materials.pdf |
[98] | 108.Greenhalgh T, Wong G. (2013) Training materials for meta-narrative reviews. Available from: http://www.ramesesproject.org/media/Meta_narrative_reviews_training_materials.pdf |
[99] | 109.Schwandt TA. (1) Constructivist, interpretivist approaches to human inquiry. In: Denzin NK, Lincoln YS, editors. Handbook of Qualitative Research. 2nd ed. Thousand Oaks, CA, 118-137. |
[100] | 110.Blumer H. (1969) Symbolic interactionism: Perspective and method. Englewood Cliffs, NJ: Prentice-Hall. |
[101] | 111.Milliken J, Schreiber R. (2001) Can you "do" symbolic interactionism without grounded theory? In: Schreiber RS, Stern PN, editors. Using grounded theory in nursing. New York: Springer, 177-190. |
[102] | 112.Münch R. (1994) Sociological theory: From the 1850s to the 1920s. Chicago: Nelson Hall Publishers. |
[103] | 113.Strauss A. (1993) Continual permutations of action. New York: Aldine de Gruyter. |
[104] | 114.Charmaz K. (2006) Constructing grounded theory: A practical guide through qualitative analysis. London: Sage. |
[105] | 115.Coryn CLS, Noakes LA, Westine CD, et al. (2011) A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval 32:199-226. |
[106] | 116.Powell S, Tod J, Cornwall J, et al. (2004) A systematic review of how theories explain learning behavior in school contexts. London: EPPI-Centre. |
[107] | 117.Baker WL, White MC, Cappelleri JC, et al. (2009) Understanding heterogeneity in meta-analysis: The role of meta-regression. Int J Clin Pract 63:1426-1434. |
[108] | 118.Thompson SG, Higgins JPT. (2001) How should meta-regression analysis be undertaken and interpreted? MRC Biostatistics Unit, Institute of Public Health, Cambridge. Available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16.4099&rep=rep1&type=pdf |
[109] | 119.Greco T, Biondi-Zoccai G, Saleh O, et al. (2015) The attractiveness of network meta-analysis: A comprehensive systematic and narrative review. Heart Lung Vessel 7:133-142. |
[110] | 120.Li T, Puhan MA, Vedula SS, Singh S, et al. (2011) Network meta-analysis – highly attractive but more methodological research is needed. BMC Medicine 9:79. Available from: http://www.biomedcentral.com/1741-7015/9/79 |
[111] | 121.Mills EJ, Thorlund K, Ioannidis JPA. (2013) Demystifying trial networks and network meta-analysis. BMJ 346:f2914. Available from: http://www.bmj.com/content/346/bmj.f2914 |
[112] | 122.Riley RD, Lambert PC, Abo-Zaid G. (2010) Meta-analysis of individual patient data: Rationale, conduct, and reporting. BMJ 340:c221. Available from: http://www.bmj.com/content/340/bmj.c221 |
[113] | 123.Thomas D, Radji S, Benedetti A. (2014) Systematic review of methods for individual patient data meta-analysis with binary outcomes. BMC Med Res Methodol 14:79. Available from: http://www.biomedcentral.com/1471-2288/14/79 |
[114] | 124.Hunt M. (1997) How science takes stock. New York: Russell Sage Foundation. |
[115] | 125.Sandelowski M, Voils C, Barroso J. (2007) Comparability work and the management of difference in research synthesis studies. Soc Sci Med 64:236-247. |
[116] | 126.Wong G, Greenhalgh T, Westhorp G, et al. (2013) RAMESES publication standards: Metanarrative reviews. J Adv Nurs 69:987-1004. |
[117] | 127.Wong G, Greenhalgh T, Westhorp G, et al. (2013) RAMESES publication standards: Realist syntheses. J Adv Nurs 69:1005-1022. |
[118] | 128.Wong G, Greenhalgh T, Westhorp G, et al. (2013) Quality standards for realist synthesis and meta-narrative reviews. Available from: http://betterevaluation.org/sites/default/files/Quality_standards_for_RS_and_MNR_v3final.pdf |
[119] | 129.Levac D, Colguhoun H, O'Brien KK. (2010) Scoping studies: Advancing the methodology. Implement Sci 5:69. |
[120] | 131.Sandelowski M, Trimble F, Woodard EK, et al. (2006) From synthesis to script: Transforming qualitative research findings for use in practice. Qual Health Res 16:1350-1370. |
[121] | 132.Grimshaw J, Santesso N, Cumpston M, et al. (2006) Knowledge for knowledge translation. The role of the Cochrane Collaboration. J Contin Educ Health 26:52-62. |
[122] | 133.Graham ID, Logan J, Harrison MB, et al. (2006) Lost in translation: time for a map. J Contin Educ Health 26:13–24. |
[123] | 134.Straus S, Tetroe J, Graham ID. (2009) Knowledge translation in health care: Moving from evidence to action. West Sussex: Wiley Blackwell. |
[124] | 135.Miles MB, Huberman AM. (1994) Qualitative data analysis: An expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage. |
[125] | 136.Oliver S, Harden A, Rees R, et al. (2008) Young people and mental health: Novel methods for systematic review of research on barriers and facilitators. Health Educ Res 23:770-790. |
[126] | 137.Harden A, Thomas J. (2005) Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Meth 8:257-271. |
[127] | 138.Dixon Woods M, Fitzpatrick R, Roberts K. (2001) Including qualitative research in systematic reviews. J Eval Clin Prac 7:125-133. |
[128] | 139.Anello C, Fleiss JL. (1995) Exploratory or analytic meta-analysis: Should we distinguish between them. J Clin Epidemiol 48:109-116. |
[129] | 140.Petticrew M. (2003) Why certain systematic reviews reach uncertain conclusions. Brit Med J 326:756-758. |
[130] | 141.Avant C. (2000) The Wilson method of concept analysis. In: Rodgers BL, Knafl KA, editors. Concept Development in Nursing: Foundations, Techniques, and Applications. 2nd ed. Philadelphia: W. B. Saunders Company, 55-64. |
[131] | 142.Knafl KA, Deatrick JA. (2000) Knowledge synthesis and concept development in nursing. In: Rodgers BL, Knafl KA, editors. Concept Development in Nursing: Foundations, Techniques, and Applications. 2nd ed. Philadelphia: W. B. Saunders Company, 39-54. |
[132] | 143.Rodgers BL. (1991) Using concept analysis to enhance clinical practice and research. Dimens Crit Care Nurs 10:28-34. |
[133] | 144.Duncan C, Cloutier JD, Bailey PH. (2007) Concept analysis: The importance of differentiating the ontological focus. J Adv Nurs 58:293-300. |
[134] | 145.Chinn P, Jacobs M. (1983) Theory and nursing: A systematic approach. 2nd ed. St. Louis: Mosby. |
[135] | 146.Chinn P, Kramer M. (1991) Theory and nursing: A systematic approach. 3rd ed. St. Louis: Mosby. |
[136] | 147.Schwartz-Barcott D, Kim H. (1993) An expansion and elaboration of the Hybrid Model of Concept Development in nursing. In Rodgers BL, Knafl KA, editors. Concept Development in Nursing: Foundations, Techniques, and Applications. Philadelphia: W. B. Saunders Company, 107-134. |
[137] | 148.Hupcey JE, Morse JM, Lenz ER, et al. (1996) Wilsonian methods of concept analysis: A critique. Sch Inq Nurs Prac 10:185-210. |
[138] | 149.Davis K, Drey N, Gould D. (2009) What are scoping studies? A review of the nursing literature. Int J Nurs Stud 46:-1400. |
[139] | 150.Valaitis R, Martin-Misener R, Wong S, et al. (2012) Strengthening Primary Health Care through Public Health and Primary Care Collaboration Team: Methods, strategies and technologies used to conduct a scoping literature review of collaboration between primary care and public health. Prim Health Care Res Dev 13:219-236. |
[140] | 151.Grant M, Booth A. (2009) A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 26:91-108. |
[141] | 152.Ganann R, Ciliska D, Thomas H. (2010) Expediting systematic reviews: Methods and implications of rapid reviews. Implement Sci doi:10.1186/1748-5908-5-56 |
[142] | 153.Khangura S, Konnyu K, Cushman R, et al. (2012) Evidence summaries: The evolution of a rapid review approach. Sys Rev doi: 10.1186/2046-4053-1-10 |
[143] | 154.Riley B, Norman CD, Best A. (2012) Knowledge integration in public health: A rapid review using system thinking. Evid Policy 8:417-431. |
[144] | 155.Mercille G. (2008) A realist approach to the systematic review. In: Potvin L, McQueen D, editors. Promotion evaluation practices in the Americas. New York: Springer Science & Business Media, 81-100. |
1. | Alina Weise, Roland Büchter, Dawid Pieper, Tim Mathes, Assessing context suitability (generalizability, external validity, applicability or transferability) of findings in evidence syntheses in healthcare—An integrative review of methodological guidance, 2020, 11, 1759-2879, 760, 10.1002/jrsm.1453 | |
2. | Niki M. Medendorp, Pomme E.A. van Maarschalkerweerd, Laxsini Murugesu, Joost G. Daams, Ellen M.A. Smets, Marij A. Hillen, The impact of communicating uncertain test results in cancer genetic counseling: A systematic mixed studies review, 2020, 103, 07383991, 1692, 10.1016/j.pec.2020.03.015 | |
3. | Md Mahbub Hossain, Umbrella Review as an Emerging Approach of Evidence Synthesis in Health Sciences: A Bibliometric Analysis, 2020, 1556-5068, 10.2139/ssrn.3551055 | |
4. | Lalit Kumar Radha Krishna, Lorraine Hui En Tan, Yun Ting Ong, Kuang Teck Tay, Jia Min Hee, Min Chiam, Elisha Wan Ying Chia, Krish Sheri, Xiu Hui Tan, Yao Hao Teo, Cheryl Shumin Kow, Stephen Mason, Ying Pin Toh, Enhancing Mentoring in Palliative Care: An Evidence Based Mentoring Framework, 2020, 7, 2382-1205, 238212052095764, 10.1177/2382120520957649 | |
5. | Kam-Fung Cheung, Michael G.H. Bell, Jyotirmoyee Bhattacharjya, Cybersecurity in logistics and supply chain management: An overview and future research directions, 2021, 146, 13665545, 102217, 10.1016/j.tre.2020.102217 | |
6. | Lisa Xin Ling Ngiam, Yun Ting Ong, Jun Xuan Ng, Joshua Tze Yin Kuek, Jeng Long Chia, Natalie Pei Xin Chan, Chong Yao Ho, Ahmad Bin Hanifah Marican Abdurrahman, Nur Haidah Ahmad Kamal, Clarissa Wei Shuen Cheong, Cheng Han Ng, Xiu Hui Tan, Lorraine Hui En Tan, Annelissa Mien Chew Chin, Stephen Mason, Muhammad Raihan Jumat, Min Chiam, Lalit Kumar Radha Krishna, Impact of Caring for Terminally Ill Children on Physicians: A Systematic Scoping Review, 2021, 38, 1049-9091, 396, 10.1177/1049909120950301 | |
7. | Amitabh Anand, Louise Brøns Kringelum, Charlotte Øland Madsen, Louisa Selivanovskikh, Interorganizational learning: a bibliometric review and research agenda, 2020, ahead-of-print, 0969-6474, 10.1108/TLO-02-2020-0023 | |
8. | Kaitlyn Tate, Sarah Hewko, Patrick McLane, Pamela Baxter, Karyn Perry, Susan Armijo-Olivo, Carole Estabrooks, Deb Gordon, Greta Cummings, Learning to lead: a review and synthesis of literature examining health care managers' use of knowledge, 2019, 24, 1355-8196, 57, 10.1177/1355819618786764 | |
9. | Sheryl Reimer-Kirkham, Barbara Astle, Ikponwosa Ero, Kristi Panchuk, Duncan Dixon, Albinism, spiritual and cultural practices, and implications for health, healthcare, and human rights: a scoping review, 2019, 34, 0968-7599, 747, 10.1080/09687599.2019.1566051 | |
10. | Joke Depraetere, Christophe Vandeviver, Tom Vander Beken, Ines Keygnaert, Big Boys Don’t Cry: A Critical Interpretive Synthesis of Male Sexual Victimization, 2020, 21, 1524-8380, 991, 10.1177/1524838018816979 | |
11. | Joshua Tze Yin Kuek, Lisa Xin Ling Ngiam, Nur Haidah Ahmad Kamal, Jeng Long Chia, Natalie Pei Xin Chan, Ahmad Bin Hanifah Marican Abdurrahman, Chong Yao Ho, Lorraine Hui En Tan, Jun Leng Goh, Michelle Shi Qing Khoo, Yun Ting Ong, Min Chiam, Annelissa Mien Chew Chin, Stephen Mason, Lalit Kumar Radha Krishna, The impact of caring for dying patients in intensive care units on a physician’s personhood: a systematic scoping review, 2020, 15, 1747-5341, 10.1186/s13010-020-00096-1 | |
12. | Katrin Gerber, Emma Maharaj, Bianca Brijnath, Josefine Antoniades, End-of-life care for older first-generation migrants: a scoping review, 2020, 2045-435X, bmjspcare-2020-002617, 10.1136/bmjspcare-2020-002617 | |
13. | Ronald B. Brown, Breakthrough Knowledge Synthesis in the Age of Google, 2020, 5, 2409-9287, 4, 10.3390/philosophies5010004 | |
14. | Kwabena Obeng Asiama, Winrich Voss, Rohan Bennett, Innocent Rubanje, Land consolidation activities in Sub-Saharan Africa towards the agenda 2030: A tale of three countries, 2021, 101, 02648377, 105140, 10.1016/j.landusepol.2020.105140 | |
15. | Chong Yao Ho, Cheryl Shumin Kow, Chin Howe Joshua Chia, Jia Ying Low, Yong Hao Melvin Lai, Sarah-Kei Lauw, Ashley Ern Hui How, Lorraine Hui En Tan, Xin Ling Lisa Ngiam, Natalie Pei Xin Chan, Tze Yin Joshua Kuek, Nur Haidah Ahmad Kamal, Jeng Long Chia, Ahmad Bin Hanifah Marican Abdurrahman, Min Chiam, Yun Ting Ong, Annelissa Mien Chew Chin, Ying Pin Toh, Stephen Mason, Lalit Kumar Radha Krishna, The impact of death and dying on the personhood of medical students: a systematic scoping review, 2020, 20, 1472-6920, 10.1186/s12909-020-02411-y | |
16. | Chaka Chaka, Translanguaging, Decoloniality, and the Global South: An Integrative Review Study, 2020, 25, 1812-5441, 6, 10.1080/18125441.2020.1802617 | |
17. | Joke Depraetere, Christophe Vandeviver, Ines Keygnaert, Tom Vander Beken, The critical interpretive synthesis: an assessment of reporting practices, 2020, 1364-5579, 1, 10.1080/13645579.2020.1799637 | |
18. | Elisha Wan Ying Chia, Huixin Huang, Sherill Goh, Marlyn Tracy Peries, Charlotte Cheuk Yiu Lee, Lorraine Hui En Tan, Michelle Shi Qing Khoo, Kuang Teck Tay, Yun Ting Ong, Wei Qiang Lim, Xiu Hui Tan, Yao Hao Teo, Cheryl Shumin Kow, Annelissa Mien Chew Chin, Min Chiam, Jamie Xuelian Zhou, Lalit Kumar Radha Krishna, A systematic scoping review of teaching and evaluating communications in the intensive care unit, 2021, 6, 24249270, 3, 10.29060/TAPS.2021-6-1/RA2351 | |
19. | Cheryl Shumin Kow, Yao Hao Teo, Yao Neng Teo, Keith Zi Yuan Chua, Elaine Li Ying Quah, Nur Haidah Binte Ahmad Kamal, Lorraine Hui En Tan, Clarissa Wei Shuen Cheong, Yun Ting Ong, Kuang Teck Tay, Min Chiam, Stephen Mason, Lalit Kumar Radha Krishna, A systematic scoping review of ethical issues in mentoring in medical schools, 2020, 20, 1472-6920, 10.1186/s12909-020-02169-3 | |
20. | Amitabh Anand, Piera Centobelli, Roberto Cerchione, Why should I share knowledge with others? A review-based framework on events leading to knowledge hiding, 2020, 33, 0953-4814, 379, 10.1108/JOCM-06-2019-0174 | |
21. | Onouma Thummapol, Tanya Park, Sylvia Barton, Exploring health services accessibility by indigenous women in Asia and identifying actions to improve it: a scoping review, 2020, 25, 1355-7858, 940, 10.1080/13557858.2018.1470607 | |
22. | H. Fernández-Sánchez, K. King, C.B. Enríquez-Hernández, Revisiones Sistemáticas Exploratorias como metodología para la síntesis del conocimiento científico, 2020, 17, 2395-8421, 10.22201/eneo.23958421e.2020.1.697 | |
23. | Alexander Gish, Niki Kiepek, Brenda Beagan, Methamphetamine use among gay men: An interpretive review of a non-sanctioned occupation, 2020, 27, 1442-7591, 26, 10.1080/14427591.2019.1643398 | |
24. | Welma Lubbe, Wilma ten Ham-Baloyi, Karlien Smit, The integrative literature review as a research method: A demonstration review of research on neurodevelopmental supportive care in preterm infants, 2020, 26, 13551841, 308, 10.1016/j.jnn.2020.04.006 | |
25. | Daniel Zhihao Hong, Annabelle Jia Sing Lim, Rei Tan, Yun Ting Ong, Anushka Pisupati, Eleanor Jia Xin Chong, Chrystie Wan Ning Quek, Jia Yin Lim, Jacquelin Jia Qi Ting, Min Chiam, Annelissa Mien Chew Chin, Alexia Sze Inn Lee, Limin Wijaya, Sandy Cook, Lalit Kumar Radha Krishna, A Systematic Scoping Review on Portfolios of Medical Educators, 2021, 8, 2382-1205, 238212052110003, 10.1177/23821205211000356 | |
26. | Xiu Hui Tan, Malia Alexandra Foo, Shaun Li He Lim, Marie Bernadette Xin Yi Lim, Annelissa Mien Chew Chin, Jamie Zhou, Min Chiam, Lalit Kumar Radha Krishna, Teaching and assessing communication skills in the postgraduate medical setting: a systematic scoping review, 2021, 21, 1472-6920, 10.1186/s12909-021-02892-5 | |
27. | Edison D. Macusi, Darshel Ester P. Estor, Elaine Q. Borazon, Misael B. Clapano, Mudjekeewis D. Santos, Environmental and Socioeconomic Impacts of Shrimp Farming in the Philippines: A Critical Analysis Using PRISMA, 2022, 14, 2071-1050, 2977, 10.3390/su14052977 | |
28. | Lynn Varagona, Nancy M. Ballard, Margot (Lisa) Hedenstrom, Virtue ethics in health care teams; its time has come: Review of the nursing virtue ethics literature, 2022, 30, 0966-0429, 2394, 10.1111/jonm.13757 | |
29. | Nail Sariyev, An assessment of selected tax burdens and reliefs of hidden champions: Theoretical comparison between Slovakia and Ireland, 2022, 11, 23066784, 346, 10.22495/jgrv11i2siart14 | |
30. | Sem Vanbelleghem, Melissa De Regge, Yves Van Nieuwenhove, Paul Gemmel, Barriers and Enablers of Second-Order Problem-Solving Behavior: How Nurses Can Break Away From the Workaround Culture, 2022, 31, 1063-8628, 130, 10.1097/QMH.0000000000000385 | |
31. | Jia Yin Lim, Simon Yew Kuang Ong, Chester Yan Hao Ng, Karis Li En Chan, Song Yi Elizabeth Anne Wu, Wei Zheng So, Glenn Jin Chong Tey, Yun Xiu Lam, Nicholas Lu Xin Gao, Yun Xue Lim, Ryan Yong Kiat Tay, Ian Tze Yong Leong, Nur Diana Abdul Rahman, Min Chiam, Crystal Lim, Gillian Li Gek Phua, Vengadasalam Murugam, Eng Koon Ong, Lalit Kumar Radha Krishna, A systematic scoping review of reflective writing in medical education, 2023, 23, 1472-6920, 10.1186/s12909-022-03924-4 | |
32. | Zhi H. Ong, Lorraine H. E. Tan, Haziratul Z. B. Ghazali, Yun T. Ong, Jeffrey W. H. Koh, Rachel Z. E. Ang, Chermaine Bok, Min Chiam, Alexia S. I. Lee, Annelissa M. C. Chin, Jamie X. Zhou, Gene W. H. Chan, Gayathri D. Nadarajan, Lalit K. R. Krishna, A Systematic Scoping Review on Pedagogical Strategies of Interprofessional Communication for Physicians in Emergency Medicine, 2021, 8, 2382-1205, 238212052110417, 10.1177/23821205211041794 | |
33. | Amanda Heffernan, Katrina MacDonald, Fiona Longmuir, The emotional intensity of educational leadership: a scoping review, 2022, 1360-3124, 1, 10.1080/13603124.2022.2042856 | |
34. | Carolyn L. Elias, Kevin M. Gorey, Online Social Networking among Clinically Depressed Young People: Scoping Review of Potentially Supportive or Harmful Behaviors, 2022, 40, 1522-8835, 79, 10.1080/15228835.2021.2010163 | |
35. | Ramana Piussi, Tora Berghdal, David Sundemo, Alberto Grassi, Stefano Zaffagnini, Mikael Sansone, Kristian Samuelsson, Eric Hamrin Senorski, Self-Reported Symptoms of Depression and Anxiety After ACL Injury: A Systematic Review, 2022, 10, 2325-9671, 232596712110664, 10.1177/23259671211066493 | |
36. | Saeedeh Rezaee Vessal, Amitabh Anand, 2022, 978-1-80262-358-1, 61, 10.1108/S2754-586520220000001004 | |
37. | Chong Yao Ho, Nicole-Ann Lim, Yun Ting Ong, Alexia Sze Inn Lee, Min Chiam, Gillian Phua Li Gek, Shiva Sarraf-Yazdi, Stephen Mason, Lalit Krishna, The impact of death and dying on the personhood of senior nurses at the National Cancer Centre Singapore (NCCS): a qualitative study, 2022, 21, 1472-684X, 10.1186/s12904-022-00974-9 | |
38. | Gabriël Rafaël Cantaert, Peter Pype, Martin Valcke, Emelien Lauwerier, Interprofessional Identity in Health and Social Care: Analysis and Synthesis of the Assumptions and Conceptions in the Literature, 2022, 19, 1660-4601, 14799, 10.3390/ijerph192214799 | |
39. | Onouma Thummapol, Werayuth Srithumsuk, Tanya Park, A scoping review of experiences and needs of older LGBTI adults in Asia, 2022, 34, 1053-8720, 403, 10.1080/10538720.2021.2006848 | |
40. | Adina Cismaru-Inescu, Stéphane Adam, Anne Nobels, Philippe Kempeneers, Marie Beaulieu, Christophe Vandeviver, Ines Keygnaert, Laurent Nisen, The Elephant in the Room – A Critical Interpretive Synthesis of Older Adults’ Sexuality, 2022, 34, 1931-7611, 90, 10.1080/19317611.2021.1958040 | |
41. | Charlotte Riordon, Sionnach Hendra, Christine Johnson, The politics of public health: A rapid review of the impact of public health reform on population health outcomes, 2021, 3, 2291-5796, 98, 10.25071/2291-5796.66 | |
42. | Qi Wang, Ying Zhu, Shitong Xie, Mohammad Golam Kibria, Qiangqiang Guo, Ahmed Atef Belal, Yanfei Li, Jingyi Zhang, Yaolong Chen, Holger J. Schünemann, Michael G. Wilson, Kehu Yang, John N. Lavis, Facilitators, barriers and strategies for health-system guidance implementation: a critical interpretive synthesis protocol, 2022, 20, 1478-4505, 10.1186/s12961-022-00908-0 | |
43. | Sadie Deschenes, Kaitlyn Tate, Shannon D. Scott, Diane Kunyk, Recommendations for navigating the experiences of moral distress: A scoping review, 2021, 122, 00207489, 104035, 10.1016/j.ijnurstu.2021.104035 | |
44. | Atousa Ghahramani, Maximilian de Courten, Maria Prokofieva, “The potential of social media in health promotion beyond creating awareness: an integrative review”, 2022, 22, 1471-2458, 10.1186/s12889-022-14885-0 | |
45. | Amitabh Anand, Ritu Tripathi, Anjana Karumathil, Tanvika Kalra, Applying systematic bibliometric methods to track a journal’s impact and review its knowledge contribution, 2022, 71, 2514-9342, 928, 10.1108/GKMC-04-2021-0064 | |
46. | Eleanor Jia Xin Chong, Ming Jia Wang, Jia Yin Lim, Grace Shen Shen, Misha Jing Yi Chow, Kai Kee Koh, Annabelle Jia Sing Lim, Daniel Zhihao Hong, Jacquelin Jia Qi Ting, Anushka Pisupati, Betrand Kai Yang Lam, Yun Ting Ong, Min Chiam, Stephen Mason, Lalit Kumar Radha Krishna, Si Ying Tan, Surgical portfolios: A systematic scoping review, 2022, 10, 26662620, 100107, 10.1016/j.sipas.2022.100107 | |
47. | Rei Tan, Jacquelin Jia Qi Ting, Daniel Zhihao Hong, Annabelle Jia Sing Lim, Yun Ting Ong, Anushka Pisupati, Eleanor Jia Xin Chong, Min Chiam, Alexia Sze Inn Lee, Laura Hui Shuen Tan, Annelissa Mien Chew Chin, Limin Wijaya, Warren Fong, Lalit Kumar Radha Krishna, Medical Student Portfolios: A Systematic Scoping Review, 2022, 9, 2382-1205, 238212052210760, 10.1177/23821205221076022 | |
48. | Shirin Ziaei, Anne Hammarström, What social determinants outside paid work are related to development of mental health during life? An integrative review of results from the Northern Swedish Cohort, 2021, 21, 1471-2458, 10.1186/s12889-021-12143-3 | |
49. | Katherine Sievert, Victoria Chen, Rebecca Voisin, Hope Johnson, Christine Parker, Mark Lawrence, Phillip Baker, Meat production and consumption for a healthy and sustainable Australian food system: Policy options and political dimensions, 2022, 33, 23525509, 674, 10.1016/j.spc.2022.08.007 | |
50. | Bijaya Pokharel, Jane Yelland, Leesa Hooker, Angela Taft, A Systematic Review of Culturally Competent Family Violence Responses to Women in Primary Care, 2023, 24, 1524-8380, 928, 10.1177/15248380211046968 | |
51. | Nail Sariyev, Janka Táborecká-Petrovičová, Performance measurement in a “hidden champion” company: An empirical study, 2022, 12, 20774303, 21, 10.22495/rgcv12i1p2 | |
52. | Rui Song Ryan Ong, Ruth Si Man Wong, Ryan Choon Hoe Chee, Chrystie Wan Ning Quek, Neha Burla, Caitlin Yuen Ling Loh, Yu An Wong, Amanda Kay-Lyn Chok, Andrea York Tiang Teo, Aiswarya Panda, Sarah Wye Kit Chan, Grace Shen Shen, Ning Teoh, Annelissa Mien Chew Chin, Lalit Kumar Radha Krishna, A systematic scoping review moral distress amongst medical students, 2022, 22, 1472-6920, 10.1186/s12909-022-03515-3 | |
53. | Mafole Sematlane, 2022, chapter 9, 9781668450345, 143, 10.4018/978-1-6684-5034-5.ch009 | |
54. | Birgit Muskat, Amitabh Anand, Christine Contessotto, Adrian Heng Tsai Tan, Guihyun Park, Team familiarity—Boon for routines, bane for innovation? A review and future research agenda, 2022, 32, 10534822, 100892, 10.1016/j.hrmr.2021.100892 | |
55. | Areni Altun, Helen Brown, Liz Sturgiss, Grant Russell, Evaluating chronic pain interventions in recent refugees and immigrant populations: A systematic review, 2022, 105, 07383991, 1152, 10.1016/j.pec.2021.08.021 | |
56. | Smruti Bulsari, Kiran Pandya, 2023, chapter 9, 9781668467459, 140, 10.4018/978-1-6684-6745-9.ch009 | |
57. | Farid Karimi, Stakeholders’ Risk Perceptions of Decarbonised Energy System: Insights into Patterns of Behaviour, 2021, 14, 1996-1073, 7205, 10.3390/en14217205 | |
58. | Kelly Jia Hui Teo, Mac Yu Kai Teo, Anushka Pisupati, Rui Song Ryan Ong, Chloe Keyi Goh, Claire Hui Xian Seah, You Ru Toh, Neha Burla, Natalie Song Yi Koh, Kuang Teck Tay, Yun Ting Ong, Min Chiam, Warren Fong, Limin Wijaya, Suzanne Pei Lin Goh, Lalit Kumar Radha Krishna, Assessing professional identity formation (PIF) amongst medical students in Oncology and Palliative Medicine postings: a SEBA guided scoping review, 2022, 21, 1472-684X, 10.1186/s12904-022-01090-4 | |
59. | Emily C. Koller, Christine Berg, Anorexia Nervosa: A Narrative Review and Conceptual Practice Model, 2021, 37, 0164-212X, 403, 10.1080/0164212X.2021.1957064 | |
60. | Paul Agu Igwe, Nnamdi O. Madichie, David Gamariel Rugara, Decolonising research approaches towards non-extractive research, 2022, 25, 1352-2752, 453, 10.1108/QMR-11-2021-0135 | |
61. | Sherill Goh, Ruth Si Man Wong, Elaine Li Ying Quah, Keith Zi Yuan Chua, Wei Qiang Lim, Aubrey Ding Rui Ng, Xiu Hui Tan, Cheryl Shumin Kow, Yao Hao Teo, Elijah Gin Lim, Anushka Pisupati, Eleanor Jia Xin Chong, Nur Haidah Ahmad Kamal, Lorraine Hui En Tan, Kuang Teck Tay, Yun Ting Ong, Min Chiam, Alexia Sze Inn Lee, Annelissa Mien Chew Chin, Stephen Mason, Lalit Kumar Radha Krishna, Mentoring in palliative medicine in the time of covid-19: a systematic scoping review, 2022, 22, 1472-6920, 10.1186/s12909-022-03409-4 | |
62. | Chrystie Wan Ning Quek, Ryan Rui Song Ong, Ruth Si Man Wong, Sarah Wye Kit Chan, Amanda Kay-Lyn Chok, Grace Shen Shen, Andrea York Tiang Teo, Aiswarya Panda, Neha Burla, Yu An Wong, Ryan Choon Hoe Chee, Caitlin Yuen Ling Loh, Kun Woo Lee, Gabrielle Hui Ning Tan, Ryan Emmanuel Jian Leong, Natalie Song Yi Koh, Yun Ting Ong, Annelissa Mien Chew Chin, Min Chiam, Crystal Lim, Xuelian Jamie Zhou, Simon Yew Kuang Ong, Eng Koon Ong, Lalit Kumar Radha Krishna, Systematic scoping review on moral distress among physicians, 2022, 12, 2044-6055, e064029, 10.1136/bmjopen-2022-064029 | |
63. | Abdul‐Ganiyu Fuseini, Lenore Ley, Helen Rawson, Bernice Redley, Debra Kerr, A systematic review of patient‐reported dignity and dignified care during acute hospital admission, 2022, 78, 0309-2402, 3540, 10.1111/jan.15370 | |
64. | Huixin Huang, Rachelle Qi En Toh, Christine Li Ling Chiang, Ashiley Annushri Thenpandiyan, Prachi Simran Vig, Randal Wei Liang Lee, Min Chiam, Alexia Sze Inn Lee, Vijayendra Ranjan Baral, Lalit Kumar Radha Krishna, Impact of Dying Neonates on Doctors' and Nurses' Personhood: A Systematic Scoping Review, 2022, 63, 08853924, e59, 10.1016/j.jpainsymman.2021.07.002 | |
65. | Federico Cavallaro, Silvio Nocera, Integration of passenger and freight transport: A concept-centric literature review, 2022, 43, 22105395, 100718, 10.1016/j.rtbm.2021.100718 | |
66. | Keith Zi Yuan Chua, Elaine Li Ying Quah, Yun Xue Lim, Chloe Keyi Goh, Jieyu Lim, Darius Wei Jun Wan, Simone Meiqi Ong, Chi Sum Chong, Kennan Zhi Guang Yeo, Laura Shih Hui Goh, Ray Meng See, Alexia Sze Inn Lee, Yun Ting Ong, Min Chiam, Eng Koon Ong, Jamie Xuelian Zhou, Crystal Lim, Simon Yew Kuang Ong, Lalit Krishna, A systematic scoping review on patients’ perceptions of dignity, 2022, 21, 1472-684X, 10.1186/s12904-022-01004-4 | |
67. | Bingjie Liu-Lastres, Beyond simple messaging: a review of crisis communication research in hospitality and tourism, 2022, 34, 0959-6119, 1959, 10.1108/IJCHM-11-2021-1404 | |
68. | Vaishnavi Venktaramana, Eleanor Kei Ying Loh, Clarissa Jing Wen Wong, Jun Wei Yeo, Andrea York Tiang Teo, Celest Sin Yu Chiam, Dillon Jie Ming Foo, Faith Teo, Jonathan Liang, Vijayprasanth Raveendran, Luke Cheng Lin Chng, Shiwei Xiao, Kevin Chong, Seng Leong Quek, Christine Li Ling Chiang, Rachelle Qi En Toh, Caleb Wei Hao Ng, Elijah Gin Lim, Shariel Leong, Kuang Teck Tay, Amos Chan, Elisha Wan Ying Chia, Laura Hui Shuen Tan, Yun Ting Ong, Krish Sheri, Jun Xuan Ng, Annelissa Mien Chew Chin, Jamie Xuelian Zhou, Min Chiam, Alexia Sze Inn Lee, Stephen Mason, Lalit Kumar Radha Krishna, A systematic scoping review of communication skills training in medical schools between 2000 and 2020, 2022, 44, 0142-159X, 997, 10.1080/0142159X.2022.2054693 | |
69. | Aamod D Shrestha, Johanne G Andersen, Dinesh Neupane, Sarita Ghimire, Christine Campbell, Per Kallestrup, Protocol for systematic literature review on implementation of cervical cancer screening and associated factors in Nepal from 2000 to 2018, 2020, 4, 2399-1623, 10.29392/001c.12505 | |
70. | Carlee Wilson, Allyson Jones, Kara Schick-Makaroff, Esther S. Kim, Understanding the impact of group therapy on health-related quality of life of people with Aphasia: a scoping review, 2021, 2050-571X, 1, 10.1080/2050571X.2021.1917216 | |
71. | Prachi Simran Vig, Jia Yin Lim, Randal Wei Liang Lee, Huixin Huang, Xiu Hui Tan, Wei Qiang Lim, Marie Bernadette Xin Yi Lim, Alexia Sze Inn Lee, Min Chiam, Crystal Lim, Vijayendra Ranjan Baral, Lalit Kumar Radha Krishna, Parental bereavement – impact of death of neonates and children under 12 years on personhood of parents: a systematic scoping review, 2021, 20, 1472-684X, 10.1186/s12904-021-00831-1 | |
72. | Elaine Li Ying Quah, Keith Zi Yuan Chua, Jun Kiat Lua, Darius Wei Jun Wan, Chi Sum Chong, Yun Xue Lim, Lalit Krishna, A Systematic Review of Stakeholder Perspectives of Dignity and Assisted Dying, 2023, 65, 08853924, e123, 10.1016/j.jpainsymman.2022.10.004 | |
73. | Onouma Thummapol, Sadaf Murad, Oluwakemi Amodu, Megan Kennedy, Exploring the Therapeutic Effects of Music Intervention Embedded With Binaural Beats on Health and Well-Being of Older People: A Scoping Review, 2024, 0733-4648, 10.1177/07334648241275965 | |
74. | Yao Hao Teo, Tan Ying Peh, Ahmad Bin Hanifah Marican Abdurrahman, Alexia Sze Inn Lee, Min Chiam, Warren Fong, Limin Wijaya, Lalit Kumar Radha Krishna, A modified Delphi approach to nurturing professionalism in postgraduate medical education in Singapore, 2024, 65, 0037-5675, 313, 10.11622/smedj.2021224 | |
75. | Natalie E. Pope, Emily A. Greenfield, Laura Keyes, Elizabeth Russell, A Review of Public Sector Engagement in Age-Friendly Community Initiatives, 2024, 0895-9420, 1, 10.1080/08959420.2024.2376934 | |
76. | Sunny C Okoroafor, Christmal Dela Christmals, Optimizing the roles of health workers to improve access to health services in Africa: an implementation framework for task shifting and sharing for policy and practice, 2023, 23, 1472-6963, 10.1186/s12913-023-09848-z | |
77. | Gábor Rónaföldi-Széll, Succession and gender dynamics in family firms – A systematic literature review and future research agenda, 2024, 55, 01330179, 59, 10.14267/VEZTUD.2024.07-08.06 | |
78. | Nami Kawakyu, Megan Coe, Bradley H. Wagenaar, Kenneth Sherr, Sarah Gimbel, Edward Nicol, Refining the Performance of Routine Information System Management (PRISM) framework for data use at the local level: An integrative review, 2023, 18, 1932-6203, e0287635, 10.1371/journal.pone.0287635 | |
79. | Michal Kaššaj, Tomáš Peráček, Synergies and Potential of Industry 4.0 and Automated Vehicles in Smart City Infrastructure, 2024, 14, 2076-3417, 3575, 10.3390/app14093575 | |
80. | Johan Högberg, Ramana Piussi, Johan Lövgren, Mathias Wernbom, Rebecca Simonsson, Kristian Samuelsson, Eric Hamrin Senorski, Restoring Knee Flexor Strength Symmetry Requires 2 Years After ACL Reconstruction, But Does It Matter for Second ACL Injuries? A Systematic Review and Meta-analysis, 2024, 10, 2198-9761, 10.1186/s40798-023-00666-5 | |
81. | Cara Evans, Julia Abelson, Nick Kates, Alice Cavanagh, John N. Lavis, Tommaso Martino, A Multilevel Framework for Complex Care: A Critical Interpretive Synthesis, 2023, 2023, 1365-2524, 1, 10.1155/2023/4487200 | |
82. | Jasmine Svantesson, Ramana Piussi, Elin Weissglas, Eleonor Svantesson, Alexandra Horvath, Erik Börjesson, Andy Williams, Robert Prill, Kristian Samuelsson, Eric Hamrin Senorski, Shedding light on the non-operative treatment of the forgotten side of the knee: rehabilitation of medial collateral ligament injuries—a systematic review, 2024, 10, 2055-7647, e001750, 10.1136/bmjsem-2023-001750 | |
83. | Zoe Guerrero, Dagmar Civišová, Petr Winkler, Mental health and access to care among the Roma population in Europe: A scoping review, 2024, 61, 1363-4615, 118, 10.1177/13634615231200853 | |
84. | Palaniappan Ganesh Nagappan, Samuel Brown, Alex McManus, Sarah Sayers, Shazia Absar, Sapphire Rou Xi Tan, Isla Kuhn, Edward Lau, Charlotte Tulinius, Changes in medical student attendance and its impact on student educational outcomes: a systematic review protocol, 2024, 14, 2044-6055, e078252, 10.1136/bmjopen-2023-078252 | |
85. | Begetayinoral Kussia Lahole, Tesfahun Simon, Wondimu Ewunetu, Wondafrash Kussia, Protocol for a systematic review and meta-analysis on intention to screen for cervical cancer and predictors among women of reproductive age in Ethiopia, 2024, 2, 2753-4294, e000585, 10.1136/bmjph-2023-000585 | |
86. | Katja Rogers, Teresa Hirzle, Sukran Karaosmanoglu, Paula Toledo Palomino, Ekaterina Durmanova, Seiji Isotani, Lennart E. Nacke, An Umbrella Review of Reporting Quality in CHI Systematic Reviews: Guiding Questions and Best Practices for HCI, 2024, 31, 1073-0516, 1, 10.1145/3685266 | |
87. | Gillian Li Gek Phua, Jasmine Lerk Juan Owyong, Ian Tze Yong Leong, Suzanne Goh, Nagavalli Somasundaram, Eileen Yi Ling Poon, Anupama Roy Chowdhury, Simon Yew Kuang Ong, Crystal Lim, Vengadasalam Murugam, Eng Koon Ong, Stephen Mason, Ruaridh Hill, Lalit Kumar Radha Krishna, A systematic scoping review of group reflection in medical education, 2024, 24, 1472-6920, 10.1186/s12909-024-05203-w | |
88. | Yvonne Fitzmaurice, Suzanne Beeke, Jytte Isaksen, Una Cunningham, Caroline Jagoe, Éidín Ní Shé, Ruth McMenamin, Communication partner training for student health and social care professionals engaging with people with stroke acquired communication difficulties: A protocol for a realist review., 2024, 6, 2515-4826, 60, 10.12688/hrbopenres.13783.2 | |
89. | Anne Hammarström, Hugo Westerlund, Urban Janlert, Pekka Virtanen, Shirin Ziaei, Per-Olof Östergren, How do labour market conditions explain the development of mental health over the life-course? A conceptual integration of the ecological model with life-course epidemiology in an integrative review of results from the Northern Swedish Cohort, 2024, 24, 1471-2458, 10.1186/s12889-024-18461-6 | |
90. | Jodie Gill, Alexis Jones, Klara Price, Egan Goodison, Philip Tyson, A mixed‐method systematic review of the perspectives of young people, carers and professionals on psychiatric diagnosis in childhood and adolescence, 2024, 37, 1073-6077, 10.1111/jcap.12459 | |
91. | Amalia Kouskoura, Eleni Kalliontzi, Dimitrios Skalkos, Ioannis Bakouros, Assessing the Key Factors Measuring Regional Competitiveness, 2024, 16, 2071-1050, 2574, 10.3390/su16062574 | |
92. | Anu-Riina Svenlin, Tiina Lehto-Lundén, Respite care from the child’s perspective – The Support Family Intervention in Finland, 2023, 47, 0308-5759, 138, 10.1177/03085759231176566 | |
93. | Ramana Piussi, Rebecca Simonson, Bálint Zsidai, Alberto Grassi, Jon Karlsson, Francesco Della Villa, Kristian Samuelsson, Eric Hamrin Senorski, Better Safe Than Sorry? A Systematic Review with Meta-analysis on Time to Return to Sport After ACL Reconstruction as a Risk Factor for Second ACL Injury, 2024, 54, 0190-6011, 161, 10.2519/jospt.2023.11977 | |
94. | Yun Xue Lim, Elaine Li Ying Quah, Keith Zi Yuan Chua, Casper Keegan Lin Ronggui, Ranitha Govindasamy, Simone Meiqi Ong, Eng Koon Ong, Gillian Li Gek Phua, Lalit Kumar Radha Krishna, A Systematic Scoping Review on Dignity Assessment Tools, 2024, 67, 08853924, e263, 10.1016/j.jpainsymman.2023.12.008 | |
95. | Elaine Li Ying Quah, Keith Zi Yuan Chua, Casper Keegan Ronggui Lin, Andrew Vimal Vijayan, Nur Amira Binte Abdul Hamid, Jasmine Lerk Juan Owyong, Neeta Satku, Natalie Woong, Crystal Lim, Gillian Li Gek Phua, Eng Koon Ong, Warren Fong, Lalit Kumar Radha Krishna, The role of patients’ stories in medicine: a systematic scoping review, 2023, 22, 1472-684X, 10.1186/s12904-023-01319-w | |
96. | Tim Stroh, Anne-Laure Mention, Cameron Duff, The impact of evolved psychological mechanisms on innovation and adoption: A systematic literature review, 2023, 125, 01664972, 102759, 10.1016/j.technovation.2023.102759 | |
97. | Neha Burla, Rui Song Ryan Ong, Ryan Choon Hoe Chee, Ruth Si Man Wong, Shao Yun Neo, Nur Amira Binte Abdul Hamid, Crystal Lim, Eng Koon Ong, Nagavalli Somasundaram, Lalit Kumar Radha Krishna, A systematic scoping review on group non-written reflections in medical education, 2024, 24, 1472-6920, 10.1186/s12909-024-06117-3 | |
98. | Rita Elaine Silver, Vinay Kumar, Deborah Chua Fengyi, Michael Tan Lip Thye, Johannis Auri Bin Abdul Aziz, For What and for Whom? Expanding the Role of Research Syntheses for Diverse Stakeholders, 2024, 53, 0013-189X, 464, 10.3102/0013189X241285414 | |
99. | Yvonne Fitzmaurice, Suzanne Beeke, Jytte Isaksen, Una Cunningham, Caroline Jagoe, Éidín Ní Shé, Ruth McMenamin, Communication partner training for student health and social care professionals engaging with people with stroke acquired communication difficulties: A realist review., 2023, 6, 2515-4826, 60, 10.12688/hrbopenres.13783.1 | |
100. | Danielle Allard, Tami Oliphant, With a Little Help from Our Friends: Applying a Critical Friends Orientation to Critical Literature Reviews, 2024, 61, 2373-9231, 13, 10.1002/pra2.1004 | |
101. | Sunday Adeola Ajagbe, Joseph Bamidele Awotunde, Ademola Temidayo Opadotun, Matthew O. Adigun, 2023, Chapter 4, 978-981-99-5084-3, 39, 10.1007/978-981-99-5085-0_4 | |
102. | Darius Wei Jun Wan, Laura Shih Hui Goh, Mac Yu Kai Teo, Celestine Jia Ling Loh, Gerald Hng Kai Yak, Joanna Jing Hui Lee, Nila Ravindran, Nur Diana Abdul Rahman, Min Chiam, Eng Koon Ong, Nagavalli Somasundaram, Ying Yin Lim, Gillian Li Gek Phua, Lalit Kumar Radha Krishna, Enhancing self-care education amongst medical students: a systematic scoping review, 2024, 24, 1472-6920, 10.1186/s12909-023-04965-z | |
103. | Tea Rosic, Elizabeth Lovell, Harriet MacMillan, Zainab Samaan, Rebecca L Morgan, Components of Outpatient Child and Youth Concurrent Disorders Programs: A Critical Interpretive Synthesis: Composantes des programmes de troubles concomitants des enfants et des jeunes ambulatoires : une synthèse interprétative critique, 2024, 69, 0706-7437, 381, 10.1177/07067437231212037 | |
104. | Katherine P. Kelly, Kathleen A. Knafl, Susan Keller, Pamela S. Hinds, Thematic expansion: A new strategy for theory development, 2021, 43, 0193-9459, 962, 10.1177/0193945920984795 | |
105. | Rahma K. Dewi, Sri Sumarni, Parenting style and family empowerment for children's growth and development: a systematic review, 2023, 14, 2038-9930, 8, 10.4081/jphia.2023.2582 | |
106. | Milou S. H. van Dieën, Wolter Paans, Massimo A. Mariani, Willem Dieperink, Fredrike Blokzijl, Roles and competencies of nurses and physicians in shared decision‐making in cardiac surgery: A scoping review, 2024, 80, 0309-2402, 60, 10.1111/jan.15811 | |
107. | Katarzyna Turoń, Factors Affecting Car-Sharing Services, 2023, 6, 2624-6511, 1185, 10.3390/smartcities6020057 | |
108. | Tiago Bigolin, Edson Talamini, Impacts of Climate Change Scenarios on the Corn and Soybean Double-Cropping System in Brazil, 2024, 12, 2225-1154, 42, 10.3390/cli12030042 | |
109. | Vitaliano Fiorillo, Biagio Maria Amico, Milk Quality and Economic Sustainability in Dairy Farming: A Systematic Review of Performance Indicators, 2024, 5, 2624-862X, 384, 10.3390/dairy5030031 | |
110. | Astrid Bertrand, Tiphaine Viard, Rafik Belloum, James R. Eagan, Winston Maxwell, 2023, On Selective, Mutable and Dialogic XAI: a Review of What Users Say about Different Types of Interactive Explanations, 9781450394215, 1, 10.1145/3544548.3581314 | |
111. | Jelena Komanchuk, Judy L. Cameron, Stefan Kurbatfinski, Linda Duffett-Leger, Nicole Letourneau, A realist review of digitally delivered child development assessment and screening tools: Psychometrics and considerations for future use, 2023, 183, 03783782, 105818, 10.1016/j.earlhumdev.2023.105818 | |
112. | Jonathan Zhen Liang, Donovan Kai Wei Ng, Vijayprasanth Raveendran, Mac Yu Kai Teo, Elaine Li Ying Quah, Keith Zi Yuan Chua, Jun Kiat Lua, Jasmine Lerk Juan Owyong, Andrew Vimal Vijayan, Nur Amira Binte Abdul Hamid, Ting Ting Yeoh, Eng Koon Ong, Gillian Li Gek Phua, Stephen Mason, Warren Fong, Crystal Lim, Natalie Woong, Simon Yew Kuang Ong, Lalit Kumar Radha Krishna, Marsa Gholamzadeh, The impact of online education during the Covid-19 pandemic on the professional identity formation of medical students: A systematic scoping review, 2024, 19, 1932-6203, e0296367, 10.1371/journal.pone.0296367 | |
113. | Rosa Angélica Priego Morales, Management leadership in the New Mexican School and teacher revaluation from the recognition of human talent: a scoping review, 2024, 29546168, 10.58763/rc2024197 | |
114. | Nofie Iman, Service modularity in the digital era: integrating environmental sustainability and post-pandemic insights, 2024, 1756-669X, 10.1108/IJQSS-05-2023-0061 | |
115. | Camila F. Pfeiffer, Wendy L. Magee, Rebecca Fülöp, Travis C. Nace, Candela Castro, Agustina Iturri, Jimena Franceschi, Gabriela Echauri, Liliana Gassull, María Julieta Russo, Exploring Music-Based Interventions for Executive Functioning and Emotional Well-Being in Stroke Rehabilitation: A Scoping Review, 2024, 5, 2673-4087, 565, 10.3390/neurosci5040041 | |
116. | Verónica Estruch-García, María Dolores Gil-Llario, Estefanía Ruiz-Palomino, Olga Fernández-García, Vicente Morell-Mengual, Zélia Ferreira Caçador Anastácio, Systematic Integrative Review: Sex Education for People with Autism Spectrum Disorder, 2025, 43, 0146-1044, 10.1007/s11195-024-09877-4 | |
117. | Kari Dee Vallury, Amanda Asher, Olivia Sarri, Nicola Sheeran, Jami Leichliter, Characteristics and determinants of quality non-directive pregnancy options counselling: a scoping review, 2025, 22, 1448-5028, 10.1071/SH24170 | |
118. | Kirsten M. Greer, Kyla M. Cary, Morgan E. PettyJohn, Gianna Casaburo, Carin Graves, Kristen N. Jozkowski, A Qualitative Meta-Synthesis of Women’s Experiences of Internalized Sociocultural Sexual Pressure, 2025, 0022-4499, 1, 10.1080/00224499.2025.2451144 | |
119. | Wei Chen Chang, Wan Zi Lin, Wen-Zhi Chen, Wen-Ko Chiou, Oliver Shinbou Lin, Design-driven innovation in the public sector: insights from case studies of initiatives in Taiwan, 2025, 1751-6234, 1, 10.1080/17516234.2025.2457166 | |
120. | Hiroyuki Suzuki, Yusuke Tsuboko, Manabu Tamura, Ken Masamune, Kiotaka Iwasaki, Synthesis of the clinical utilities and issues of intraoperative imaging devices in clinical reports: a systematic review and thematic synthesis, 2025, 25, 1472-6947, 10.1186/s12911-025-02915-x | |
121. | Charles Birungi, Michael A. Obst, Using economics to advance health equity: What we know, don't know and need to know (but may never know) from Markus Haacker, 2025, 2, 2994-4694, 10.1080/29944694.2025.2456791 | |
122. | Abdullah Almunifi, Impact of Artificial Intelligence on Metabolic Bariatric Surgery (MBS) and Minimally Invasive Surgery (MIS): A Literature Review, 2025, Volume 17, 1178-7082, 161, 10.2147/OAS.S482690 |
Types of Research Synthesis | Definition | Data Types Used | Products | Examples |
1. Conventional Synthesis | Older forms of review with less-systematic examination, critique, and synthesis of the literature on a mature topic for re-conceptulization or on a new topic for preliminary conceptualization | · Quantitative studies | · Narrative expression and summary | · Integrative review [14,18,19,20,21] |
· Qualitative studies | · Tables, charts, graphical displays, diagrams and maps | · Narrative synthesis [16] | ||
· Other types of data e.g., theoretical literature, policy | · Theory, theoretical/conceptual frameworks, or conceptual maps | · Conventional literature review[17] | ||
2. Quantitative Synthesis | Combining, aggregating, or integrating quantitative empirical research with data expressed in numeric form | · Quantitative studies | · Narrative expression and summary | · Systematic review [13,26,27] |
· Mathematical scores | · Meta-analysis [22,23,24,25] | |||
· Statements of generalizability | · Best evidence synthesis [28,29,30] | |||
3. Qualitative Synthesis | Combining, aggregating, or integrating qualitative empirical research and/or theoretical work expressed in narrative form | · Qualitative studies | · Narrative expression and summary | · Meta-synthesis [31,32,33,34,35,36,37,38,39,40,41,42,43,44] |
· Other types of data e.g., theoretical literature | · Theory, theoretical/conceptual frameworks, or conceptual maps | · Concept analysis [45,46,47] | ||
· A definition | · Grounded formal theory [37,48,49,50,51,52] | |||
· Meta-study [31,37,48,49,50,53,54] | ||||
· Meta-analysis [37,38,55,56,57,58] | ||||
· Meta-interpretation [59] | ||||
· Meta-ethnography [49,50,60,61,62] | ||||
4. Emerging Synthesis | Newer syntheses that provide a systematic approach to synthesizing varied literature in a topic area that includes diverse data types | · Quantitative studies | · Narrative expression and summary | · Scoping review [63] |
· Qualitative studies | · Tables, charts, graphical displays, diagrams and maps | · Rapid review [64,65] | ||
· Other types of data e.g., theoretical work, grey literature, editorials, commentaries, policy, evaluations | · Mathematical scores | · Rapid realist review [66] | ||
· Theory, theoretical/conceptual frameworks, or conceptual maps | · Meta-narrative synthesis [67,68] | |||
· A report written for decision-makers | · Realist synthesis [68,69] | |||
· Meta-summary [70] | ||||
· Critical interpretive synthesis [71,72,73,74] | ||||
· Other types of mixed-research synthesis [49,72,75,76,77,78,79] |
Types of Research Synthesis | Key Characteristics | Purpose | Methods | Product |
CONVEN-TIONAL | What is it? “The integrative literature review is a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated” [[14], p.356]. | Integrative reviews are used to address mature topics in order to re-conceptualize the expanding and diverse literature on the topic. They are also used to comprehensively review new topics in need of preliminary conceptualization [14]. | Integrative reviews generally contain similar steps [14,18], which include the following: | Conclusions are often presented in a table/diagram. Explicit details from primary sources to support conclusions must be provided to demonstrate a logical chain of evidence. |
1) Identify a clear problem. | ||||
Integrative Review | Data type: Integrative literature reviews include studies using diverse methodologies (i.e., experimental and non-experimental research, as well as qualitative research) in order to more fully understand a phenomenon of interest. It may also include theoretical and empirical literature. | Integrative reviews should ultimately present the “state of the art” of knowledge, depict the breadth and depth of the topic, and contribute to greater understanding of the phenomenon [18]. | 2) Determine the variables of interest (e.g., population, concept). | Torraco [14] suggests they can be represented in four forms: |
3) State a specific research purpose. | 1) A research agenda, | |||
Research question: Start by clearly identifying the problem that the review is addressing and the purpose of the review. There usually is not a specific research question, but rather a research purpose. | 4) Define and clearly document a search strategy. Aim to locate as many of the existing studies as possible. Purposive sampling may be used along with a more comprehensive approach. | 2) A taxonomy or conceptual classification of constructs, | ||
5) Critically evaluate the quality of primary reviews depending on the sampling frame used in the integrative review. | 3) Alternative models/conceptual framework, and | |||
Quality appraisal: The quality of primary sources may be appraised using broad criteria. How quality is evaluated will depend upon the sampling frame [18]. | 6) Identify a systematic analytic method. The constant comparative method [86,135] is one overarching approach commonly used. | 4) Metatheory. | ||
7) Keep a record of the process of data analysis (e.g., hunches, decisions, ideas about interpretation). | Results should emphasize implications for policy/practice [18]. | |||
8) State methodological limitations. | ||||
QUANT-ITATIVE | What is it? A SR is a review of literature that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies. Conducting a SR is analogous to conducting a primary study in that there are steps and protocols. It may or may not be done in conjunction with a meta-analysis. | The purpose of a SR is to integrate empirical research for the purpose of generalizing from a group of studies. The reviewer is also seeking to discover the limits of generalization [27]. | A number of authors have provided guidelines for conducting a SR [27] but they generally contain similar steps: | The products of a SR may include: |
1) Specify study aims and define research question. | 1) A statement about the relative “effectiveness” of health care interventions, or about the appropriateness, feasibility, or meaningfulness of findings for particular purposes; | |||
In Cochrane [81], a SR is identified as the highest form of evidence in support of interventions. By contrast, the Joanna Briggs Institute [104] does not define a SR as necessarily the highest form of evidence. | Often, the review focuses on questions of intervention effectiveness. Thus, the intent is to summarize across studies to obtain a summative judgment about the effectiveness of interventions. However, the Joanna Briggs Institute [104] suggests that for nursing, there is a concern not just with effectiveness but also with questions of appropriateness, meaningfulness and feasibility of health practices and delivery methods. Thus, SR’s may have purposes other than to assess the effectiveness of interventions. | 2) Set inclusion criteria for evidence. | 2) A statement about the strength of the relationship between a particular intervention and specific outcomes. | |
Systematic Review (SR) | 3) Design search strategy. | 3) More recently, the product might be a statement about the convergence of theoretical perspectives on a topic. | ||
As noted below, a meta-analysis is always a SR, but a SR is not always a meta-analysis. | 4) Screen potential evidence against criteria for assessing quality. | 4) When done in conjunction with meta-analysis, the product is a mathematic score that represents the statements above. | ||
5) Design data collection protocol. | ||||
Data type: There is nothing that specifies data have to be quantitative, and the definition can apply to qualitative findings. Generally, however, the term has been used most frequently to apply to reviews of quantitative studies - traditional RCTs and experimental or quasi-experimental designs. More recently, both the Campbell and the Cochrane collaborations have been grappling with the need to, and the process of, integrating qualitative research into a SR. A number of studies have been published that do this [13,75,78,135,136,137,138]. | 6) Select appropriate metric to represent the magnitude of findings and assess likelihood they are due to chance. | |||
7) Code the primary studies. | ||||
Research question: A well-defined research question is required. | 8) Analyze and display data using appropriate methods. | |||
9) Draw conclusions based on data. | ||||
Quality appraisal: The Quality Appraisal section under MA above also applies to SR. Some researchers are developing standard reliable and valid quality appraisal tools to judge the quality of primary studies but there remains no consensus on which tools should be used. The Joanna Briggs Institute [104] has developed their own criteria to ensure that only the highest quality studies are included in SRs for nursing, but they hold that studies from any methodological position are relevant. | 10) Discuss alternate interpretations in light of studies’ strengths and limitations. | |||
QUANT-ITATIVE | What is it? M-A is the statistical analysis of a large collection of results from individual studies (usually interventions) for the purposes of integrating the findings, based on conversion to a common metric (effect size) to determine the overall effect and its magnitude. The term was coined by Gene Glass [22,23] but dates back to 1904 [17]. A M-A is always a SR (see above). | Analytic M-As are conducted for the purpose of summarizing and integrating the results of individual primary studies to increase the power for detecting intervention effects, which may be small and insignificant in the individual studies [139,140]. | Specific steps include [25]: | The product for M-A includes a narrative summary of the findings with a conclusion about the effectiveness of interventions. |
1) Define the dependent and independent variables of interest. | 1) Analytic Products: | |||
Meta-Analysis (M-A) | Data type: Data are from quantitative research studies and findings, primarily randomized control trials. Increasingly there is use of experimental, quasi-experimental and some types of observational studies. Each primary study is abstracted and coded into a database. | Exploratory M-As are conducted to resolve controversy in a field or to pose and answer new questions. The main concern is to explain the variation in effect sizes. | 2) Collect the studies in a systematic way attempting to find all published and unpublished studies. | · Graphical displays of the data and a table that displays the key elements of each study. |
3) Read methods carefully and if effect sizes are not reported, identify articles for information to calculate these. | 2) Final product: | |||
Research question: A clear, well-defined research question or hypothesis is required. | 4) Examine variability among the obtained effect sizes informally with graphs and charts, to identify the possibility that moderator variables may account for the variability. | · A mathematic score that represents the strength of the effect of an intervention or the relationships between two variables. | ||
5) Combine effects using several measures of their central tendency and explore reasons for differences if found. | · Identification of variables that moderate or mediate the effects or relationships. | |||
Quality appraisal: Articles are usually appraised according to a set of pre-defined criteria but these criteria vary considerably and there are many methodological limitations [83]. Lower quality studies are not necessarily excluded and there is some debate about whether these should be included [24,29]. When lower quality studies are included, the validity of the findings is often discussed in relation to the study quality. | 6) Examine the significance level of the indices of central tendency, usually employing confidence intervals around unweighted mean effect sizes in a random effects model. | |||
7) Using an examination of the binomial effect size display, evaluate the importance of the obtained effect size. | ||||
QUAL-ITATIVE | What is it? “Meta-study is a research approach involving analysis of the theory, methods, and findings of qualitative research and the synthesis of these insights into new ways of thinking about phenomenon” [[54], p. 1]. | Analysis of research findings, methods, and theory across qualitative studies are compared and contrasted to create a new interpretation [53]. | Paterson et al. [54] propose a clear set of techniques: | Through the three meta-study processes, researchers create a “meta-synthesis” which brings together ideas to develop a mid-range theory as the product. |
1) Choose an analytic approach (e.g. grounded theory, thematic analysis). | ||||
Meta-Study | Data type: Three analytic components are undertaken prior to synthesis. Data includes qualitative findings (meta-data), research methods (meta-method), and/or philosophical/theoretical perspectives (meta-theory). | 2) Use specific sampling techniques according to inclusion/exclusion criteria, including searching for disconfirming cases that challenge the emerging theory. | ||
3) Regardless of approach, group studies according to characteristics (e.g., disease) and treat each group as a case [49]. | ||||
Research question: A relevant, well-defined research question is used. | 4) Engage in three distinct types of analysis, i.e. meta-data, meta-study, meta-theory (may be undertaken concurrently). | |||
5) Synthesize analysis into a theory. | ||||
Critical appraisal: According to Paterson et al. [54], primary articles are appraised according to specific criteria; however the specific appraisal will depend on the requirements of the meta-study. Studies of poor quality will be excluded. Data from included studies may also be excluded if reported themes are not supported by the presented data. | ||||
QUAL-ITATIVE | What is it? Meta-ethnography entails choosing relevant empirical studies to synthesize through repetitive reading while noting metaphors [61,62]. Noblit and Hare explain that “metaphors” refer to “themes, perspectives, organizers, and/or concepts revealed by qualitative studies” [[61], p. 15]. These metaphors are then used as data for the synthesis through (at least) one of three strategies including reciprocal translation, refutational synthesis, and/or line of argument syntheses. A meta-ethnographic synthesis is the creation of interpretive (abstract) explanations that are essentially metaphoric. The goal is to create, in a reduced form, a representation of the abstraction through metaphor, all the while preserving the relationships between concepts [61]. | To synthesize qualitative studies through a building of “comparative understanding” [[61], p. 22] so that the result is greater than the sum of the parts. | Methods used in meta-ethnography generally following the following: | The product of a meta-ethnography is a mid-range theory that has greater explanatory power than could be otherwise achieved in a conventional literature review. |
· Frame the study broadly by an interest, aim or purpose and ultimately, a research question. | ||||
Meta-Ethno-graphy | Data type: Qualitative research studies and findings on a specific topic. | Noblit and Hare summarize that meta-ethnography is “a form of synthesis for ethnographic or other interpretive studies. It enables us to talk to each other about our studies; to communicate to policy makers, concerned citizens, and scholars what interpretive research reveals; and to reflect on our collective craft and the place of our own studies within it” [[61], p.14]. | · Create inclusion/exclusion criteria. | |
· Conduct a review of the literature based on who the audience will be, what is credible to the audience, what accounts are available, and what the researchers’ interests are in the study [61]. | ||||
Research question: An “intellectual interest” [61], p.26] begins the process. Then, a relevant research question, aim, or purpose is developed. | · Identify all the appropriate studies in a field through repeated readings. | |||
Quality appraisal: Researchers are divided on the merits of critical appraisal and whether or not it should be a standard element in meta-ethnography [60]. Some researchers choose to follow pre-determined criteria based on critical appraisal [e.g., 62], whereas others do not critically appraise. | Noblit and Hare [61] identified three possible analysis strategies (all do not have to be completed): | |||
1) Reciprocal translational analysis. Key themes, metaphors, or concepts are identified and translated into each other to create the most representative concept. | ||||
2) Refutational synthesis. Contradictions between key themes, metaphors, or concepts are examined and explained. | ||||
3) Lines of argument synthesis. Interpretation is created from comparison of findings across distinct studies. | ||||
QUAL-ITATIVE | What is it? A grounded formal theory (GFT) is a synthesis of substantive grounded theories (GTs) to produce a higher order, more abstract theory that goes beyond the specifics of the original theories. GFT takes into account the conditions under which the primary study data were collected and analyzed to develop a more generalized and abstract model [31]. | The intent of GFT is to expand the applicability of individual GTs by synthesizing the findings to provide a broad meaning that is based in data and is applicable to people who experience a common phenomenon across populations and context [51]. | GFT uses the same methods that were used to create the original GTs in the synthesis [48,51]. Specific elements of the analytic process include: | A GFT is a mid-range GT that has “fit, work and grab”: that is, it fits the data (concepts and categories from primary studies), works to explain the phenomenon under review, and resonates with the readers’ experiences and understandings. |
1) Theoretical sampling - sample size is determined through purposive and theoretical sampling strategies to answer emerging questions [37,51]. | ||||
Data type: Substantive GTs were originally constructed using the methodology developed by Glaser & Strauss [86]. While some synthesis approaches emphasize including all possible primary GT studies, the concept of saturation in GFT (see Methods column) allows limiting the number of reviewed papers to emphasize robustness rather than completeness [50]. | The focus is on the conditions under which theoretical generalizations apply. GFT aims “to bring cultural and individual differences into dialogue with each other by seeking a metaphor through which those differences can be understood by others” [[31], p.1354]. | 2) Constant comparative analysis -the analyst identifies concepts and their relationship with other data, and compares theoretical ideas to prior and subsequent data. | Thorne et al. suggest that a GFT is “an artistic explanation that works for now, a model created on the basis of limited materials and a specific, situated perspective within known and unconscious limits of representation” [31], p. 1354]. | |
Grounded Formal Theory (GFT) | 3) Memoing - documentation of hunches, decisions, and modifications during analysis. | |||
Research question: GFT begins with a phenomenon of focus [51]. Analytic questions and the overall research question emerge throughout the process. | 4) Saturation - the point at which continued data collection and analysis brings only repeated concepts or ideas. | |||
5) Coding - begins at a descriptive level and progresses towards a more abstract and theoretical level. Findings are synthesized and translated across studies. | ||||
Quality appraisal: There is no discussion in the GFT literature about critically appraising the studies to be included. However, the nature of the analytic process suggests that critical appraisal may not be relevant. The authenticity and accuracy of data in a GFT are not an issue because, for the purposes of generating theory, what is important is the conceptual category and not the accuracy of the evidence. The constant comparative method of GFT will correct for such inaccuracies because each concept must “earn” its way into the theory by repeatedly showing up [67,68]. | ||||
QUAL-ITATIVE | What is it? Concept analysis is a systematic procedure to extract attributes of a concept from literature, definitions and case examples to delineate the meaning of that concept with respect to a certain domain or context. | Concept analysis is used to extend the theoretical meaning of a concept or to understand a conceptual practice problem [142,143]. In this case, concepts are cognitive descriptive meanings utilized for theoretical or practical purposes. | There are varied procedural techniques attributed to various authors such as Wilson [98], Walker & Avant [45], Chinn & (Jacobs) Kramer [145,146], Rodgers & Knafl, [46], Rodgers [99], Schwartz-Barcott & Kim [147], and Morse [47]. | Concept analysis generates a definition of a concept that may be used to operationalize phenomena for further research study [143] or theory development [144]. |
Concept Analysis | Data type: Most writings on concept analysis do not specify the data type. However, our scan of the methodological and empirical literature on concept analysis suggests that although the analytic approach in concept analysis is qualitative, quantitative study designs and data can be used to address the questions related to defining the meaning of a concept [e.g. 99, 141-142]. | Concept analysis is used to identify, clarify, and refine or define the meaning of a concept and can be used as a first step in theory development [47,144]. | Despite varied techniques, steps generally include: | |
1) Determine the purpose and aims. | ||||
Research question: Requires the researcher to isolate or identify a conceptual question or concept of interest. | 2) Delineate domains or boundaries of the concept. | |||
Quality appraisal: Quality appraisal is not typically attended to in concept analyses. Rather, researchers are interested in all instances of actual use of a concept (or surrogate terms) [142]. | 3) Draw on literature, dictionary meanings and/or cases. | |||
4) Analyze data sources to determine qualifying attributes. | ||||
5) Develop a prototype case and compare against contrary or borderline cases. | ||||
6) Test the practical significance. | ||||
7) Formulate defining features. | ||||
8) Relate to theoretical importance or practice application [46,141,148]. | ||||
EMERGING | What is it? Although no universal definition exists, there are some common elements of scoping reviews [129,149]. They are exploratory projects that systematically map the literature on a topic, identifying the key concepts, theories, sources of evidence, and gaps in the research. It involves systematically selecting, collecting and summarizing knowledge in a broad area [130]. | The purpose of a scoping review is to examine the extent, range and nature of research activity in an area. It is done to identify where there is sufficient evidence to conduct a full synthesis or to determine that insufficient evidence exists and additional primary research is needed [130,151]. It may be done for the purpose of disseminating research findings [63] or to clarify working definitions and the conceptual boundaries of a topic area [129]. | Arksey and O’Malley [63] recommend a 5 step process for conducting a scoping review: | The product of a scoping review will depend on the purpose for which it is conducted. In general, however, the narrative report provides an overview of all reviewed material. |
1) Identification of a broad research question. | ||||
Scoping Review | A scoping review is used to address broad topics where many different study designs and methods might be applicable. It may be conducted as part of an ongoing review, or as a stand-alone summary of research. Whereas a systematic review assesses a narrow range of quality-assessed studies to synthesize or aggregate findings, a scoping review assesses a much broader range of literature with a wide focus and does not synthesize or aggregate the findings [59]. | 2) Identification of relevant studies covering a wide breadth of literature and a variety of sources via databases, reference lists, and hand-searching key journals. This process may include consultation with key stakeholders. | The product generally includes: | |
3) Inclusion and exclusion criteria are identified as the review progresses. | 1) Basic numerical or narrative analysis of the extent, nature and distribution of the studies included with tables, graphs, and charts. | |||
Data type: Includes studies using any data type or method. May include empirical, theoretical or conceptual papers. Exclusion and inclusion criteria are inductively derived and based on relevance rather than on the quality of the primary studies or articles [150]. | 4) The data are sifted, sorted, compared and contrasted according to key issues and themes. Data are charted to allow for comparison and to ensure a uniform approach. | 2) Thematic organization of the literature (e.g., by intervention type, or by competing theoretical perspectives). | ||
5) Finally, the information is summarized and reported. Clear documentation of the methodology is important so that the reader can determine any potential reporting bias. | 3) Summary statement about what is known and not known (e.g., in the literature). | |||
Research question: The question is stated broadly and often becomes refined as the study progresses. One or more general questions may guide the review. | More recently, Levac et al. [129] have proposed recommendations to clarify and enhance each stage of the framework described above. | |||
Quality appraisal: The scoping review does not provide an appraisal of the quality of the evidence. It presents the existing literature without weighting the evidence in relation to specific interventions. | ||||
EMERGING | What is it? Rapid review of the literature provides a quick, rather than comprehensive, overview of the literature on a narrowly defined issue. Rapid review evolved out of a need to inform policy makers about issues and interventions in a timely manner [152]. It is often proposed as an intermediary step to be followed by a more comprehensive review. | The purpose is to produce a fast review of the literature, within a defined and usually limited time frame, on a question of immediate importance to a stakeholder group. | There is no standardized methodology as yet, but the depth and breadth of the review depends upon the specific purpose and the allotted time frame. Rapid reviews typically take one to nine months. | Typically a concise report is written for macro-level decision-makers that answer the specific review question. |
1) They begin with a needs assessment followed by formulation of a purpose statement and research question, definition of the context, and review of the literature [152,153,154]. | ||||
Rapid Review | Data type: The literature is often narrowly defined, focusing on a specific issue or a specific local, regional, or federal context [152]. It can include diverse study designs, methods, and data types as well as peer reviewed and gray literature. | 2) A review of the literature is streamlined in numerous ways including: | ||
· Accessing only published or online literature; | ||||
Research question: Rapid reviews require a thorough understanding of the intended audience and a specific, focused research question. | · Limiting by publication date, the number of databases, or language; | |||
·Searching electronic journals only; | ||||
Quality appraisal: Rapid reviews typically do not include an assessment of the quality of the literature, nor do they always include the views of experts and/or reviews by peers [152]. | · Narrowing to specific geographic settings or contexts; | |||
· Restricting the timeframe during which articles are assessed; | ||||
· Limiting contact with authors/industry or key stakeholders for clarification, follow-up, or input [152,153,154]. | ||||
3) References are retrieved, selected, summarized or synthesized, and a report is created. The public may be consulted about the results [152]. | ||||
It is important that those conducting a rapid review describe the methodology in detail to promote transparency, support transferability, and avoid misrepresenting the veracity of the findings [152]. | ||||
EMERGING | What is it? MNS is a new form of systematic review that addresses the issues of synthesizing a large and complex body of data from diverse and heterogeneous sources. At the same time, it is systematic in that it is conducted “according to an explicit, rigorous and transparent method” [[67], p. 418]. | The purpose is to summarize, synthesize and interpret a diverse body of literature from multiple traditions that use different methods, theoretical perspectives, and data types. | The steps to conduct a MNS [67,84,85] include the following: | The product of a MNS is: |
1) Planning Phase: | 1) A set of meta-narratives illustrating the story lines of various research traditions related to a common area or question; | |||
Meta-Narrative Synthesis (MNS) | The approach moves from logico-scientific reasoning (which underlies many approaches to synthesis) to narrative-interpretive reasoning. The unit of analysis for the synthesis is the unfolding “storyline” of a research tradition over time. Five key principles underlie the methodology: pragmatism, pluralism, historicity, contestation, and peer review. | · Assemble a multidisciplinary team, outline an initial broad question, and agree on outputs. | 2) An overarching conceptual framework that explains the phenomenon of interest. | |
Data type: This methodology involves the judicious combination of qualitative and quantitative evidence, and the theoretical and empirical literature. | 2) Search Phase: | |||
· Initially search by intuition, informal networking, browsing to map diversity of perspectives. | ||||
Research question: The original research question is outlined in a broad, open-ended format, and may shift and change through the process. | · Search for seminal papers. | |||
· Search for empirical papers in databases, hand searching key journals, and snowballing. | ||||
Quality appraisal: MNS uses the criteria of the research tradition of the primary study to judge the quality of the research, generally as set out in key sources within that tradition. | 3) Mapping Phase: | |||
· For each research tradition, identify key elements of the research paradigm, key actors and events in unfolding traditions, and prevailing language/imagery. | ||||
4) Appraisal Phase: | ||||
· Evaluate each study for validity/ relevance, extract and collate key results, group comparable studies. | ||||
5) Synthesis Phase: | ||||
· Identify all key dimensions of the problem/issue, provide a narrative account of each contribution, treat conflicting findings as higher order data and explain in terms of contestation between different paradigms from the original data. | ||||
6) Recommendations Phase: | ||||
· Summarize overall messages and relevant evidence; distil and discuss recommendations for policy, practice, and research. | ||||
EMERGING | What is it? A realist synthesis is a review of complex social interventions and programs that seek to unpack the mechanisms by which complex programs produce outcomes, and the context in which the relationship occurs. This is in contrast to systematic reviews, which aim to synthesize studies on whether interventions are effective. Realist synthesis seeks to answer the question: What works for whom, in what ways and under what circumstances? | The purpose of a realist synthesis is to guide program and policy development by providing decision makers with a set of program theories that identify potential policy levers for change. Within its explanatory intent, there are four general purposes: | Pawson et al. [69] identify 5 steps: | Pawson [68] explains that realist synthesis ends up with useful, middle-range theory. However, the product of a realist review combines theoretical understanding with empirical evidence. It focuses on explaining the relationships among the context in which an intervention takes place, the mechanisms by which it works, and the outcomes produced [68,69]. |
1) Reviewing for program theory integrity. | 1) Clarify scope: | Recommendations for dissemination and implementation are explicitly articulated. The result is a series of contextualized decision points that describe the contingencies of effectiveness. That is, a realist review provides an explanatory analysis that answers the original question of “what works for whom, in what circumstances, in what respects, and how” [[69], p. 21]. | ||
Realist Synthesis | This form of synthesis represents a review logic not a review technique [69]. Instead of a replicable method that follows rigid rules, the logic of realist review is based on principles. It reflects a shift away from an ontology of empirical realism to one of critical realism [155]. | 2) Reviewing to adjudicate between rival program theories. | · Identify the review question, nature of the intervention, circumstances for its use, and policy objectives; | |
3) Reviewing the same theory in different settings or with different populations. | · Refine the purpose of the review; | |||
Data type: There is no specific data preference but will include quantitative, qualitative and grey literature. Because the focus is on the mechanisms of action and their context, seemingly disparate bodies of literature and diverse methodologies are included. The focus is upon literature that emphasizes process with detailed descriptions of the interventions and context. | 4) Reviewing official expectations against actual practice [see 69, 107]. | · Make explicit the program theory or theories (e.g., the underlying assumptions about how the intervention is meant to work), synthesize theories, and design a theoretical framework. | ||
2) Search for evidence: | ||||
Research question: The review question is carefully articulated, prioritizing different aspects of an intervention [69]. It can be a broad question. | · Conduct an exploratory search; | |||
· Identify key program theories and refine inclusion criteria; | ||||
Quality appraisal: Realist review supports the principle that high quality evidence should be used but takes a different position than in systematic reviews on how the evidence is to be judged. It rejects a hierarchical approach to quality because multiple methods are needed to identify all aspects of the context, mechanisms and outcomes. Appraisal checklists are viewed skeptically because they cannot be applied evenly across the diverse study types and methods being reviewed. Thus, quality appraisal is seen as occurring in stages with a focus on the relevance of the study or article to the theory under consideration, and the extent to which an inference drawn has sufficient weight to make a credible contribution to the test of a particular intervention theory [69]. | · Purposively sample to test a subset of theories, with additional snowball sampling; | |||
· Search for new studies when review is almost completed. | ||||
3) Appraise primary studies and extract data: | ||||
· Use judgment to supplement critical appraisal checklists; | ||||
· Develop data extraction forms; | ||||
· Extract data. | ||||
4) Synthesize evidence and draw conclusions: | ||||
· Synthesize data to refine program theory; | ||||
· Let the purpose of the review lead the synthesis process; | ||||
· Use contradictory evidence to create insights about the impact of context; | ||||
· Present conclusions as a set of decision points. | ||||
5) Disseminate, implement and evaluate: | ||||
· Draft and test recommendations with key stakeholders focusing on what may influence policy; | ||||
· Work with policy makers and practitioners to apply recommendations; | ||||
· Evaluate the extent to which recommendations lead to program adjustments. | ||||
EMERGING | What is it? CIS is a methodology with an explicit orientation to theory generation, developed to respond to the need identified in the literature for rigorous methods to synthesize diverse types of research evidence generated by diverse methodologies [71] particularly when the body of evidence is very complex [72]. Thus, it was developed to address the limitations of conventional systematic review techniques. It involves an iterative process and recognizes the need for flexibility and reflexivity. It addresses the criticism that many approaches to syntheses are insufficiently critical and do not question the epistemological and normative assumptions reflected in the literature [72]. CIS is “sensitized to the kinds of processes involved in a conventional systematic review while drawing on a distinctively qualitative tradition of inquiry” [[72], p.35]. | The purpose of CIS is to develop an in-depth understanding of an issue/research question “by drawing on broadly relevant literature to develop concepts and theories that integrate those concepts” [[73], p. 71]. The overarching aim is to generate theory. | The developers of CIS explicitly reject a staged approach to the review. Rather, the processes are iterative, interactive, dynamic and recursive. It includes these general categories of activities [71,72]: | The product is a “synthesizing argument” that “links existing constructions from the findings to ‘synthetic constructs’ (new constructs generated through synthesis)” [[73], p. 71]. The synthesizing argument integrates evidence from across the studies in the review into a coherent theoretical framework [71,72]. This may be represented as a “conceptual map” that identifies the main synthetic constructs and illustrates the relationships among them [73]. |
1) Formulate the research question: | ||||
Critical Interpretive Synthesis (CIS) | Data type: CIS utilizes data from quantitative and qualitative empirical studies, conceptual and theoretical papers, reviews and commentaries. | · The question is not formulated in advance because the aim is to allow the definition of the phenomenon of interest to emerge from analysis. | ||
2) Search the literature: | ||||
Research question: It is neither possible nor desirable to specify a precise review question in advance. Rather the process is highly iterative and may not be finalized until the end of the review. | · Involves an organic approach using multiple search strategies (e.g., websites, reference chaining, contacting experts) in addition to a more structured approach; | |||
· Draw on the expertise of the team to identify relevant studies; | ||||
Quality appraisal: There is no hierarchy of designs for determining the quality of qualitative studies and, furthermore, no consensus exists on whether qualitative studies should even be assessed for quality [72]. Studies for inclusion are not selected on the basis of study design or methodological quality. Rather, papers that are relevant are prioritized. However, papers that are determined to be fatally flawed are excluded on the basis of a set of questions for determining quality [see 71]. Often, however, judgments about quality are deferred until the synthesis phase because even methodologically weak papers can provide important theoretical or conceptual insights [73]. | · Identify relevant papers that can form a sampling frame. | |||
3) Sample: | ||||
· May be selective and purposive, with emergent and flexible inclusion criteria; | ||||
· Ongoing selection is guided by theoretical sampling based on the emerging conceptual framework. | ||||
4) Determination of quality: | ||||
· See “quality appraisal” section. | ||||
5) Data extraction: | ||||
· Forms to guide this process can be useful, but with a huge database may be practically impossible; | ||||
· An informal process (highlighting text) can prove helpful. | ||||
6) Interpretive synthesis: | ||||
· Synthesis is based, in part, on the meta-ethnography strategies of reciprocal translational analysis, refutational synthesis, and lines of argument synthesis, but the authors greatly modified these to accommodate the diversity of literature (meta-ethnography used purely qualitative studies); | ||||
· The aim of the analysis is to produce a synthesizing argument, beginning with a detailed inspection of papers, gradually identifying recurring themes and developing a critique, constantly comparing concepts developed against the data and identifying the relationships among them. |
Types of Research Synthesis | Definition | Data Types Used | Products | Examples |
1. Conventional Synthesis | Older forms of review with less-systematic examination, critique, and synthesis of the literature on a mature topic for re-conceptulization or on a new topic for preliminary conceptualization | · Quantitative studies | · Narrative expression and summary | · Integrative review [14,18,19,20,21] |
· Qualitative studies | · Tables, charts, graphical displays, diagrams and maps | · Narrative synthesis [16] | ||
· Other types of data e.g., theoretical literature, policy | · Theory, theoretical/conceptual frameworks, or conceptual maps | · Conventional literature review[17] | ||
2. Quantitative Synthesis | Combining, aggregating, or integrating quantitative empirical research with data expressed in numeric form | · Quantitative studies | · Narrative expression and summary | · Systematic review [13,26,27] |
· Mathematical scores | · Meta-analysis [22,23,24,25] | |||
· Statements of generalizability | · Best evidence synthesis [28,29,30] | |||
3. Qualitative Synthesis | Combining, aggregating, or integrating qualitative empirical research and/or theoretical work expressed in narrative form | · Qualitative studies | · Narrative expression and summary | · Meta-synthesis [31,32,33,34,35,36,37,38,39,40,41,42,43,44] |
· Other types of data e.g., theoretical literature | · Theory, theoretical/conceptual frameworks, or conceptual maps | · Concept analysis [45,46,47] | ||
· A definition | · Grounded formal theory [37,48,49,50,51,52] | |||
· Meta-study [31,37,48,49,50,53,54] | ||||
· Meta-analysis [37,38,55,56,57,58] | ||||
· Meta-interpretation [59] | ||||
· Meta-ethnography [49,50,60,61,62] | ||||
4. Emerging Synthesis | Newer syntheses that provide a systematic approach to synthesizing varied literature in a topic area that includes diverse data types | · Quantitative studies | · Narrative expression and summary | · Scoping review [63] |
· Qualitative studies | · Tables, charts, graphical displays, diagrams and maps | · Rapid review [64,65] | ||
· Other types of data e.g., theoretical work, grey literature, editorials, commentaries, policy, evaluations | · Mathematical scores | · Rapid realist review [66] | ||
· Theory, theoretical/conceptual frameworks, or conceptual maps | · Meta-narrative synthesis [67,68] | |||
· A report written for decision-makers | · Realist synthesis [68,69] | |||
· Meta-summary [70] | ||||
· Critical interpretive synthesis [71,72,73,74] | ||||
· Other types of mixed-research synthesis [49,72,75,76,77,78,79] |
Types of Research Synthesis | Key Characteristics | Purpose | Methods | Product |
CONVEN-TIONAL | What is it? “The integrative literature review is a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated” [[14], p.356]. | Integrative reviews are used to address mature topics in order to re-conceptualize the expanding and diverse literature on the topic. They are also used to comprehensively review new topics in need of preliminary conceptualization [14]. | Integrative reviews generally contain similar steps [14,18], which include the following: | Conclusions are often presented in a table/diagram. Explicit details from primary sources to support conclusions must be provided to demonstrate a logical chain of evidence. |
1) Identify a clear problem. | ||||
Integrative Review | Data type: Integrative literature reviews include studies using diverse methodologies (i.e., experimental and non-experimental research, as well as qualitative research) in order to more fully understand a phenomenon of interest. It may also include theoretical and empirical literature. | Integrative reviews should ultimately present the “state of the art” of knowledge, depict the breadth and depth of the topic, and contribute to greater understanding of the phenomenon [18]. | 2) Determine the variables of interest (e.g., population, concept). | Torraco [14] suggests they can be represented in four forms: |
3) State a specific research purpose. | 1) A research agenda, | |||
Research question: Start by clearly identifying the problem that the review is addressing and the purpose of the review. There usually is not a specific research question, but rather a research purpose. | 4) Define and clearly document a search strategy. Aim to locate as many of the existing studies as possible. Purposive sampling may be used along with a more comprehensive approach. | 2) A taxonomy or conceptual classification of constructs, | ||
5) Critically evaluate the quality of primary reviews depending on the sampling frame used in the integrative review. | 3) Alternative models/conceptual framework, and | |||
Quality appraisal: The quality of primary sources may be appraised using broad criteria. How quality is evaluated will depend upon the sampling frame [18]. | 6) Identify a systematic analytic method. The constant comparative method [86,135] is one overarching approach commonly used. | 4) Metatheory. | ||
7) Keep a record of the process of data analysis (e.g., hunches, decisions, ideas about interpretation). | Results should emphasize implications for policy/practice [18]. | |||
8) State methodological limitations. | ||||
QUANT-ITATIVE | What is it? A SR is a review of literature that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies. Conducting a SR is analogous to conducting a primary study in that there are steps and protocols. It may or may not be done in conjunction with a meta-analysis. | The purpose of a SR is to integrate empirical research for the purpose of generalizing from a group of studies. The reviewer is also seeking to discover the limits of generalization [27]. | A number of authors have provided guidelines for conducting a SR [27] but they generally contain similar steps: | The products of a SR may include: |
1) Specify study aims and define research question. | 1) A statement about the relative “effectiveness” of health care interventions, or about the appropriateness, feasibility, or meaningfulness of findings for particular purposes; | |||
In Cochrane [81], a SR is identified as the highest form of evidence in support of interventions. By contrast, the Joanna Briggs Institute [104] does not define a SR as necessarily the highest form of evidence. | Often, the review focuses on questions of intervention effectiveness. Thus, the intent is to summarize across studies to obtain a summative judgment about the effectiveness of interventions. However, the Joanna Briggs Institute [104] suggests that for nursing, there is a concern not just with effectiveness but also with questions of appropriateness, meaningfulness and feasibility of health practices and delivery methods. Thus, SR’s may have purposes other than to assess the effectiveness of interventions. | 2) Set inclusion criteria for evidence. | 2) A statement about the strength of the relationship between a particular intervention and specific outcomes. | |
Systematic Review (SR) | 3) Design search strategy. | 3) More recently, the product might be a statement about the convergence of theoretical perspectives on a topic. | ||
As noted below, a meta-analysis is always a SR, but a SR is not always a meta-analysis. | 4) Screen potential evidence against criteria for assessing quality. | 4) When done in conjunction with meta-analysis, the product is a mathematic score that represents the statements above. | ||
5) Design data collection protocol. | ||||
Data type: There is nothing that specifies data have to be quantitative, and the definition can apply to qualitative findings. Generally, however, the term has been used most frequently to apply to reviews of quantitative studies - traditional RCTs and experimental or quasi-experimental designs. More recently, both the Campbell and the Cochrane collaborations have been grappling with the need to, and the process of, integrating qualitative research into a SR. A number of studies have been published that do this [13,75,78,135,136,137,138]. | 6) Select appropriate metric to represent the magnitude of findings and assess likelihood they are due to chance. | |||
7) Code the primary studies. | ||||
Research question: A well-defined research question is required. | 8) Analyze and display data using appropriate methods. | |||
9) Draw conclusions based on data. | ||||
Quality appraisal: The Quality Appraisal section under MA above also applies to SR. Some researchers are developing standard reliable and valid quality appraisal tools to judge the quality of primary studies but there remains no consensus on which tools should be used. The Joanna Briggs Institute [104] has developed their own criteria to ensure that only the highest quality studies are included in SRs for nursing, but they hold that studies from any methodological position are relevant. | 10) Discuss alternate interpretations in light of studies’ strengths and limitations. | |||
QUANT-ITATIVE | What is it? M-A is the statistical analysis of a large collection of results from individual studies (usually interventions) for the purposes of integrating the findings, based on conversion to a common metric (effect size) to determine the overall effect and its magnitude. The term was coined by Gene Glass [22,23] but dates back to 1904 [17]. A M-A is always a SR (see above). | Analytic M-As are conducted for the purpose of summarizing and integrating the results of individual primary studies to increase the power for detecting intervention effects, which may be small and insignificant in the individual studies [139,140]. | Specific steps include [25]: | The product for M-A includes a narrative summary of the findings with a conclusion about the effectiveness of interventions. |
1) Define the dependent and independent variables of interest. | 1) Analytic Products: | |||
Meta-Analysis (M-A) | Data type: Data are from quantitative research studies and findings, primarily randomized control trials. Increasingly there is use of experimental, quasi-experimental and some types of observational studies. Each primary study is abstracted and coded into a database. | Exploratory M-As are conducted to resolve controversy in a field or to pose and answer new questions. The main concern is to explain the variation in effect sizes. | 2) Collect the studies in a systematic way attempting to find all published and unpublished studies. | · Graphical displays of the data and a table that displays the key elements of each study. |
3) Read methods carefully and if effect sizes are not reported, identify articles for information to calculate these. | 2) Final product: | |||
Research question: A clear, well-defined research question or hypothesis is required. | 4) Examine variability among the obtained effect sizes informally with graphs and charts, to identify the possibility that moderator variables may account for the variability. | · A mathematic score that represents the strength of the effect of an intervention or the relationships between two variables. | ||
5) Combine effects using several measures of their central tendency and explore reasons for differences if found. | · Identification of variables that moderate or mediate the effects or relationships. | |||
Quality appraisal: Articles are usually appraised according to a set of pre-defined criteria but these criteria vary considerably and there are many methodological limitations [83]. Lower quality studies are not necessarily excluded and there is some debate about whether these should be included [24,29]. When lower quality studies are included, the validity of the findings is often discussed in relation to the study quality. | 6) Examine the significance level of the indices of central tendency, usually employing confidence intervals around unweighted mean effect sizes in a random effects model. | |||
7) Using an examination of the binomial effect size display, evaluate the importance of the obtained effect size. | ||||
QUAL-ITATIVE | What is it? “Meta-study is a research approach involving analysis of the theory, methods, and findings of qualitative research and the synthesis of these insights into new ways of thinking about phenomenon” [[54], p. 1]. | Analysis of research findings, methods, and theory across qualitative studies are compared and contrasted to create a new interpretation [53]. | Paterson et al. [54] propose a clear set of techniques: | Through the three meta-study processes, researchers create a “meta-synthesis” which brings together ideas to develop a mid-range theory as the product. |
1) Choose an analytic approach (e.g. grounded theory, thematic analysis). | ||||
Meta-Study | Data type: Three analytic components are undertaken prior to synthesis. Data includes qualitative findings (meta-data), research methods (meta-method), and/or philosophical/theoretical perspectives (meta-theory). | 2) Use specific sampling techniques according to inclusion/exclusion criteria, including searching for disconfirming cases that challenge the emerging theory. | ||
3) Regardless of approach, group studies according to characteristics (e.g., disease) and treat each group as a case [49]. | ||||
Research question: A relevant, well-defined research question is used. | 4) Engage in three distinct types of analysis, i.e. meta-data, meta-study, meta-theory (may be undertaken concurrently). | |||
5) Synthesize analysis into a theory. | ||||
Critical appraisal: According to Paterson et al. [54], primary articles are appraised according to specific criteria; however the specific appraisal will depend on the requirements of the meta-study. Studies of poor quality will be excluded. Data from included studies may also be excluded if reported themes are not supported by the presented data. | ||||
QUAL-ITATIVE | What is it? Meta-ethnography entails choosing relevant empirical studies to synthesize through repetitive reading while noting metaphors [61,62]. Noblit and Hare explain that “metaphors” refer to “themes, perspectives, organizers, and/or concepts revealed by qualitative studies” [[61], p. 15]. These metaphors are then used as data for the synthesis through (at least) one of three strategies including reciprocal translation, refutational synthesis, and/or line of argument syntheses. A meta-ethnographic synthesis is the creation of interpretive (abstract) explanations that are essentially metaphoric. The goal is to create, in a reduced form, a representation of the abstraction through metaphor, all the while preserving the relationships between concepts [61]. | To synthesize qualitative studies through a building of “comparative understanding” [[61], p. 22] so that the result is greater than the sum of the parts. | Methods used in meta-ethnography generally following the following: | The product of a meta-ethnography is a mid-range theory that has greater explanatory power than could be otherwise achieved in a conventional literature review. |
· Frame the study broadly by an interest, aim or purpose and ultimately, a research question. | ||||
Meta-Ethno-graphy | Data type: Qualitative research studies and findings on a specific topic. | Noblit and Hare summarize that meta-ethnography is “a form of synthesis for ethnographic or other interpretive studies. It enables us to talk to each other about our studies; to communicate to policy makers, concerned citizens, and scholars what interpretive research reveals; and to reflect on our collective craft and the place of our own studies within it” [[61], p.14]. | · Create inclusion/exclusion criteria. | |
· Conduct a review of the literature based on who the audience will be, what is credible to the audience, what accounts are available, and what the researchers’ interests are in the study [61]. | ||||
Research question: An “intellectual interest” [61], p.26] begins the process. Then, a relevant research question, aim, or purpose is developed. | · Identify all the appropriate studies in a field through repeated readings. | |||
Quality appraisal: Researchers are divided on the merits of critical appraisal and whether or not it should be a standard element in meta-ethnography [60]. Some researchers choose to follow pre-determined criteria based on critical appraisal [e.g., 62], whereas others do not critically appraise. | Noblit and Hare [61] identified three possible analysis strategies (all do not have to be completed): | |||
1) Reciprocal translational analysis. Key themes, metaphors, or concepts are identified and translated into each other to create the most representative concept. | ||||
2) Refutational synthesis. Contradictions between key themes, metaphors, or concepts are examined and explained. | ||||
3) Lines of argument synthesis. Interpretation is created from comparison of findings across distinct studies. | ||||
QUAL-ITATIVE | What is it? A grounded formal theory (GFT) is a synthesis of substantive grounded theories (GTs) to produce a higher order, more abstract theory that goes beyond the specifics of the original theories. GFT takes into account the conditions under which the primary study data were collected and analyzed to develop a more generalized and abstract model [31]. | The intent of GFT is to expand the applicability of individual GTs by synthesizing the findings to provide a broad meaning that is based in data and is applicable to people who experience a common phenomenon across populations and context [51]. | GFT uses the same methods that were used to create the original GTs in the synthesis [48,51]. Specific elements of the analytic process include: | A GFT is a mid-range GT that has “fit, work and grab”: that is, it fits the data (concepts and categories from primary studies), works to explain the phenomenon under review, and resonates with the readers’ experiences and understandings. |
1) Theoretical sampling - sample size is determined through purposive and theoretical sampling strategies to answer emerging questions [37,51]. | ||||
Data type: Substantive GTs were originally constructed using the methodology developed by Glaser & Strauss [86]. While some synthesis approaches emphasize including all possible primary GT studies, the concept of saturation in GFT (see Methods column) allows limiting the number of reviewed papers to emphasize robustness rather than completeness [50]. | The focus is on the conditions under which theoretical generalizations apply. GFT aims “to bring cultural and individual differences into dialogue with each other by seeking a metaphor through which those differences can be understood by others” [[31], p.1354]. | 2) Constant comparative analysis -the analyst identifies concepts and their relationship with other data, and compares theoretical ideas to prior and subsequent data. | Thorne et al. suggest that a GFT is “an artistic explanation that works for now, a model created on the basis of limited materials and a specific, situated perspective within known and unconscious limits of representation” [31], p. 1354]. | |
Grounded Formal Theory (GFT) | 3) Memoing - documentation of hunches, decisions, and modifications during analysis. | |||
Research question: GFT begins with a phenomenon of focus [51]. Analytic questions and the overall research question emerge throughout the process. | 4) Saturation - the point at which continued data collection and analysis brings only repeated concepts or ideas. | |||
5) Coding - begins at a descriptive level and progresses towards a more abstract and theoretical level. Findings are synthesized and translated across studies. | ||||
Quality appraisal: There is no discussion in the GFT literature about critically appraising the studies to be included. However, the nature of the analytic process suggests that critical appraisal may not be relevant. The authenticity and accuracy of data in a GFT are not an issue because, for the purposes of generating theory, what is important is the conceptual category and not the accuracy of the evidence. The constant comparative method of GFT will correct for such inaccuracies because each concept must “earn” its way into the theory by repeatedly showing up [67,68]. | ||||
QUAL-ITATIVE | What is it? Concept analysis is a systematic procedure to extract attributes of a concept from literature, definitions and case examples to delineate the meaning of that concept with respect to a certain domain or context. | Concept analysis is used to extend the theoretical meaning of a concept or to understand a conceptual practice problem [142,143]. In this case, concepts are cognitive descriptive meanings utilized for theoretical or practical purposes. | There are varied procedural techniques attributed to various authors such as Wilson [98], Walker & Avant [45], Chinn & (Jacobs) Kramer [145,146], Rodgers & Knafl, [46], Rodgers [99], Schwartz-Barcott & Kim [147], and Morse [47]. | Concept analysis generates a definition of a concept that may be used to operationalize phenomena for further research study [143] or theory development [144]. |
Concept Analysis | Data type: Most writings on concept analysis do not specify the data type. However, our scan of the methodological and empirical literature on concept analysis suggests that although the analytic approach in concept analysis is qualitative, quantitative study designs and data can be used to address the questions related to defining the meaning of a concept [e.g. 99, 141-142]. | Concept analysis is used to identify, clarify, and refine or define the meaning of a concept and can be used as a first step in theory development [47,144]. | Despite varied techniques, steps generally include: | |
1) Determine the purpose and aims. | ||||
Research question: Requires the researcher to isolate or identify a conceptual question or concept of interest. | 2) Delineate domains or boundaries of the concept. | |||
Quality appraisal: Quality appraisal is not typically attended to in concept analyses. Rather, researchers are interested in all instances of actual use of a concept (or surrogate terms) [142]. | 3) Draw on literature, dictionary meanings and/or cases. | |||
4) Analyze data sources to determine qualifying attributes. | ||||
5) Develop a prototype case and compare against contrary or borderline cases. | ||||
6) Test the practical significance. | ||||
7) Formulate defining features. | ||||
8) Relate to theoretical importance or practice application [46,141,148]. | ||||
EMERGING | What is it? Although no universal definition exists, there are some common elements of scoping reviews [129,149]. They are exploratory projects that systematically map the literature on a topic, identifying the key concepts, theories, sources of evidence, and gaps in the research. It involves systematically selecting, collecting and summarizing knowledge in a broad area [130]. | The purpose of a scoping review is to examine the extent, range and nature of research activity in an area. It is done to identify where there is sufficient evidence to conduct a full synthesis or to determine that insufficient evidence exists and additional primary research is needed [130,151]. It may be done for the purpose of disseminating research findings [63] or to clarify working definitions and the conceptual boundaries of a topic area [129]. | Arksey and O’Malley [63] recommend a 5 step process for conducting a scoping review: | The product of a scoping review will depend on the purpose for which it is conducted. In general, however, the narrative report provides an overview of all reviewed material. |
1) Identification of a broad research question. | ||||
Scoping Review | A scoping review is used to address broad topics where many different study designs and methods might be applicable. It may be conducted as part of an ongoing review, or as a stand-alone summary of research. Whereas a systematic review assesses a narrow range of quality-assessed studies to synthesize or aggregate findings, a scoping review assesses a much broader range of literature with a wide focus and does not synthesize or aggregate the findings [59]. | 2) Identification of relevant studies covering a wide breadth of literature and a variety of sources via databases, reference lists, and hand-searching key journals. This process may include consultation with key stakeholders. | The product generally includes: | |
3) Inclusion and exclusion criteria are identified as the review progresses. | 1) Basic numerical or narrative analysis of the extent, nature and distribution of the studies included with tables, graphs, and charts. | |||
Data type: Includes studies using any data type or method. May include empirical, theoretical or conceptual papers. Exclusion and inclusion criteria are inductively derived and based on relevance rather than on the quality of the primary studies or articles [150]. | 4) The data are sifted, sorted, compared and contrasted according to key issues and themes. Data are charted to allow for comparison and to ensure a uniform approach. | 2) Thematic organization of the literature (e.g., by intervention type, or by competing theoretical perspectives). | ||
5) Finally, the information is summarized and reported. Clear documentation of the methodology is important so that the reader can determine any potential reporting bias. | 3) Summary statement about what is known and not known (e.g., in the literature). | |||
Research question: The question is stated broadly and often becomes refined as the study progresses. One or more general questions may guide the review. | More recently, Levac et al. [129] have proposed recommendations to clarify and enhance each stage of the framework described above. | |||
Quality appraisal: The scoping review does not provide an appraisal of the quality of the evidence. It presents the existing literature without weighting the evidence in relation to specific interventions. | ||||
EMERGING | What is it? Rapid review of the literature provides a quick, rather than comprehensive, overview of the literature on a narrowly defined issue. Rapid review evolved out of a need to inform policy makers about issues and interventions in a timely manner [152]. It is often proposed as an intermediary step to be followed by a more comprehensive review. | The purpose is to produce a fast review of the literature, within a defined and usually limited time frame, on a question of immediate importance to a stakeholder group. | There is no standardized methodology as yet, but the depth and breadth of the review depends upon the specific purpose and the allotted time frame. Rapid reviews typically take one to nine months. | Typically a concise report is written for macro-level decision-makers that answer the specific review question. |
1) They begin with a needs assessment followed by formulation of a purpose statement and research question, definition of the context, and review of the literature [152,153,154]. | ||||
Rapid Review | Data type: The literature is often narrowly defined, focusing on a specific issue or a specific local, regional, or federal context [152]. It can include diverse study designs, methods, and data types as well as peer reviewed and gray literature. | 2) A review of the literature is streamlined in numerous ways including: | ||
· Accessing only published or online literature; | ||||
Research question: Rapid reviews require a thorough understanding of the intended audience and a specific, focused research question. | · Limiting by publication date, the number of databases, or language; | |||
·Searching electronic journals only; | ||||
Quality appraisal: Rapid reviews typically do not include an assessment of the quality of the literature, nor do they always include the views of experts and/or reviews by peers [152]. | · Narrowing to specific geographic settings or contexts; | |||
· Restricting the timeframe during which articles are assessed; | ||||
· Limiting contact with authors/industry or key stakeholders for clarification, follow-up, or input [152,153,154]. | ||||
3) References are retrieved, selected, summarized or synthesized, and a report is created. The public may be consulted about the results [152]. | ||||
It is important that those conducting a rapid review describe the methodology in detail to promote transparency, support transferability, and avoid misrepresenting the veracity of the findings [152]. | ||||
EMERGING | What is it? MNS is a new form of systematic review that addresses the issues of synthesizing a large and complex body of data from diverse and heterogeneous sources. At the same time, it is systematic in that it is conducted “according to an explicit, rigorous and transparent method” [[67], p. 418]. | The purpose is to summarize, synthesize and interpret a diverse body of literature from multiple traditions that use different methods, theoretical perspectives, and data types. | The steps to conduct a MNS [67,84,85] include the following: | The product of a MNS is: |
1) Planning Phase: | 1) A set of meta-narratives illustrating the story lines of various research traditions related to a common area or question; | |||
Meta-Narrative Synthesis (MNS) | The approach moves from logico-scientific reasoning (which underlies many approaches to synthesis) to narrative-interpretive reasoning. The unit of analysis for the synthesis is the unfolding “storyline” of a research tradition over time. Five key principles underlie the methodology: pragmatism, pluralism, historicity, contestation, and peer review. | · Assemble a multidisciplinary team, outline an initial broad question, and agree on outputs. | 2) An overarching conceptual framework that explains the phenomenon of interest. | |
Data type: This methodology involves the judicious combination of qualitative and quantitative evidence, and the theoretical and empirical literature. | 2) Search Phase: | |||
· Initially search by intuition, informal networking, browsing to map diversity of perspectives. | ||||
Research question: The original research question is outlined in a broad, open-ended format, and may shift and change through the process. | · Search for seminal papers. | |||
· Search for empirical papers in databases, hand searching key journals, and snowballing. | ||||
Quality appraisal: MNS uses the criteria of the research tradition of the primary study to judge the quality of the research, generally as set out in key sources within that tradition. | 3) Mapping Phase: | |||
· For each research tradition, identify key elements of the research paradigm, key actors and events in unfolding traditions, and prevailing language/imagery. | ||||
4) Appraisal Phase: | ||||
· Evaluate each study for validity/ relevance, extract and collate key results, group comparable studies. | ||||
5) Synthesis Phase: | ||||
· Identify all key dimensions of the problem/issue, provide a narrative account of each contribution, treat conflicting findings as higher order data and explain in terms of contestation between different paradigms from the original data. | ||||
6) Recommendations Phase: | ||||
· Summarize overall messages and relevant evidence; distil and discuss recommendations for policy, practice, and research. | ||||
EMERGING | What is it? A realist synthesis is a review of complex social interventions and programs that seek to unpack the mechanisms by which complex programs produce outcomes, and the context in which the relationship occurs. This is in contrast to systematic reviews, which aim to synthesize studies on whether interventions are effective. Realist synthesis seeks to answer the question: What works for whom, in what ways and under what circumstances? | The purpose of a realist synthesis is to guide program and policy development by providing decision makers with a set of program theories that identify potential policy levers for change. Within its explanatory intent, there are four general purposes: | Pawson et al. [69] identify 5 steps: | Pawson [68] explains that realist synthesis ends up with useful, middle-range theory. However, the product of a realist review combines theoretical understanding with empirical evidence. It focuses on explaining the relationships among the context in which an intervention takes place, the mechanisms by which it works, and the outcomes produced [68,69]. |
1) Reviewing for program theory integrity. | 1) Clarify scope: | Recommendations for dissemination and implementation are explicitly articulated. The result is a series of contextualized decision points that describe the contingencies of effectiveness. That is, a realist review provides an explanatory analysis that answers the original question of “what works for whom, in what circumstances, in what respects, and how” [[69], p. 21]. | ||
Realist Synthesis | This form of synthesis represents a review logic not a review technique [69]. Instead of a replicable method that follows rigid rules, the logic of realist review is based on principles. It reflects a shift away from an ontology of empirical realism to one of critical realism [155]. | 2) Reviewing to adjudicate between rival program theories. | · Identify the review question, nature of the intervention, circumstances for its use, and policy objectives; | |
3) Reviewing the same theory in different settings or with different populations. | · Refine the purpose of the review; | |||
Data type: There is no specific data preference but will include quantitative, qualitative and grey literature. Because the focus is on the mechanisms of action and their context, seemingly disparate bodies of literature and diverse methodologies are included. The focus is upon literature that emphasizes process with detailed descriptions of the interventions and context. | 4) Reviewing official expectations against actual practice [see 69, 107]. | · Make explicit the program theory or theories (e.g., the underlying assumptions about how the intervention is meant to work), synthesize theories, and design a theoretical framework. | ||
2) Search for evidence: | ||||
Research question: The review question is carefully articulated, prioritizing different aspects of an intervention [69]. It can be a broad question. | · Conduct an exploratory search; | |||
· Identify key program theories and refine inclusion criteria; | ||||
Quality appraisal: Realist review supports the principle that high quality evidence should be used but takes a different position than in systematic reviews on how the evidence is to be judged. It rejects a hierarchical approach to quality because multiple methods are needed to identify all aspects of the context, mechanisms and outcomes. Appraisal checklists are viewed skeptically because they cannot be applied evenly across the diverse study types and methods being reviewed. Thus, quality appraisal is seen as occurring in stages with a focus on the relevance of the study or article to the theory under consideration, and the extent to which an inference drawn has sufficient weight to make a credible contribution to the test of a particular intervention theory [69]. | · Purposively sample to test a subset of theories, with additional snowball sampling; | |||
· Search for new studies when review is almost completed. | ||||
3) Appraise primary studies and extract data: | ||||
· Use judgment to supplement critical appraisal checklists; | ||||
· Develop data extraction forms; | ||||
· Extract data. | ||||
4) Synthesize evidence and draw conclusions: | ||||
· Synthesize data to refine program theory; | ||||
· Let the purpose of the review lead the synthesis process; | ||||
· Use contradictory evidence to create insights about the impact of context; | ||||
· Present conclusions as a set of decision points. | ||||
5) Disseminate, implement and evaluate: | ||||
· Draft and test recommendations with key stakeholders focusing on what may influence policy; | ||||
· Work with policy makers and practitioners to apply recommendations; | ||||
· Evaluate the extent to which recommendations lead to program adjustments. | ||||
EMERGING | What is it? CIS is a methodology with an explicit orientation to theory generation, developed to respond to the need identified in the literature for rigorous methods to synthesize diverse types of research evidence generated by diverse methodologies [71] particularly when the body of evidence is very complex [72]. Thus, it was developed to address the limitations of conventional systematic review techniques. It involves an iterative process and recognizes the need for flexibility and reflexivity. It addresses the criticism that many approaches to syntheses are insufficiently critical and do not question the epistemological and normative assumptions reflected in the literature [72]. CIS is “sensitized to the kinds of processes involved in a conventional systematic review while drawing on a distinctively qualitative tradition of inquiry” [[72], p.35]. | The purpose of CIS is to develop an in-depth understanding of an issue/research question “by drawing on broadly relevant literature to develop concepts and theories that integrate those concepts” [[73], p. 71]. The overarching aim is to generate theory. | The developers of CIS explicitly reject a staged approach to the review. Rather, the processes are iterative, interactive, dynamic and recursive. It includes these general categories of activities [71,72]: | The product is a “synthesizing argument” that “links existing constructions from the findings to ‘synthetic constructs’ (new constructs generated through synthesis)” [[73], p. 71]. The synthesizing argument integrates evidence from across the studies in the review into a coherent theoretical framework [71,72]. This may be represented as a “conceptual map” that identifies the main synthetic constructs and illustrates the relationships among them [73]. |
1) Formulate the research question: | ||||
Critical Interpretive Synthesis (CIS) | Data type: CIS utilizes data from quantitative and qualitative empirical studies, conceptual and theoretical papers, reviews and commentaries. | · The question is not formulated in advance because the aim is to allow the definition of the phenomenon of interest to emerge from analysis. | ||
2) Search the literature: | ||||
Research question: It is neither possible nor desirable to specify a precise review question in advance. Rather the process is highly iterative and may not be finalized until the end of the review. | · Involves an organic approach using multiple search strategies (e.g., websites, reference chaining, contacting experts) in addition to a more structured approach; | |||
· Draw on the expertise of the team to identify relevant studies; | ||||
Quality appraisal: There is no hierarchy of designs for determining the quality of qualitative studies and, furthermore, no consensus exists on whether qualitative studies should even be assessed for quality [72]. Studies for inclusion are not selected on the basis of study design or methodological quality. Rather, papers that are relevant are prioritized. However, papers that are determined to be fatally flawed are excluded on the basis of a set of questions for determining quality [see 71]. Often, however, judgments about quality are deferred until the synthesis phase because even methodologically weak papers can provide important theoretical or conceptual insights [73]. | · Identify relevant papers that can form a sampling frame. | |||
3) Sample: | ||||
· May be selective and purposive, with emergent and flexible inclusion criteria; | ||||
· Ongoing selection is guided by theoretical sampling based on the emerging conceptual framework. | ||||
4) Determination of quality: | ||||
· See “quality appraisal” section. | ||||
5) Data extraction: | ||||
· Forms to guide this process can be useful, but with a huge database may be practically impossible; | ||||
· An informal process (highlighting text) can prove helpful. | ||||
6) Interpretive synthesis: | ||||
· Synthesis is based, in part, on the meta-ethnography strategies of reciprocal translational analysis, refutational synthesis, and lines of argument synthesis, but the authors greatly modified these to accommodate the diversity of literature (meta-ethnography used purely qualitative studies); | ||||
· The aim of the analysis is to produce a synthesizing argument, beginning with a detailed inspection of papers, gradually identifying recurring themes and developing a critique, constantly comparing concepts developed against the data and identifying the relationships among them. |