Conducting a Multivocal Thematic Synthesis on an Extensive Body of Literature Réaliser une synthèse thématique multivocale d’un vaste ensemble de documentation

This paper will provide a methodology and progress report from a multivocal thematic synthesis being conducted on an extensive, diverse body of empirical studies. The study data includes a corpus of peer-reviewed empirical literature sharing a common reference published in English between 2000 and 2014. In this study, data to be synthesized share a common reference to the Garrison, Anderson, and Archer (2000) seminal paper introducing the Community of Inquiry framework, one of the most influential distance and blended learning theories published (Halverson, Graham, Spring, Drysdale, & Henrie, 2013). The study has two deliverables: 1) a multivocal thematic synthesis analysing contexts, populations, learning and teaching environments, and aspects of learning being studying through the lens of the CoI framework; and 2) an online, open, dynamic compendium of all CoI empirical research citing the Garrison et al., 2000, seminal paper. The purpose of the study is two-fold: to bring systematization and organization to a large body of research to support the work of other researchers, practitioners, and policy makers; and to provide an up-to-date methodology framework for creating an online database from large bodies of literature with a common reference. The data used in the thematic synthesis study consisted of full-text copies of peerreviewed research articles. The data was analysed using an open, constant-comparative, threestage thematic synthesis methodology. As the procedures used to gather, organize, and archive the data in this study could be applied to any multivocal body of research, the methodological processes was documented in detail to provide a framework for future research and may serve as an exemplar for sharing research data as an open-access resource. CJLT/RCAT Vol. 42(2) – Special Issue Conducting a Multivocal Thematic Synthesis on an Extensive Body of Literature 2 Resume Cet article offrira une méthodologie et un rapport sur la progression d’une synthèse thématique multivocale en cours de réalisation sur un vaste ensemble d’études empiriques variées. Les données d’étude comprennent un corpus de documents empiriques évalués par les pairs ayant une référence commune et publiés en anglais de 2000 à 2014. Dans cette étude, les données à être synthétisées ont en commun une référence à l’article phare de Garrison, Anderson et Archer (2000) ayant introduit le cadre de la communauté d’apprentissage, l’une des théories les plus influentes publiées sur l’apprentissage à distance et l’apprentissage mixte (Halverson, Graham, Spring, Drysdale et Henrie, 2013). L’étude vise deux réalisations : 1) une synthèse thématique multivocale analysant les contextes, les populations, les environnements d’apprentissage et d’enseignement, ainsi que les aspects de l’apprentissage étudiés dans l’optique du cadre de la communauté d’apprentissage; et (2) une collection en ligne, ouverte et dynamique, de toutes les études empiriques sur la communauté d’apprentissage citant l’article fondamental de Garrison et collègues (2000). L’étude a deux raisons d’être : systématiser et organiser un vaste ensemble de documents de recherche pour appuyer le travail d’autres chercheurs, praticiens et décideurs, et fournir un cadre méthodologique mis à jour pour la création d’une base de données en ligne à partir de vastes ensembles de documents ayant une référence commune. Les données utilisées dans la synthèse thématique sont formées des textes intégraux des articles de recherche évalués par les pairs. Les données ont été analysées à l’aide d’une méthode comparative constante et ouverte de synthèse thématique en trois phases. Comme les procédures utilisées pour recueillir, organiser et archiver les données dans cette étude pourraient être appliquées à tout ensemble de documentation multivocale, les processus méthodologiques ont été documentés en détail afin de fournir un cadre pour des recherches futures et peuvent servir d’exemple pour le partage de données de recherche comme ressource ouverte.


Introduction
The project described in this paper arises from a doctoral dissertation researching the influence of the Community of Inquiry (CoI) framework (Garrison, Anderson, & Archer, 2000) in distance and online education.Common to many academic dissertation undertakings, the purpose of the current project is to bring organization to, and to make sense of previous studies and publications in a specific topic or field; to delineate what is known or not known about the topic; and to identify a specific area of research within the topic that both interests the researcher and fulfills the expectation that the research will add important information to the body of knowledge on the topic.
For many research projects, the preceding steps of reviewing existing knowledge and identifying areas that make a case for further research are followed by steps that involve design and implementation of an appropriate research strategy to gather data, analyze data, describe findings, and then write up of the conclusions.However, the research project described in this paper diverges somewhat from the traditional research project pattern described above as the data being gathered and analyzed consists of empirical studies conducted by others.The use of previously published research materials as data places this project in the "literature review" article category of the American Psychological Association (APA) and fits within the description of articles "including research synthesis and meta-analyses, [which] are citical evaluations of material that has already been published" (American Psychological Association [APA], 2010, p. 10).
The body of literature chosen for this project shares a common reference to a seminal paper published by Garrison, Anderson, and Archer (2000).Since the Community of Inquiry framework was posited by Garrison et al. in 2000, hundreds of researchers (Garrison, Anderson, & Archer, 2010;Garrison & Arbaugh, 2007;Halverson, Graham, Spring, & Drysdale, 2012;Rourke & Kanuka, 2009) have referenced the framework and its three elements of social presence, cognitive presence, and teaching presence as the basis for empirical studies of distance, blended and online learning, teaching, student success and retention, as well as course and program design.Few education-focused academic publications have generated this amount of interest or empirical action (Halverson, Graham, Spring, Drysdale, & Henrie, 2013).Chronological categorization of Google Scholar© citations of the seminal CoI article provides evidence that the model continues to resonate within the education community despite extraordinary distance education and technological innovation, and copious distance education research in the intervening 14 years (See Appendix A).
The corpus of empirical research on the CoI model is substantial.Despite the extent of empirical CoI research, an up-to-date thematic research synthesis of the CoI model or an easily accessible database of this body of research literature does not exist.The proliferation of journals, online databases and archives and search engines such as Google Scholar© do facilitate literature searches (Fink, 2013) but there exists no easy way to organize and assimilate hundreds or thousands of articles on a particular topic.The absence of a CoI research synthesis or compendium means that researchers interested in the CoI model must seek and construct this literary resource individually and probably repetitively -a daunting and time-consuming task considering the quantity and diversity of CoI empirical research.As the body of CoI research continues to grow by several hundred articles each year, collecting and making sense of the accumulated knowledge becomes an even more formidable task, particularly for novice researchers.
In 1976, in his book introducing meta-analysis, Glass observed "Our problem is to find the knowledge in the information.We need methods for the orderly summarization of studies so that knowledge can be extracted from the myriad individual researches" (p.4).The preceding statement aligns closely with the problem that this study is based upon, namely how to make sense of a diverse body of research reflecting different purposes, perspectives and contents but sharing a common foundation of the Community of Inquiry framework.Ogawa and Malen (1991) confronted this problem by devising a pragmatic research method they termed an exploratory case study method for multivocal literatures.They define multivocal literature as being comprised of all accessible writings on a common topic embodying diverse views or voices, appearing in a variety of forms and reflecting different purposes, perspectives and informational basis, addressing different aspects of the topic and incorporating different research or non-research logics.
Ogawa and Malen describe exploratory case study as "well-suited to the examination of complex contemporary phenomena…because the distinctive features of the exploratory case study parallel the major characteristics of multivocal literatures" (p.274).This present study integrates multivocal exploratory concepts that include rigorous examination and recording of document characteristics; organizing, categorizing and open coding in order to elicit themes and patterns; and creation of a "formal, retrievable data base" (p.280).

Deliverables
The study will produce two deliverables: (a) a multivocal thematic synthesis analysing contexts, populations, learning and teaching environments, and aspects of learning being studying through the lens of the CoI framework, and (b) an online, open, dynamic compendium of all CoI empirical research citing the Garrison et al. (2000) seminal paper.The purpose of the study is two-fold: to bring systematization and organization to a large body of research to support the work of other researchers, practitioners, and policy makers; and to provide an up-todate methodology framework for creating an online, digital database from large bodies of literature with a common reference.
A thematic synthesis of empirical CoI research for the wider community of distance education researchers, practitioners, administrators, and instructional designers may facilitate transference of empirical research to applied practice.Of the 12 synthesis methods discussed by Barnett-Page and Thomas (2009), thematic synthesis is identified as one of the methods most likely to "directly inform policy and practice" (p. 6).For the distance education research community, the synthesis may also identify areas of abundant or sparse research, dissention or agreement on aspects of the model, as well as innovative and pragmatic applications.
Upon completion of the project, the online archive will contain abstracts and publication information for all CoI empirical research articles that cite the Garrison et al. (2000) seminal paper.The archive will be structured such that users will be able to sort, search, and query the collection for their own research or synthesis purposes.In keeping with copyright stipulations for fair dealing for academic purposes, Canadian Copyright Act (Copyright Act, 2001) full-text documents will be used by the researcher for research purposes only and not added to the online archive unless allowed by copyright.

Method
For the pilot study, Google Scholar© was used to identify all documents citing Garrison, Anderson and Archer (2000); Zotero ™ research management software was then used to facilitate downloading, saving, and tagging the first 50 documents from this citation list as a calibration and methodological refinement exercise.Zotero TM automatically creates academic citations, downloads, and attaches full-text files (where accessible), saving substantial amounts of time both in acquiring and cataloguing documents particularly when studies are published in open journals.Full-text copies of all 50 documents were obtained through online digital databases or institutional libraries.As all study dates were 2000 or later, 49 (98%) of the first 50 full-text versions were readily available in digital format with meta-data which also facilitated document examination for key study parameters making it possible to retrieve and code documents rapidly.
The abstract, body, and reference sections of each document were examined to determine whether the paper contained a valid citation of the Garrison et al., (2000) seminal paper.A proforma tool, developed specifically for this project to record key document categorization information, was created for each study artefact (See Appendix B).The pro-forma tool is an original design, but is loosely based on examples provided by the Campbell Collaboration (Hammerstrøm, Wade, & Jørgensen, 2010).While the pro-forma tool was invaluable in ensuring consistency in document acquisition, identification, and categorization to safeguard data collection integrity and facilitate coding, it is not to be considered an a priori code table.The pro-forma tool has been piloted, and reviewed by colleagues and has proven to be an essential part of the data collection, coding, and database methodology used in this project.
As of March 19, 2014, Google Scholar© listed 1,902 citations of the Garrison et al., (2000) paper.The completed study will eventually include all documents citing the seminal paper, but this present pilot phase of the project was limited to the first 50 most relevant documents as identified by Google Scholar©.Identical document citation search parameters were applied using Harzing's (2014) open-source citation analysis tool, Publish or Perish© version 4.6.3,and returned the same citation count.Approximately 50 percent of the papers examined so far contained only one very brief reference to Garrison et al., (2000).Studies containing a sole citation of the seminal paper will not be included in the project analysis phase, but they will, however, be tagged accordingly and listed in the final project database to inform other researchers.
The populated and tagged Zotero ™ database was then exported to a spreadsheet program which was used to analyze document characteristics and timelines.Subsequent phases of this project will involve exporting representational full-text research articles from the Zotero™ database into ATLAS-ti™ for thematic coding and content analysis.A three-stage free coding protocol for thematic synthesis developed by Thomas and Harden (2008) will frame the content analysis work.

Analysis
Analysis in the full study will take place in two distinct analysis phases as described below; however, for this pilot, only phase 1 was completed.
Analysis Phase 1. Research study characteristics were analyzed quantitatively using data recorded on the year of artefact publication, level of evidence, study sample size, geographic location, research type, data collection method, data analysis method, education environment (online, etc.), educational context of study (grad, undergrad, etc.) participants, study discipline, and component of CoI component being researched.This analysis was conducted using a spreadsheet program.
Analysis Phase 2. Qualitative thematic content analysis coding will be conducted on at least 20 studies from each chronological subsample group.Studies to be included in the thematic synthesis will be identified during analysis phase 1 and will include studies that are based primarily on the CoI framework or components thereof.With each data subset sorted and grouped chronologically by year, 20 full-text articles will be chosen based on the Google Scholar ranked relevance from each chronological group.These 20 artefacts will be subjected to paragraph-by-paragraph open-coding based on the research questions, namely: What is the nature, focus, and context of empirical research that has been undertaken on the CoI framework since the publication of the Garrison et al. (2000) seminal paper?; and What factors have contributed to the continuing popularity of the CoI framework?
The coding protocol proposed for this dissertation research is similar to the thematic synthesis process followed by Thomas and Harden (2008).Each included study will be read and re-read using a problematizing approach that "goes beyond the semantic content of the data, and starts to identify or examine the underlying ideas, assumptions, conceptualizations, and ideologies that are theorized as shaping or informing the semantic content of the data" (Braun & Clarke, 2006, p. 84).Attride-Sterling (2001) refers to this initial set of codes as basic themes or "lower-order premises evident in the text" (p.388).
When all subsample groups have been coded, qualitative data analysis software will be used to analyze lower-order codes to discover analytical themes.Finally, analytical themes will be analyzed and mapped chronologically, subjectively, and technologically against the CoI framework itself as well as against developments in distance education pedagogy and technology, and major events in the life of the CoI model such as book publications, CoI measurement tool development, and special issues of journals.Analysis by year may reflect use of the CoI with technology such as chat, audio-, or video-conferencing that didn't exist when the original paper was published, augmentation of asynchronous discussion forums with increased visual elements such as still pictures and use of wikis, blogs, and social media.
As the goal of this study is to gather and report factual knowledge, not to judge or critique other studies or interpret data, researcher honesty and meticulousness have more influence on the outcomes of the research than potential biases.Further, the choice of thematic synthesis as a methodology lessens the influence of bias.Barnett-Page and Thomas (2009) state that while thematic synthesis involves some interpretation of data, its synthetic product is more likely to be "reproducible and correspond to a shared reality" (p.6), than most other research synthesis methods.

Results
Preliminary analysis of the pilot sample of 50 peer-reviewed CoI research studies for the period 2000 through to 2013 that referenced the seminal Garrison et al. (2000) paper revealed a wide range of study types, some based on the entire Community of Inquiry (CoI) framework, others on individual or combinations of social, teaching, and cognitive presences.The pilot dataset included 41 journal articles, 4 conference papers, 3 book sections, 1 book, and 1 thesis.The earliest item was published in 1999, the most recent in 2014.If there was only one brief intext citation of the seminal article included as background in the literature review section, the item was categorized as "1BC -one brief citation," tagged and excluded from further examination.As a result of this initial document examination, 24 articles were excluded from further examination.
Three main categories of study foci appear to be: (a) validation of the CoI model itself and of CoI measurement instruments (Arbaugh et al, 2008), (b) applicability of the framework and relationships of the three component presences in particular contexts, and (c) use of the CoI framework as an underpinning framework for course or program design or redesign, and for measuring the effect of educational interventions primarily in post-secondary or higher distance, in blended or online formal education settings.
Challenges associated with accessing this corpus of literature became evident during the pilot study of CoI empirical research articles.As anticipated, the pilot study revealed a wide variety of closed and open publication venues, many requiring subscriptions and passwords, complicating the acquisition of articles for inclusion in research literature reviews.However, the processes developed, documented, and calibrated during the pilot have provided a viable framework for the full study.In particular, the Proforma template form provided very effective for systematic study characteristic recording and database entry.Version 1 of the Proforma template was revised significantly as a result of the pilot study (See Appendix B).
The online database produced during the pilot study has been beta tested and some revisions to the Zotero tagging procedure based on pilot study results are recommended for the full study.This pilot study has been presented to an online research group with encouraging results.The pilot database has already proved useful in answering queries from other researchers seeking specific CoI articles.

Summary and Conclusion
Many significant distance education developments and innovations have occurred since the Garrison et al. (2000) asynchronous, text-based Community of Inquiry study was completed.Some of those developments include widespread adoption of learning management systems (LMS) (WebCT 1997, Blackboard TM , Desire to Learn TM 1999, Canvass TM , epath TM  Although the volume of studies to be included in the full study will be prodigious, incorporation of software tools such as the Zotero ™ online research tool, availability of digital documents with embedded meta-data, online database programs with import and export capabilities, and qualitative research analysis software such as ATLAS-ti™, render the study feasible.In addition, the fact that academic journals now publish in both digital and analogue formats, and have digitized past volumes, facilitates identification and retrieval through online journals and academic publication databases.The practical logistics of identifying and gathering a large body of scholarly literature are significantly easier to address since the advent of Google Scholar©.According to Harzing (2013), Google Scholar© has reached maturity, and is a comprehensive, stable, reliable, un-biased source for citation information, particularly for recent social science publications.This partially completed project adds to the research synthesis knowledge base by demonstrating how emerging digital research tools and processes can be applied in innovative ways to bring order and insights to extensive bodies of literature.The successful pilot calibration and methodological refinement work completed so far has verified the feasibility of the full project, and it is anticipated that the completed research project and the online database will serve as exemplars for sharing open, online research literature resources.Although this particular study centers on the CoI corpus of research, the methodology could be applied to any multi-vocal body of research.

Appendix A: Google Scholar Citations
CoI Chronological Google Scholar© Citations of Garrison, Anderson and Archer (2000) completed March 14, 2014.
1999, Moodle TM 2002); significantly improved synchronous web conferencing and communication media such as Skype TM , Macromedia Breeze (now Adobe Connect TM ) in 2003; improved intercompatibility between computer operating systems (Sharable Content Object Reference Model or SCORM) in 2004; Connectivism, Facebook TM , Web 2.0, social networks in 2005; iPhone TM , mobile learning in 2007; massive open online courses (MOOCs); and the Android OS in 2008.Yet, studies based on the CoI framework continue to proliferate.Providing an accessible, organized, tagged database of Community of Inquiry research will support the work of future researchers, practitioners, and policy makers.Categorization of Google Scholar© citations sorted by publication date demonstrate that the CoI framework continues to resonate with educational researchers and practitioners despite these significant evolutions in distance education technology and context (See Appendix A).

Table 1
Study Level of Evidence

Table 7
Research Population Education Level

Proposed Study Document Characterization Pro-forma Tool
* Garrison, D. R., Anderson, T., & Archer, W. (2000).Critical inquiry in a text-based environment: Computer conferencing in higher education.The Internet and Higher