Dental training in evidence-based dentistry: A replication study

Background: In the area of education research, it is well-known that studies of a defi ned question are seldom replicated. Furthermore, e-learning resources with evidence-based content in dentistry have received relatively little attention from researchers. The Context and Purpose of the Study: The aim of this clinical study was to evaluate how dentistry students from two consecutive cohorts in their fi rst clinical semester rate a long-standing evidencebased dentistry (EbD) resource in an e-learning environment using a questionnaire of 43 specifi c items on 1) general questions regarding user-friendliness and acceptability, as well as 2) specifi c questions on content and functional range (A), handling and technical aspects (B), and didactics and educational value (C) based on a Likert scale from 0 = ‘strongly disagree’ to 3 = ‘strongly agree’, and how this compares to a primary study in which the resource was addressed as a novelty. The data were analyzed statistically using a one-way ANOVA followed by a Kruskal-Wallis multiple-comparison Z-test. Results: A response rate of 100% was achieved. The majority of the users thought the topic of EbD to be important. The e-learning resource was rated with a score of 2.40 ± 0.66 (on a Likert scale from 1-6 where 1 = “very good” and 6 = “insuffi cient”). 86.15% of the students stated that they consider the resource benefi cial for their study in clinical simulation and in patient treatment courses. The results averaged for A: 1.92 (±0.57; median: 1.928), B: 1.48 (±0.60), and C: 2.27 (±0.67). The obtained results in the replication study showed no statistical signifi cant differences to the primary study. Conclusions: The e-learning resource with dentistry vignettes cases and learning components on evidence-based principles was consistently rated positively by the students. Owing to their agreement with the data of the primary study, the results of the present study point to the remarkable validity of the method of evaluation. This should be addressed in future studies with larger cohorts. Research Article Dental training in evidence-based dentistry: A replication study Stefanie Dreher1, Tobias Weberschock2,3, Maria Giraki4, Alexander Uhse5, Puria Parvini6, Stefan Rüttermann4 and Susanne Gerhardt-Szep4* 1Dental offi ce, Frankfurt am Main, Germany 2Department of Dermatology, J.W. Goethe University, Frankfurt am Main, Germany 3Institute for General Practice, J.W. Goethe University, Frankfurt am Main, Germany 4Department of Operative Dentistry, Carolinum Dental University-Institute GmbH, J.W. Goethe University, Frankfurt am Main, Germany 5Department of Orthodontics, Carolinum Dental University-Institute GmbH, J.W. Goethe University, Frankfurt am Main, Germany 6Department of Dental surgery and Implantology, Carolinum Dental University-Institute GmbH, J.W. Goethe University, Frankfurt am Main, Germany Dates: Received: 07 September, 2017; Accepted: 14 October, 2017; Published: 16 October, 2017 *Corresponding author: Susanne Gerhardt-Szep, Doctor, PD, MME Department of Operative Dentistry, Carolinum Dental University-Institute GmbH, J.W. Goethe University, Theodor-Stern-Kai 7, Frankfurt am Main 60590, Germany, Tel.: +49-69-6301-7505; E-mail:


Introduction
In the area of education research, it is well-known that studies of a defi ned question are seldom replicated [1]. Such replication, however, is extremely valuable in assessing the validity of a method, which may be limited by the well-known Hawthorne effect, for example [2][3][4]. Even in cases in which randomization is not appropriate to the question replication is indicated as a matter of principle [5].
A recent study analyzed the complete publication history of the current top 100 education journals ranked by 5-year impact factor and found that only 0.13% of education articles were replications [1]. The authors emphasize the "importance of direct replications in helping education research improve its ability to shape education policy and practice" [1]. Such replication studies may be little publicized for two reasons, namely, that the original data from the primary study are not available or that numerous other parameters are intentionally or unintentionally altered in the replication. Concluding their recent evaluation, Gerhardt-Szep and colleagues came to recommend the e-learning environment in an e-learning resource on evidence-based dentistry in a cohort of dental students [6]. A modifi ed method of evaluation formed the basis of this commendation [7,8]. On behalf of the German Association of Medical Informatics, Biometry and Epidemiology (GMDS), the authors have coordinated the development of a catalogue of "Quality Criteria for Electronic Publications in Medicine" (QCEPM) [7]. This catalogue is subdivided into categories including contents, technical aspects, and didactics. evaluation [7]. The catalogue is available in German, English, Portuguese and Spanish, at: http://www.imbi.uni-freiburg. de/medinf/gmdsqc/ [7]. A modifi ed method of evaluation has already been applied in a dental context [9,10]. The reliability values determined here for the individual factors studies were designated "good" with a score of 0.83-0.89 and for the entire questionnaire "excellent" with a score of 0.92. It is important to stress that all the studies which used this method of evaluation addressed without exception newly introduced implementations of e-learning programmes. To what extent the internal validity of each is infl uenced by, for example, the well-known Hawthorne effect is diffi cult to predict and thereby represents a serious limitation of the results. The fact that the e-learning resource was newly introduced and that participants may therefore have paid especial regard to it could possibly have played an important role [4]. To reduce the impact of this limitation and thereby make the results of the evaluation more generalizable, it would be sensible to reevaluate the resource at a later date, for example, at one year after the initial implementation. Following from this, the focus of the present identical replication study was the following: how do dental students receive and respond to the as yet little researched teaching of EbD with multimedia components in comparison to a prior study in a different population? [6]. The research focus was the method of evaluation developed on the basis of the QCEPM. This was to be evaluated descriptively by direct comparison of the primary and replication studies, focusing on two concrete research questions: 1) How do students in their fi rst clinical semester of conservative dentistry rate the EbD e-learning resource in general and with respect to defi ned parameters?
2) Are there differences in individual results between the present replication study and the results of the primary study?

The e-learning resource
The e-learning resource forms one part of the socalled Frankfurt Dentistry Initiative (FranZi, Frankfurter Zahnmedizinische Initiative). The component presented in the present work ("Toothache Clinic -Focus") includes several interactive case vignettes. The resource was developed using an open-source authoring tool (WebKit Freiburg, Version 3.1 / beta) and is accessible without limitation online (http:// elearning.med.uni-frankfurt.de/spielwiese-fokus/). Many functions were incorporated in the 56-page resource (multiple choice questions drag and drop drop-down menus, draw functions, video clips, feedback and tips). EbD content was incorporated into over 57% of the total resource. EbD content was integrated with respect to the stipulation of the evidencebased medicine workgroup of the Frankfurt University which refers to newcomers to EbD. The initial implementation (primary study) of the online course "Toothache Clinic -Focus" was four semesters, i.e. two years, prior.

Student group and examination intervals
Dental students from two consecutive cohorts (summer semester 2014 and winter semester 2014/15) in their fi rst clinical semester (sixth academic semester) after using the voluntary e-learning resource were surveyed using a written questionnaire. In the fi rst meeting (c. 30 minutes long), they were informed about the use of the resources and told that they have been available for several years. None of the students had previous received educational content on evidence-based learning. A questionnaire conceived on the basis of the Ulm Quality Criteria Catalogue for Medical Learning Programmes served as the method of evaluation [8,9]. This has previously been used in connection with FranZi [9]. The fi rst part of the questionnaire concerned general details of semester number, sex, age, previous marks on major state exams (Abitur, preliminary scientifi c and medical examinations) as well as work experience and familiarity with electronic media (Likert scale from 1 = "very good" to 5 = "poor"). Furthermore, special questions also addressed the utility of the content in patient treatment and the openness to implementation of EbD. The overall rating for the module was elicited using a scale from 1 (very good) to 6 (insuffi cient) according to the German school marking system. Part two of the questionnaire concerned the evaluation of the resource in the dimensions of content and functional range (A), handling and technical aspects (B), and didactics and educational value (C) on a four-level Likert scale from 0 = "strongly disagree" to 3 = "strongly agree". Students could use the resource for 14 days and were asked to return the questionnaires at a second meeting (c. 30 minutes long).

Statistics
The analysis was performed using PASS and NCSS (Version 6.0.2.1. Kaysville, Utah), using using a one-way analysis of variance. Signifi cant differences were determined using the nonparametric Kruskal-Wallis multiple-comparison Z-value test according to Bonferroni. Means, standard deviations (SD) or frequencies were also evaluated.
After consultation with the university's ethics committee, it was decided that an ethics committee vote was not necessary since the evaluation was anonymous.

Results
The descriptive results of the study are shown in tables 1-5. The dental students gave the e-learning module in the present study an average score of 2.40 ± 0.66 (primary study: 2.26 ± 0.64). For the questions on the learning programme in the present study, dimension A received an overall average score of 1.90 ± 0.57 (primary study: 1.90 ± 0.63), dimension B 1.48 ± 1.60 (primary study: 1.55 ± 1.93) and dimension C 2.27 ± 0.67 (primary study 2.23 ± 0.79). All obtained results in the replication study showed no statistical signifi cant differences to the primary study.

Discussion
The present replication study has shown that an e-learning resource with dental case vignettes and evidence-based dentistry content, which has been available for several years, is on average rated positively by students of dentistry in their fi rst clinical semester of conservative dentistry and that the The agreement of the results with those of the primary study points to the remarkable validity of the chosen method of evaluation. Currently, there exist few replication studies in the fi eld of e-learning. Specifi cally, in the context of dentistry and with EbD content, no such publications are known to the authors [3,11].
Analogous to the present study, similar research in the fi eld of medicine focussing on online-learning of evidencebased medicine (EbM) has shown that participants judge the courses positively and that the e-learning environment is covariant with clinically relevant increase in knowledge [12][13][14][15][16][17]. It must be noted that the foundation for the use of EbD content is as a rule formed from real clinical situations and that novices are usually involved voluntarily [18,[14][15][16]. In a similar manner, the e-learning resource investigated in the present replication study concerned real patient cases and combined the basic knowledge of EbD with orientation towards patients, resulting in a kind of practical knowledge. According to a 2013 survey on courses in EbM, two of the fi ve steps of EbM, namely integration of results with patients and the evaluation of particular merit, fell signifi cantly behind the others and are thus addressed less frequently [15,16]. All steps were integrated into the present resource.
Taken as a whole, the methodology of evidence-based dentistry (EbD) takes centre stage in the endeavour to place dentistry on a scientifi c foundation [19]. In Germany in particular, however, EbD is not fi rmly established in dentistry curricula. This is in stark contrast to the USA, for example, where the inclusion of EbD in dentistry curricula has already been publicised several times [20][21][22][23]. This present problem in Germany, that dentistry graduates show minimal competence in EbD at the beginning of their working life as practicing dentists, should in the future be solved by the medical licensing act for dentists currently being planned in Germany and the National Catalogue of Competence-based Learning Objectives in Dentistry containing operationalised EbM learning objectives which has already been adopted [24,25]. Moreover, a 2016 study reported that fi ve defi ned parameters should be considered before a forthcoming implementation of EbM in curricula, namely 1) integrating EbM with other courses and Table 1: General population data (* = achieved before university entrance, **= achieved in preliminary preclinical education, *** = achieved in fi nal preclinical education. All marks ranged from 1 = very good, 2 = good, 3 = satisfactory, 4 = suffi cient, 5 = defi cient and 6 = insuffi cient).   Overall, I would give the module the following mark from 1 to 6, where 1 = 'very good', 2 = 'good', 3 = 'satisfactory', 4 = 'suffi cient', 5 = 'defi cient' and 6 = 'insuffi cient'.
2,40 ± 0,66 2,26 ± 0,64 3) EbM faculty development, 4) EbM whole-task exercises, and 5) longitudinal integration of EbM [26]. Based on these considerations, especial care was taken in the present study to integrate EbD content into the course of conservative dentistry particularly in the case of diagnosis and treatment of toothache.
In addition, EbD content was related to realistic clinical cases.
Faculty development was likewise apparent, since creation of the resource content in co-operation with the local EbM workgroup clearly required it. A longitudinal integration into several semesters of the dentistry study has not yet been realised, but it should be addressed in future studies.
According to the literature, teaching of EbM content can be successfully achieved in various settings (teacher-centred, e-learning, blended learning) [13,27,28]. In the present study, the resource was defi ned as an optional means for students' self-study. In the future, however, it will also be used in the form of blended learning. Comparable studies that used an identical evaluation protocol with the dimensions described here, albeit in other e-learning tutorials ("Toothache Clinic -Basic"), came to comparable conclusions with respect to  the dimension of "didactics and educational value" (2.29 ± 0.38) [8,9]. The two other dimensions were rated more highly, however (content and functional range: 2.32 ± 0.47 and handling and technical aspects: 2.47 ± 0.36) [8,9]. Here, it may be worth noting that these previous studies were conducted in 2009 when the e-learning resource was completely new and the students were unfamiliar with it [8,9]. Three years later, at the time of the primary study of this investigation, the fi eld of electronic media and resources seems to have developed greatly so that, understandably, the users gave lower ratings in questions of function and technical aspects. A period of 1.5 years lay between the data of the primary and replication studies with EbD content which led at least to more conspicuous judgements for several items in the dimension "didactics and educational value" of the programme. For example, in the primary study the students gave a higher rating in regard to the appropriateness of the learning unit for self-study or group study. The users tended to take the view that the resource is well suited to self-study. None of the students had experience using the studied resource in a group or in a blended learning scenario, however, so that an according implementation is quite conceivable.
In the study at hand, no data concerning increases in

Conclusion
In summary, the results of this clinical replication study are in agreement with the primary study and show that e-learning resources with dentistry case vignettes and integrated EbD content is rated positively by students of dentistry in their fi rst clinical semester on conservative dentistry.