124 S.B. Goldin

Some features of rigorous studies that may help improve study design are shown in Table 8.2, but it should be realized that it is possible to have a study with more than one of these features that is still not valid. Therefore, designing a rigorous research study takes knowledge about study design and implementation. Clinicians are typically not trained in these areas, and involving a colleague skilled in study design should be a priority prior to embarking on any educational research project.

Data Collection and Analysis

Once a study protocol has been approved for implementation, recruitment activities and data collection and analysis ensue. Although these may appear to be straightforward, the devil is in the details. A database with a data manual must be designed, data must be entered, data must be analyzed, and the results must be presentable and understandable. Each step can be fraught with problems involving even the seemingly simplest aspects, such as data entry. Who is doing the data entry? What method is being used to enter data? What is the error rate for the data entry?

Minimizing errors, maximizing flexibility, and setting up the data collection and maintenance system (including the actual database) in such a way as to promote effective data analysis begins with the hypothesis and an understanding of the goals of the study. It is important to design the data collection and maintenance system properly to facilitate the process. Again, clinicians are typically not trained in database design, management, or in statistical analysis and, therefore, consultation with a database management expert and statistician is advisable prior to beginning a research project.

Surveys

Surveys are one of the most important tools of measurement used in social research. They are also frequently used in educational research projects. Educational researchers often use surveys because they believe they are a simple and quick

126 S.B. Goldin

and the concerns about coercion, as well as physical or psychological harm. Medical students and residents are considered a vulnerable population. Therefore, there are strict guidelines that must be followed to protect them from potential harm. Institutional review boards for the protection of human subjects (IRBs) are mandated to protect participants.

IRB approval is not needed when data are collected for internal use only, i.e., not considered “human research.” For example, end-of-clerkship questionnaires do not require an IRB approval if they are to be used internally to evaluate the clerkship. If any data, however, collected in an end-of-clerkship survey is used for “human research” as defined by the Common Rule, IRB approval must be obtained. Human research involves studying a small group in order to generalize the findings to a larger group, which is often the reason educators would like to see the results of their surveys published. Therefore, if the results of educational surveys are to be published, IRB approval must usually be obtained. The highest quality research projects are designed prospectively, so developing a hypothesis that the end-of-clerkship questionnaire tests, developing a short research protocol describing the data collection and analysis activities, and obtaining IRB approval for the protocol prior to consenting students to take the questionnaire may be all that is required to develop a well-designed study and significantly improve its value as a scientific and scholarly product.

Most of the ethical concerns raised by IRBs in educational research are easily overcome by implementing methods that protect participants (normally medical trainees) from being individually identified by individuals in an authoritative position over the participants. Involving collaborators from outside colleges in data collection and analysis is therefore helpful in obtaining IRB approval, since these individuals have no formal interaction with the participants in terms of grading or evaluation. Clerkship directors and faculty directly involved with the participants have authoritative positions

over them and should not have access to participant-identi- fied data sets. In addition, involvement of psychologists, epidemiologists, sociologists, and statisticians on the research team who routinely collect and maintain data can provide a pool of expertise to provide consultation on methods of maintaining confidentiality of data throughout the entire research process, including data entry, data quality checks, data extraction, and data analysis.

The Common Rule holds that IRBs should be locally designed to address concerns of the immediate community. Therefore, at some institutions, two IRBs exist – one focusing on biomedical research, and one focusing on social and behavioral research. In such institutions, IRB approval of surgical education research may also be streamlined, as research protocols will most likely be more understandable and more easily vetted by the social and behavioral IRB as compared to the biomedical IRB. Most social and behavioral research is found to be “no greater than minimal risk” to participants. By contrast, clinical studies involving patient care generally involve “greater than minimal risk” and are therefore reviewed by IRBs well versed in biomedical research activities, reflecting the greater level of risk to research participants that must be considered. These higher risk studies require significantly more paperwork and time for ethical evaluation. On the other hand, it is often much easier and faster to submit educational research projects to an IRB accustomed to reviewing social and behavioral health studies, which often are less risky and require less participant protection.

Developing a good working relationship with the Associate Dean of Student Affairs (medical students) and/or the Associate Dean for Graduate Medical Education (residents and fellows) can also be extremely helpful when navigating the requirements of the IRB. Obtaining letters of support from these or similar student advocates for any educational research project stating that they feel there is no greater than minimal risk to the participants can often be key to obtaining more expedient IRB approval.