120 S.B. Goldin

of life of medical students. Measurement of quality of life would likely rely upon various psychological inventories, thus an expert in quality-of-life measurement should be invited to join the research team.

Universities often have significant expertise in departments outside of the medical school. Depending upon the research question, involving a psychologist, sociologist, medical education specialist, statistician or biostatistician, database management expert, and possibly an epi-informaticist, epidemiologist, or others from the discipline of public health, a bioethicist, representatives from the participant population for guidance (e.g., medical students, residents, or fellows), and individuals skilled in the humanities, could ultimately determine whether the research succeeds or fails. A research team composed of multi-disciplinary experts often enhances the quality of the research, increases the likelihood of yielding useful results, and improves the chance of successful funding. It is also the experience of many academic surgeons that experts in these various disciplines welcome the opportunity to work on collaborative projects, and a simple inquiry often results in very strong and prosperous collaborative efforts. Medical students, residents, and faculty, as working members of the research team, can provide insight to the project, maintain momentum, and assist with data collection, writing, data entry, human subjects approval, and other study activities. Inclusion of the junior members is also a way to not only stimulate interest, but also to provide mentoring in research, and may lead the member to become an academic surgeon, as well as a surgical researcher.

Pilot Testing

Another aspect of implementing a research project that should be commented upon includes the process of piloting and testing various aspects of the research activities. Even the best-designed research projects on paper have unforeseen issues that become apparent only with implementation.

For example, pilot testing questionnaires or other measurement activities on a small group of participants or volunteers often reveals potential problems at the design stage; these problems often have simple solutions that can be implemented prior to formally beginning data collection. Pilot testing may save significant resources in terms of both time and money, and produce a superior research product.

Demonstrating Reliability and Validity

The research steps documented in Table 8.1 clearly involve developing a strong study design. Two important aspects of study design involve reliability and validity, which are the two major criteria that will determine the quality of the study. Reliability has to do with the consistency of results from a repeat of the same activity, be it a particular measurement (such as a quality-of-life questionnaire) or an entire experiment (such as an experiment that finds two groups of similar participants respond in a similar fashion to the same curriculum). However, just because a measurement or experiment may demonstrate consistency of results, these results may not be correct. A poorly calibrated scale may weigh everyone as five pounds heavier than they actually are, and therefore be reliable, but incorrect.

Validity has to do with the correctness of the measurement, and may be categorized as either internal or external. Two especially important types of validity are internal and external validity. Internal validity refers to whether the data collected and results obtained actually pertain to answering the research question. It is imperative that studies be internally valid so that what was observed in the research study actually applies to the phenomenon being studied (such as the educational value of a particular curriculum). External validity refers to the process of being able to generalize the results to the population from which the study population was theoretically drawn (e.g., all medical students, not just medical students at a particular institution).

122 S.B. Goldin

This chapter cannot hope to describe all of the various aspects or types of validity, which includes multiple subcategories. Construct validity includes face validity, content validity, predictive validity, concurrent validity, divergent validity, convergent validity, and discriminant validity. Each type of validity seeks to evaluate the “correctness” of measurements. Each type of validity has various “threats,” including many types of bias, confounding variables or constructs, evaluation apprehension, investigator expectancies, and choices made with regard to the study cohort, location, and the timing of the intervention being studied, to name a few that may directly or indirectly impact validity. These threats must be taken into consideration when proving the validity of a research study.

Developing a Study Design

The concepts of reliability and validity must be taken into account when designing a research study, regardless of the actual study design. Reliability refers to the consistency of study results when independently repeated, and validity refers to the ability of a test to measure what it is intended to measure. Study designs may be descriptive, such as a study describing the attitudes of medical students toward a particular clerkship. Studies can be relational, in which the relationships between two or more variables can be examined, such as the relationship between level of depression and year of medical school in medical students. Finally, a study can be designed to look for causes, where one variable may cause a particular relationship with another variable, such as seeing if a particular educational intervention enhanced knowledge by conducting a pretest and posttest.

Clearly, understanding the terminology used in study design is important. Studies may be cross-sectional, where measurements are taken at a single point in time. These studies are useful in determining relationships between variables, but cannot establish causality. Longitudinal studies have repeated measures over time, such as a pretest and posttest.

These studies can be used to establish causality. Time series is another type of study design that can be applied in a different context. For example, to study the effect of the implementation of a new policy (such as a restriction on working hours for residents), a time series of measurements before and after the implementation of the policy can be useful.

Once a study design is selected, a list of variables to be measured must be discerned. Since educational research takes place in the context of an educational institution, many variables that are not the subject of study nevertheless influence aspects of the study, and must be measured. Measuring all pertinent variables is necessary to elucidate the relationship between all the variables. For example, the same two variables can simply be correlational (e.g., it is just personal preference that fewer women choose surgery as a career compared to men) or causal (e.g., fewer women choose surgery as a career because of gender discrimination), and these variables have relationships with other variables in the study (e.g., one institution may be more preferable to women for studying surgery due to less discrimination compared to others). Variables can be independent (e.g., thought to be a cause) or dependent (e.g. thought to be an effect of a cause), and the values of the variables (data) may be quantitative (e.g., numerical or categorical) or qualitative (e.g., in the form of narrative text, image, or video).

The academic surgeon cannot be expected to be expert at all aspects of study design. Therefore, while it is important for academic surgeons to learn as much as possible about study design, measurement, and data collection, maintenance, and analysis, other experts such as psychologists, epidemiologists, and statisticians are required to serve on the research team to assist with study design.

In summary, in surgical education research, as in all research, a rigorous study design with high reliability and validity is imperative, since study results can be reliable without being valid, valid without being reliable, or neither reliable nor valid. Clearly, a study with high levels of reliability and validity has significant meaning and is worth the effort of completing.