Project Results

Crowd sourced assessments of skill for cataract surgery were highly correlated with expert assessments. However, correlation is not the same as equivalence. We found that video length (time) was a better predictor of expert score than crowd score. However, by using both variables in combination (crowd score and video length) we were able to derive a conversion factor that very closely predicted expert score.

About This Project

I am trying to find a cheap, easy and reliable way to grade the skill of surgeons in training. Experts can judge resident surgical skill, but they are slow, have biases, and are expensive. If you watched the video, I'll bet you could tell the difference between the expert and the trainee, even without the labels. We will test the hypothesis that crowd sourcing surgical skill assessment is a cheap, fast, and standardized alternative to expert grading.

Ask the Scientists

What is the context of this research?

Despite the widely held belief that only surgeons have the expertise to assess surgical skill, growing evidence contradicts this assumption. When you watch figure skating in the winter Olympics, you probably can tell who deserves the gold medal. That is despite the fact that you were never a figure skater and never trained as a judge. Similarly, we believe that lay raters can determine the quality of a defined surgical maneuver in cataract removal as well as expert graders can. If this is true, frequent grading of surgical skill can be incorporated into training programs in a way not possible with expert evaluators. Furthermore, it permits comparison of resident skill across different training sites since the graders are not constrained by location.

What is the significance of this project?

How do you know that your grandmother's eye doctor is a good cataract surgeon? Surgeon skill is a factor in surgical outcomes, yet there is almost no data describing what level of skill or experience defines surgical competence. This is because we lack practical means to measure surgical skill. Resident surgical skill is decreed by committee of faculty surgeons who have operated with the resident. There is no standardized, objective measure of skill to accompany this determination. Our study will test the feasibility and validity of “crowd-sourcing” evaluations of surgical videos to lay raters as a cheaper, faster, easy and reliable way to measure resident operative surgical skill. If it works, it could be used like an "SAT test" to standardize measurement of skill.

What are the goals of the project?

We will collect videos of cataract surgery by all trainees who operate in the last month of the academic year, including soon-to-be-graduates. Both experts and lay raters will assess surgical skill in the videos using a modified OSATS grading tool. In this cross-sectional sample of residents in varied years of training, we will determine whether lay raters and surgical experts agree on their scores and can discriminate between groups of residents with more or less surgical experience (group differences). To determine if lay raters can detect improvement in surgical skill over time for individual trainees we will assess videos from residents in their final year of training as they progress along the learning curve. We expect lay raters' assessments to equal surgical experts.