An analysis on the effect of body tissues and surgical tools on workflow recognition in first person surgical videos

Hisako Tomita, Naoto Ienaga, Hiroki Kajita, Tetsu Hayashida, Maki Sugimoto

研究成果: Article査読


Purpose : Analysis of operative fields is expected to aid in estimating procedural workflow and evaluating surgeons’ procedural skills by considering the temporal transitions during the progression of the surgery. This study aims to propose an automatic recognition system for the procedural workflow by employing machine learning techniques to identify and distinguish elements in the operative field, including body tissues such as fat, muscle, and dermis, along with surgical tools. Methods : We conducted annotations on approximately 908 first-person-view images of breast surgery to facilitate segmentation. The annotated images were used to train a pixel-level classifier based on Mask R-CNN. To assess the impact on procedural workflow recognition, we annotated an additional 43,007 images. The network, structured on the Transformer architecture, was then trained with surgical images incorporating masks for body tissues and surgical tools. Results : The instance segmentation of each body tissue in the segmentation phase provided insights into the trend of area transitions for each tissue. Simultaneously, the spatial features of the surgical tools were effectively captured. In regard to the accuracy of procedural workflow recognition, accounting for body tissues led to an average improvement of 3 % over the baseline. Furthermore, the inclusion of surgical tools yielded an additional increase in accuracy by 4 % compared to the baseline. Conclusion : In this study, we revealed the contribution of the temporal transition of the body tissues and surgical tools spatial features to recognize procedural workflow in first-person-view surgical videos. Body tissues, especially in open surgery, can be a crucial element. This study suggests that further improvements can be achieved by accurately identifying surgical tools specific to each procedural workflow step.

ジャーナルInternational Journal of Computer Assisted Radiology and Surgery
出版ステータスAccepted/In press - 2024

ASJC Scopus subject areas

  • 外科
  • 生体医工学
  • 放射線学、核医学およびイメージング
  • コンピュータ ビジョンおよびパターン認識
  • コンピュータ サイエンスの応用
  • 健康情報学
  • コンピュータ グラフィックスおよびコンピュータ支援設計


「An analysis on the effect of body tissues and surgical tools on workflow recognition in first person surgical videos」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。