A deep learning spatial-temporal framework for detecting surgical tools in laparoscopic videos

摘要

Background and objective: Image-based surgical tool presence detection is an indispensable component for developing various intelligent applications in future operating rooms (ORs). To date, tool presence detection in laparoscopic videos has been investigated, and some recent studies tackled it in a spatial-temporal manner. The promising performance demonstrates the value of temporal information to develop robust methods for surgical tool detection. Therefore, a deep learning framework that considers spatial and temporal information for detecting surgical tools in laparoscopic videos is proposed. Methods: The proposed approach consists of a hierarchical organised neural architecture consisting of a convolutional neural network (CNN) with two long short-term memory (LSTM) models. The CNN model was used to learn spatial features from laparoscopic images. Since the data was sparsely labelled at 1 Hz, an LSTM network (LSTM-clip) -based on the CNN output- was employed to learn temporal dependencies from short intermediate partially labelled video clips. Finally, temporal dependencies along the complete surgical videos were modelled using another LSTM (LSTM-video). The models were trained and validated using six-fold Monte Carlo crossvalidation (MCCV). Results: Six-fold cross-validation experiments on the large publicly available dataset (Cholec80) explicate the advantage of temporal information to the tool detection task by improving the mean average precision (mAP) by 3.00 %. The proposed approach achieved a mAP of 94.74 % that exceeds the state-of-the-art methods. Conclusion: The overall approach demonstrates the value of modelling temporal dependencies across consecutive laparoscopic images to enhance surgical tool presence detection.