CLIMAT : clinically-inspired multi-agent transformers for knee osteoarthritis trajectory forecasting
Nguyen, Huy Hoang; Saarakkala, Simo; Blaschko, Matthew B.; Tiulpin, Aleksei (2022-04-26)
H. H. Nguyen, S. Saarakkala, M. B. Blaschko and A. Tiulpin, "CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting," 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), 2022, pp. 1-5, doi: 10.1109/ISBI52829.2022.9761545.
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
https://rightsstatements.org/vocab/InC/1.0/
https://urn.fi/URN:NBN:fi-fe2022101361858
Tiivistelmä
Abstract
In medical applications, deep learning methods are designed to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents — a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at https://github.com/MIPT-Oulu/CLIMAT.
Kokoelmat
- Avoin saatavuus [31928]