TY - JOUR A1 - Loda, Sophia A1 - Krebs, Jonathan A1 - Danhof, Sophia A1 - Schreder, Martin A1 - Solimando, Antonio G. A1 - Strifler, Susanne A1 - Rasche, Leo A1 - Kortüm, Martin A1 - Kerscher, Alexander A1 - Knop, Stefan A1 - Puppe, Frank A1 - Einsele, Hermann A1 - Bittrich, Max T1 - Exploration of artificial intelligence use with ARIES in multiple myeloma research JF - Journal of Clinical Medicine N2 - Background: Natural language processing (NLP) is a powerful tool supporting the generation of Real-World Evidence (RWE). There is no NLP system that enables the extensive querying of parameters specific to multiple myeloma (MM) out of unstructured medical reports. We therefore created a MM-specific ontology to accelerate the information extraction (IE) out of unstructured text. Methods: Our MM ontology consists of extensive MM-specific and hierarchically structured attributes and values. We implemented “A Rule-based Information Extraction System” (ARIES) that uses this ontology. We evaluated ARIES on 200 randomly selected medical reports of patients diagnosed with MM. Results: Our system achieved a high F1-Score of 0.92 on the evaluation dataset with a precision of 0.87 and recall of 0.98. Conclusions: Our rule-based IE system enables the comprehensive querying of medical reports. The IE accelerates the extraction of data and enables clinicians to faster generate RWE on hematological issues. RWE helps clinicians to make decisions in an evidence-based manner. Our tool easily accelerates the integration of research evidence into everyday clinical practice. KW - natural language processing KW - ontology KW - artificial intelligence KW - multiple myeloma KW - real world evidence Y1 - 2019 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:20-opus-197231 SN - 2077-0383 VL - 8 IS - 7 ER - TY - JOUR A1 - Brinker, Titus J. A1 - Hekler, Achim A1 - Hauschild, Axel A1 - Berking, Carola A1 - Schilling, Bastian A1 - Enk, Alexander H. A1 - Haferkamp, Sebastian A1 - Karoglan, Ante A1 - von Kalle, Christof A1 - Weichenthal, Michael A1 - Sattler, Elke A1 - Schadendorf, Dirk A1 - Gaiser, Maria R. A1 - Klode, Joachim A1 - Utikal, Jochen S. T1 - Comparing artificial intelligence algorithms to 157 German dermatologists: the melanoma classification benchmark JF - European Journal of Cancer N2 - Background Several recent publications have demonstrated the use of convolutional neural networks to classify images of melanoma at par with board-certified dermatologists. However, the non-availability of a public human benchmark restricts the comparability of the performance of these algorithms and thereby the technical progress in this field. Methods An electronic questionnaire was sent to dermatologists at 12 German university hospitals. Each questionnaire comprised 100 dermoscopic and 100 clinical images (80 nevi images and 20 biopsy-verified melanoma images, each), all open-source. The questionnaire recorded factors such as the years of experience in dermatology, performed skin checks, age, sex and the rank within the university hospital or the status as resident physician. For each image, the dermatologists were asked to provide a management decision (treat/biopsy lesion or reassure the patient). Main outcome measures were sensitivity, specificity and the receiver operating characteristics (ROC). Results Total 157 dermatologists assessed all 100 dermoscopic images with an overall sensitivity of 74.1%, specificity of 60.0% and an ROC of 0.67 (range = 0.538–0.769); 145 dermatologists assessed all 100 clinical images with an overall sensitivity of 89.4%, specificity of 64.4% and an ROC of 0.769 (range = 0.613–0.9). Results between test-sets were significantly different (P < 0.05) confirming the need for a standardised benchmark. Conclusions We present the first public melanoma classification benchmark for both non-dermoscopic and dermoscopic images for comparing artificial intelligence algorithms with diagnostic performance of 145 or 157 dermatologists. Melanoma Classification Benchmark should be considered as a reference standard for white-skinned Western populations in the field of binary algorithmic melanoma classification. KW - benchmark KW - artificial intelligence KW - deep learning KW - melanoma Y1 - 2019 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:20-opus-220569 VL - 111 ER - TY - JOUR A1 - Brinker, Titus J. A1 - Hekler, Achim A1 - Enk, Alexander H. A1 - Berking, Carola A1 - Haferkamp, Sebastian A1 - Hauschild, Axel A1 - Weichenthal, Michael A1 - Klode, Joachim A1 - Schadendorf, Dirk A1 - Holland-Letz, Tim A1 - von Kalle, Christof A1 - Fröhling, Stefan A1 - Schilling, Bastian A1 - Utikal, Jochen S. T1 - Deep neural networks are superior to dermatologists in melanoma image classification JF - European Journal of Cancer N2 - Background Melanoma is the most dangerous type of skin cancer but is curable if detected early. Recent publications demonstrated that artificial intelligence is capable in classifying images of benign nevi and melanoma with dermatologist-level precision. However, a statistically significant improvement compared with dermatologist classification has not been reported to date. Methods For this comparative study, 4204 biopsy-proven images of melanoma and nevi (1:1) were used for the training of a convolutional neural network (CNN). New techniques of deep learning were integrated. For the experiment, an additional 804 biopsy-proven dermoscopic images of melanoma and nevi (1:1) were randomly presented to dermatologists of nine German university hospitals, who evaluated the quality of each image and stated their recommended treatment (19,296 recommendations in total). Three McNemar's tests comparing the results of the CNN's test runs in terms of sensitivity, specificity and overall correctness were predefined as the main outcomes. Findings The respective sensitivity and specificity of lesion classification by the dermatologists were 67.2% (95% confidence interval [CI]: 62.6%–71.7%) and 62.2% (95% CI: 57.6%–66.9%). In comparison, the trained CNN achieved a higher sensitivity of 82.3% (95% CI: 78.3%–85.7%) and a higher specificity of 77.9% (95% CI: 73.8%–81.8%). The three McNemar's tests in 2 × 2 tables all reached a significance level of p < 0.001. This significance level was sustained for both subgroups. Interpretation For the first time, automated dermoscopic melanoma image classification was shown to be significantly superior to both junior and board-certified dermatologists (p < 0.001). KW - deep learning KW - melanoma KW - skin cancer KW - artificial intelligence Y1 - 2019 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:20-opus-220539 VL - 119 ER -