Lei Feng network: According to this article translated from Google Blog, the author is Google brain product manager Craig Mermel and technical director Martin Stumpe.
Deep learning technology has recently shown great application prospects in medical disciplines such as ophthalmology, dermatology, radiology, and pathology. It can help provide patients around the world with more accurate and available high-quality medical services. Google also recently released a study that showed that the accuracy of convolutional neural networks to detect breast cancer metastasis in lymph nodes is comparable to that of a trained pathologist. However, so far, direct observation of the tissue with a compound optical microscope is still the pathologist's primary means of diagnosing diseases. How to digitally display the microstructure has become a key challenge for the large-scale application of deep learning techniques in pathology.
Today, in a speech at the American Association for Cancer Research Annual Conference (AACR), we introduced an Augmented Reality Microscope (ARM) using a paper titled “Enhanced Reality Microscope Real-Time Automatic Detection of Cancer (Current Review)”. Platform prototype, we believe this product can help accelerate the application of deep learning technology in the field of global pathology.
The platform consists of a modified light microscope that allows real-time analysis of images and displays the results of machine learning algorithms directly in the user's field of view.
It is worth mentioning that the use of low-cost, off-the-shelf components can transform this augmented reality microscope into a common optical microscope common in hospitals and clinics around the world, without the need for a complete digital system upgrade. Conduct organizational analysis.
Modern computing components and deep learning models —— such as those built on the TensorFlow platform, make this augmented reality microscope platform capable of running a large number of pre-training models. As with the traditional microscope method, the user observes the sample through the eyepiece and the results of the machine learning algorithm are projected in real time onto the light path, superimposed on the original image of the sample, helping the observer quickly locate and quantify the features of interest. Moreover, the platform's computational and visual feedback is very fast, and the current speed has reached 10 frames per second, which means that when the user moves the organization or magnification for further observation, a smooth and seamless visual experience can be obtained.
Left: Overview of the principle of the augmented reality microscope. A digital camera captures the same field of view (FoV) as the user and then passes the image to an additional computing unit that can run a machine learning model of real-time reasoning. The reasoning result is then fed back to a custom AR display that is inline with the eyepiece and displays the model output in the same plane as the specimen. Right: This image shows how our platform prototype was transformed into a typical clinical grade optical microscope.
In theory, augmented reality microscopes can provide a wide variety of visual feedback, including text, arrows, outlines, heat maps, and animations, and can run multiple types of machine learning algorithms to handle different tasks such as object detection, quantization, and Classification and so on.
To demonstrate the capabilities of augmented reality microscopy, we let it run two different cancer detection algorithms: one to detect breast cancer metastasis in lymph node specimens and the other to detect prostate cancer in prostatectomy specimens. These algorithms can be run at 4-40x magnification and outline the detected tumor area with a green outline. These contours can help the pathologist notice the area of interest without missing the appearance of obscure tumor cells.
Lei Fengwang Note: The view observed by the augmented reality microscope. These images show lymph node specimens at 4x, 10x, 20x, and 40x magnification.
Although these two cancer detection models were not trained with images captured directly by augmented reality microscopes, they performed very well on the latter without additional training. We believe that the performance of these algorithms will continue to increase if they are directly trained with images acquired from augmented reality microscopes.
Originally trained with images from a complete specimen scanner with a completely different optical structure, they performed very well on augmented reality microscope platforms without additional training. For example, when the lymph node metastasis detection model was run on an augmented reality microscope, the curve area reached 0.98, and the curve area of the prostate cancer detection model reached 0.96, which was only slightly lower than that obtained on the WSI.
We believe that this augmented reality microscope will have a major impact on global health, especially in the diagnosis of infectious diseases in developing countries, including tuberculosis and malaria. In addition, augmented reality microscopes can be used in conjunction with digital workflows in hospitals that will soon adopt digital pathological workflows. Optical microscopy has proven its worth in many industries, but its role in pathology is limited. We believe that augmented reality microscopes can be applied in many fields such as medical treatment, life science research, and materials science. We are pleased to continue to explore this augmented reality microscope to help accelerate the positive impact of machine learning technology around the world.