ML workflow for tissue ablation assessment in hyperspectral imaging using PCA, Faster R-CNN, and Mean Shift clustering.

OncoCell Vision
AI-powered microscopy for detecting and counting cancer cell biomarkers
Highlights
Cancer biologists at Volastra Therapeutics needed a scalable, accurate solution to detect and count cellular biomarkers (such as micronuclei and mitotic bodies) in microscopy images to measure chromosomal instability, the mechanism behind early cancer progression and treatment response.
Lead the development of an object detection pipeline to identify primary nuclei, micronuclei, mitosis, and apoptosis across diverse cancer cell lines.
Used the EfficientDet architecture with BiFPN, applied advanced tiling strategies for high-resolution images, and trained on 1,004 of 1,664 annotated images from internal and public datasets.
Achieved 85% mAP for primary nuclei and 66% for micronuclei, enabling automated, high-throughput cellular analysis critical for cancer research and drug development.
Core Team
Overview
In collaboration with Volastra Therapeutics, this project aimed to prevent cancer progression by accurately detecting and quantifying key cellular structures (primary nuclei, micronuclei, mitosis, and apoptosis) in microscopy images. These structures serve as biomarkers for cancer diagnosis and treatment efficacy.
Utilizing the EfficientDet object detection architecture, pre-trained on the ImageNet dataset, we developed a pipeline capable of localizing and counting these cellular components across diverse cell lines. The model achieved a mean average precision (mAP) of 85% for primary nuclei and 66% for micronuclei, demonstrating its effectiveness in complex biological imaging scenarios.
Data
The dataset comprised 1,664 microscopy images from 36 different cancer cell lines, sourced from:
- •Volastra's internal database
- •Image Data Resource
- •HMS LINCS database
The data was split into 1,004 images for training, 460 for validation, and 200 for testing. Pre-processing steps included image tiling with various strategies, such as regular window shift, padding, and unfixed-size windows, to handle high-resolution images and enhance model performance.
Methods
To detect and count various types of cell nuclei in microscopy images, we adopted an object detection approach centered on the EfficientDet architecture (Figure 1). EfficientDet is a state-of-the-art detection network that strikes a balance between accuracy and computational efficiency, making it suitable for high-throughput biomedical imaging tasks.
The backbone of the architecture is EfficientNet, a family of convolutional neural networks that scales depth, width, and resolution in a compound manner for improved performance. On top of this, we employed a Bi-directional Feature Pyramid Network (BiFPN), which enhances feature extraction at multiple scales by allowing bidirectional flow of information through top-down and bottom-up paths. This is particularly critical in microscopy images where cellular structures vary in size and location.
To optimize performance on high-resolution microscopy images, we implemented a tiling strategy. Large images were divided into smaller overlapping or non-overlapping tiles to enable manageable input sizes for the model and to preserve context at the borders. Three strategies were evaluated:
- •Regular window shift: Divides the image into uniform tiles
- •Window padding: Adds zero-padding to retain context
- •Unfixed-size windows: Adapts to cell density variations
(a) Regular window shift
(b) Window padding
(c) Unfixed-size windows
The model was trained on 1,004 images, validated on 460, and tested on 200 from a dataset comprising 36 cell lines. Labeling included annotations for primary nuclei, micronuclei, mitotic cells, and apoptotic bodies, allowing the model to learn to discriminate among these biologically significant substructures. To evaluate the model, we calculated mAP across object classes using the standard intersection-over-union (IoU) metric.
Results
The trained EfficientDet model demonstrated high accuracy in detecting and localizing cancer cell substructures across a range of cell lines and magnifications. The model achieved:
- •85% mAP for primary nuclei, indicating excellent capability in identifying and counting these dominant and morphologically clear structures
- •66% mAP for micronuclei, a more challenging target due to their small size, lower contrast, and morphological variability
These results show a strong performance relative to the complexity of the task, particularly for detecting micronuclei, which are critical indicators of genomic instability – a hallmark of cancer progression. The lower, yet still meaningful, performance on micronuclei reflects both the difficulty of the task and the potential for further model refinement.
Visual inspections of predictions across cell lines (MB-231 at 63X, HeLa, and CAL51) demonstrated that the model generalizes well to different imaging resolutions and biological conditions (Figures 3-5). Additionally, the comparison of different tiling strategies during preprocessing revealed that unfixed-size tiling preserved biological context more effectively in dense cellular environments, slightly boosting detection performance in certain cell lines.
Conclusion
This project successfully developed a deep learning pipeline for detecting and quantifying key cancer cell structures in microscopy images. The high accuracy achieved underscores the potential of AI in enhancing cancer diagnostics and treatment monitoring.
Future work may involve expanding the dataset to include more diverse cell lines and further refining the model to improve detection of less prevalent structures like micronuclei. The results validate the use of deep learning, and of EfficientDet with BiFPN in particular, for high-resolution biomedical image analysis, holding promise for scalable implementation in clinical research settings.
Keep Exploring
High-precision 3D organ segmentation via V-net architecture with dense skip connections, achieving up to 96% Dice score across 5 anatomical structures.






