Skip to content
Back to Portfolio
OncoCell Vision - Automated cell detection in histopathology

OncoCell Vision

AI-powered microscopy for detecting and counting cancer cell biomarkers

Client: Volastra Therapeutics · New York · United States 🇺🇸
TensorFlowPythonEfficientDetscikit-learnWeights & BiasesOpenCV

Highlights

Situation

Cancer biologists at Volastra Therapeutics needed a scalable, accurate solution to detect and count cellular biomarkers (such as micronuclei and mitotic bodies) in microscopy images to measure chromosomal instability, the mechanism behind early cancer progression and treatment response.

Task

Lead the development of an object detection pipeline to identify primary nuclei, micronuclei, mitosis, and apoptosis across diverse cancer cell lines.

Action

Used the EfficientDet architecture with BiFPN, applied advanced tiling strategies for high-resolution images, and trained on 1,004 of 1,664 annotated images from internal and public datasets.

Result

Achieved 85% mAP for primary nuclei and 66% for micronuclei, enabling automated, high-throughput cellular analysis critical for cancer research and drug development.

Core Team

Viacheslav Danilov

Viacheslav Danilov

Senior Data Scientist

Quantori

Cambridge · United States 🇺🇸

Oleg Talalov

Oleg Talalov

Senior ML Engineer

Quantori

Cambridge · United States 🇺🇸

Vladislav Laptev

Vladislav Laptev

Data Scientist

Quantori

Cambridge · United States 🇺🇸

Akanksha Verma

Akanksha Verma

Senior Bioinformatician

Volastra Therapeutics

New York · United States 🇺🇸

Christina Eng

Christina Eng

Lead Bioinformatician

Volastra Therapeutics

New York · United States 🇺🇸

Overview

In collaboration with Volastra Therapeutics, this project aimed to prevent cancer progression by accurately detecting and quantifying key cellular structures (primary nuclei, micronuclei, mitosis, and apoptosis) in microscopy images. These structures serve as biomarkers for cancer diagnosis and treatment efficacy.

Utilizing the EfficientDet object detection architecture, pre-trained on the ImageNet dataset, we developed a pipeline capable of localizing and counting these cellular components across diverse cell lines. The model achieved a mean average precision (mAP) of 85% for primary nuclei and 66% for micronuclei, demonstrating its effectiveness in complex biological imaging scenarios.

Data

The dataset comprised 1,664 microscopy images from 36 different cancer cell lines, sourced from:

The data was split into 1,004 images for training, 460 for validation, and 200 for testing. Pre-processing steps included image tiling with various strategies, such as regular window shift, padding, and unfixed-size windows, to handle high-resolution images and enhance model performance.

Methods

To detect and count various types of cell nuclei in microscopy images, we adopted an object detection approach centered on the EfficientDet architecture (Figure 1). EfficientDet is a state-of-the-art detection network that strikes a balance between accuracy and computational efficiency, making it suitable for high-throughput biomedical imaging tasks.

The backbone of the architecture is EfficientNet, a family of convolutional neural networks that scales depth, width, and resolution in a compound manner for improved performance. On top of this, we employed a Bi-directional Feature Pyramid Network (BiFPN), which enhances feature extraction at multiple scales by allowing bidirectional flow of information through top-down and bottom-up paths. This is particularly critical in microscopy images where cellular structures vary in size and location.

Figure 1. EfficientDet architecture with EfficientNet backbone and BiFPN for multi-scale feature extraction.

To optimize performance on high-resolution microscopy images, we implemented a tiling strategy. Large images were divided into smaller overlapping or non-overlapping tiles to enable manageable input sizes for the model and to preserve context at the borders. Three strategies were evaluated:

  • •Regular window shift: Divides the image into uniform tiles
  • •Window padding: Adds zero-padding to retain context
  • •Unfixed-size windows: Adapts to cell density variations

(a) Regular window shift

(b) Window padding

(c) Unfixed-size windows

Figure 2. Comparison of tiling strategies for processing high-resolution microscopy images: (a) regular window shift, (b) window padding, and (c) unfixed-size windows.

The model was trained on 1,004 images, validated on 460, and tested on 200 from a dataset comprising 36 cell lines. Labeling included annotations for primary nuclei, micronuclei, mitotic cells, and apoptotic bodies, allowing the model to learn to discriminate among these biologically significant substructures. To evaluate the model, we calculated mAP across object classes using the standard intersection-over-union (IoU) metric.

Results

The trained EfficientDet model demonstrated high accuracy in detecting and localizing cancer cell substructures across a range of cell lines and magnifications. The model achieved:

  • •85% mAP for primary nuclei, indicating excellent capability in identifying and counting these dominant and morphologically clear structures
  • •66% mAP for micronuclei, a more challenging target due to their small size, lower contrast, and morphological variability

These results show a strong performance relative to the complexity of the task, particularly for detecting micronuclei, which are critical indicators of genomic instability – a hallmark of cancer progression. The lower, yet still meaningful, performance on micronuclei reflects both the difficulty of the task and the potential for further model refinement.

Visual inspections of predictions across cell lines (MB-231 at 63X, HeLa, and CAL51) demonstrated that the model generalizes well to different imaging resolutions and biological conditions (Figures 3-5). Additionally, the comparison of different tiling strategies during preprocessing revealed that unfixed-size tiling preserved biological context more effectively in dense cellular environments, slightly boosting detection performance in certain cell lines.

Figure 3. Detection results on MB-231 cell line (63X magnification) showing identification of primary nuclei and micronuclei.
Figure 4. Detection results on HeLa cell line showing identification of primary nuclei and micronuclei.
Figure 5. Detection results on CAL51 cell line showing identification of primary nuclei and micronuclei.

Conclusion

This project successfully developed a deep learning pipeline for detecting and quantifying key cancer cell structures in microscopy images. The high accuracy achieved underscores the potential of AI in enhancing cancer diagnostics and treatment monitoring.

Future work may involve expanding the dataset to include more diverse cell lines and further refining the model to improve detection of less prevalent structures like micronuclei. The results validate the use of deep learning, and of EfficientDet with BiFPN in particular, for high-resolution biomedical image analysis, holding promise for scalable implementation in clinical research settings.

Keep Exploring

Biomedical Research

ML workflow for tissue ablation assessment in hyperspectral imaging using PCA, Faster R-CNN, and Mean Shift clustering.

July 2024
Medical Imaging

High-precision 3D organ segmentation via V-net architecture with dense skip connections, achieving up to 96% Dice score across 5 anatomical structures.

July 2020