Skip to content

Kaggle Challenge dataset

This folder holds the test set of the "Human Protein Atlas - Single Cell Classification" Kaggle challenge (2021), organised by Emma Lundberg's lab. It contains single-cell crops of the test images, the cell masks the annotations refer to, the hand-made single-cell labels that were used to score the competition, and the full images of the public leaderboard part of the test set. This document explains what the challenge was, what each file contains, and the pitfalls you should know about before using it.

The challenge

The HPA has run two image classification challenges on Kaggle. This dataset belongs to the second one:

  • Human Protein Atlas - Single Cell Classification (26 January to 11 May 2021, https://www.kaggle.com/c/hpa-single-cell-image-classification). The task was to predict, for each segmented cell, the subcellular localization of the protein stained in the green channel. The difficulty was that the training images only carried image-level labels from the standard HPA annotation (a pattern present in the image is not necessarily present in every cell), while the test set was annotated cell by cell. 757 teams took part. The metric was the mean average precision over the 19 classes, with cell masks matched at IoU 0.6 (the OpenImages instance segmentation metric).
  • The first challenge, Human Protein Atlas Image Classification (October 2018 to January 2019, https://www.kaggle.com/c/human-protein-atlas-image-classification), was image-level only, with 28 classes and a macro F1 metric. It has nothing in this folder, but the image lists of its train and test sets are in the HPA Subcellular dataset: other/kaggle_master_training.csv and other/kaggle_master_test.csv (HPA FOV name -> Kaggle id).

The 2021 test set was 1,776 images: 559 scored the public leaderboard and 1,217 the private leaderboard (the final ranking). They were unpublished images chosen for pronounced cell-to-cell variation, and four annotators labelled their cells individually (10% of the cells were annotated twice, with 90% agreement). The images are HPA confocal images with the usual four channels: blue (nucleus), red (microtubules), yellow (ER) and green (protein of interest).

The 19 classes of the challenge are:

id class id class
0 Nucleoplasm 10 Microtubules
1 Nuclear membrane 11 Mitotic spindle
2 Nucleoli 12 Centrosome
3 Nucleoli fibrillar center 13 Plasma membrane
4 Nuclear speckles 14 Mitochondria
5 Nuclear bodies 15 Aggresome
6 Endoplasmic reticulum 16 Cytosol
7 Golgi apparatus 17 Vesicles and punctate cytosolic patterns
8 Intermediate filaments 18 Negative (negative and unspecific signal)
9 Actin filaments

Some classes group several HPA locations. "Vesicles and punctate cytosolic patterns" covers vesicles, peroxisomes, endosomes, lysosomes, lipid droplets and cytoplasmic bodies, and "Actin filaments" includes focal adhesion sites.

What is in this folder

  • all_crops.csv: one row per annotated cell, 33,308 cells from 1,771 FOVs. This is the main file. Its columns are:
  • cell_id: the cell number, the same as the cell's value in the FOV mask (see masks).
  • x1, y1, x2, y2: the cell bounding box inside the 1024 x 1024 crop, not in the FOV. The crop is centred on the cell, so the box is centred on 512,512. 1,113 cells are larger than 1024 pixels and are cut off by the crop.
  • annotated_label: the single-cell annotation, as HPA location names (not the 19 class ids), comma-separated when the cell has several. The values are the full HPA vocabulary plus Negative, Unsure (321 cells) and Discarded (70 cells); 27 cells have an empty label. 12,997 cells have more than one location.
  • if_plate_id, position, sample: the FOV, i.e. <plate>_<position>_<sample>.
  • atlas_name, antibody, ensembl_ids, gene_names: the cell line, antibody and target gene.
  • crops/<plate>/: four images per cell, <FOV>_cell<N>_crop_[blue|red|yellow|green].png, 1024 x 1024 with the cell at the centre. They keep the bit depth of the source image (8-bit for the old 1728 x 1728 plates, 16-bit otherwise). Despite their name these crops are masked: every pixel outside the cell's own mask is 0, so neighbouring cells are not visible. Each plate folder also has one <FOV>_cell_bbox.csv per FOV, which is simply that FOV's rows of all_crops.csv.
  • masks/<plate>/<FOV>_cellmask.png: the cell mask of the whole FOV, at the FOV's resolution (1728, 2048 or 3072 pixels), as an 8-bit label image: 0 is background and every other value is a cell_id of all_crops.csv.
  • private_leaderboard.csv: the 1,217 images of the private leaderboard, one row per image, with:
  • Image_ID: the image's id in the Kaggle test set.
  • basename: the HPA FOV name.
  • locations_merged, locations_reindex: its image-level labels, as class names and as the class ids of the table above (|-separated).
  • The rest are the image's columns from the HPA IF-image.csv, as they were when the file was made.
  • public_leaderboard.csv: the 559 images of the public leaderboard, one row per image, with Image_ID (the Kaggle id), ImageWidth and ImageHeight, basename (the HPA FOV), has_crops and n_annotated_cells, followed by the image's columns from the HPA IF-image.csv (as of HPA v25, not as they were during the challenge). There are no image-level labels in Kaggle's numbering for this set.
  • images/<plate>/<FOV>_[blue|red|yellow|green].png: the full, unmasked images of the 559 public leaderboard FOVs, exactly as Kaggle distributed them (only renamed from the Kaggle id to the FOV name). They are the test images of the public Kaggle download; the 1,217 private leaderboard images were never distributed, so they are not here.
  • mismatched_images.txt: the five FOVs of the test set that have no crops here: three of the private leaderboard (1103_B6_2, 1694_D11_8, 1694_F7_28) and two of the public leaderboard (1018_G12_6, 652_F9_3). Why they were left out is not documented: for the two public ones, the Kaggle image is identical to the HPA image, so it was not the pixels.
  • Kaggle2021/: the complete public download of the 2021 challenge as Kaggle distributes it (one 158 GB zip, including the training set), with its own Kaggle2021DatasetInfo.md.
  • location_group_mapping.tsv: a mapping from each original HPA location name (Original annotation) to several coarser groupings used in the lab (Grouping 1, Grouping 2, Grouping 3, Grouping dl2 and Grouping OpenCell). An empty cell means the location has no counterpart in that grouping. It is a general utility; none of its columns is exactly the 19-class scheme of the challenge.

Numbers

FOVs cells
private leaderboard (private_leaderboard.csv) 1,217, of which 1,214 have crops 25,037
public leaderboard (public_leaderboard.csv, full images in images) 559, of which 557 have crops 8,271
total 1,776, of which 1,771 have crops 33,308

The challenge paper reports 41,597 annotated cells, more than the 33,308 here. The reason is not documented. Keep it in mind if you compare your numbers with the published ones.

All the images use the "Normal" channel scheme and come from 17 cell lines, with U-2 OS being the most common.

Relation to the HPA Subcellular dataset

Only 527 of the 1,776 FOVs are also in the HPA Subcellular dataset (515 in images and 12 in removedFromV23). Most test images had never been published when the challenge ran, and many still are not.

  • Public leaderboard: all 559 full images are in this folder's images, whether HPA holds them or not.
  • Private leaderboard: only 101 of the 1,217 FOVs are in the HPA dataset. For the other 1,116, the masked crops in this folder are the only copy of the pixels we hold; there is no unmasked version anywhere in our storage.

When the full FOV is available, you can rebuild an unmasked crop from it. The crop centre is the centre of the cell's bounding box in masks, ((y_min + y_max + 1) // 2, (x_min + x_max + 1) // 2), and the crop spans 512 pixels on each side, zero-padded outside the image. This reproduces the crops here exactly once the cell mask is applied.

Using it

  • Take the cell list from all_crops.csv, not from the masks alone.
  • Translate annotated_label to the 19 classes yourself. Map the HPA names with the grouping described in "The challenge" section (for example Lysosomes -> 17, Focal adhesion sites -> 9), and keep Negative as class 18.
  • Decide what to do with Unsure, Discarded and empty labels. These cells were not usable annotations, and you should normally leave them out of any evaluation.
  • Remember that the crops are masked and that large cells are truncated at 1024 pixels. Models trained on unmasked crops, or on full images, will see a different kind of input.
  • To tell the two leaderboard sets apart, use private_leaderboard.csv and public_leaderboard.csv: every FOV of all_crops.csv is in exactly one of them.

Additional resources