Pixel Classification

How it works, what it can do

The pixel classification workflow can classify the pixels of an image given user annotations. The classification of the image pixels can be used to segment the image into different objects, such as for example individual cells. The workflow is especially suited if the objects of interests are visually (brightness, color, texture) distinct from their surrounding. The algorithm is applicable for a wide range of segmentation problems that fulfill these properties.

A typical cell segmentation use case is depicted below.

Nice properties of the algorithm and workflow are

  • Interactive mode: the user gets immediate feedback after giving additional annotations.
  • Batch mode: the trained classifier can be applied to previously unseen images. Results are written to disk.
  • Uncertainty guidance: the user can view an uncertainty map, this indicates areas where the classifier is unsure about the results. Additional annotations in these regions help most.

Selecting good features

Assuming the user has already created or loaded an existing ilastik project and added a dataset, the first step is to switch to the Feature Selection Applet where the filter selection and computation are performed. The selected features and scales will be used later on for the training of a classifier. The selected features and scales should roughly correspond to the visual attributes that separate the objects and the background. A click on the Select features button brings up a feature selection dialog.

Here the user can select from several different feature types and scales.

  • Color/Intensity: these features should be selected if the color or brightness can be used to discern objects
  • Edge: should be selected if brightness or color gradients can be used to discern objects.
  • Texture: this might be an important feature if the objects in the image have a special textural appearance.

All of these features can be selected on different scales. The scales correspond to the pixel diameter that is used to calculate the respective feature. I.e. if a typical textural pattern has a pixel size of 4, this should be selected as the scale.

In general we advise to initially select a wide range of feature types and scales. Later on, this selection can always be refined. The selected features can be inspected in the bottom left after clicking OK in the feature selection dialog.

Training the classifier

The next step in the pixel classification is the training of a classifier that can separate the object classes. This training is done in an iterative fashion, the user gives some labels, evaluates the interactive prediction and then gives additional labels to correct eventual mistakes. To begin with the training of the classifier, we switch to the Training applet and add some labels. Each added label should correspond to a object type that we want to separate. In the simplest case, we add one label for the object class (in this example the object class would be cell) and an additional label fore the background class.

After adding at least two labels, the user can begin to mark the objects and the background. To mark an object, the label corresponding to the object class has to be selected, then the user can draw a scribble over the image. The background can be annotated in the same way after selecting the background class label on the left. The user can select the size of the brush, and switch between drawing and erasing mode just below the label list on the left.

To visualize the results of the classification, the Live update mode can now be switched on. The pixel classification result is displayed as an overlay on the image. Now classification errors can be corrected by giving additional annotations. The updated classification results will be displayed immediately in the main window.

To display the hard classification results, i.e. the final class assignment the Segmentation overlays can be turned on by clicking on the Segmentation checkbox.

How to import labels from an external file

To access the “Import Labels” feature in the GUI, do the following:

  1. Create N label classes (click “Add Label” N times).

  2. Right-click on the list item in the lower left-hand corner titled “Labels”. (See screenshot.) That will open up a window to allow you to import labels.

  3. If your label image is the same size as your input data, and the label image pixels already have consecutive values 1..N, then the default settings may suffice. Otherwise, you can modify the settings in that window to specify how to offset the label image relative to your input data, and also how to map label image pixel values to the label values ilastik needs (1..N).

Window Leveling

If labeling grayscale images the Training applet has an additional option: Window Leveling. This option can facilitate the labeling but has no impact on the training process itself. It can be used to adjust the data range used for visualization and thus helps to bring out small difference in contrast which might be useful when placing the labels. Pressing the left mouse button while moving the mouse back and forth changes the window width (data range) used for display. Moving the mouse in the left-right plane changes the window level, i.e. the center of the window. Of course, combinations of back-forth and left-right movements are possible to find just the right contrast needed. Pressing the right mouse button leads to an automatic range adjustment based on the intensity values currently displayed. To activate this feature either press the button outlined in the image below or use its keyboard shortcut (default ‘t’).

Note: if you can not see the button, you are either not working with grayscale images or you did not set the Channel Display to Grayscale in the Dataset Properties of your Raw Data.

Processing new images in batch mode

After the classifier is trained, it can be applied to unseen images as batch processing (without further training). This follows a general procedure in ilastik and is demonstrated here.
The results of this workflow (probability maps or segmentations) can be exported as images (.tiff, .png , etc.. ) or .h5 files. Details on all export options can be found on this page. In case you select to export a probability map (this is the default), it will be saved as a multichannel image, where each channel corresponds to a class you defined during training. For example, if you are performing binary classification into foreground/background, the probability map at pixel (px, py) will have the value of the foreground probability in the first channel and the value of the background probability in the second channel. If you choose to save a simple segmentation, the result will be a label image, where pixels are assigned the value of the most probable class. For example, suppose you are performing classification with three classes and the classifier output (probability map) for pixel (px, py) is 0.3, 0.3, 0.4 for classes 0, 1, 2 respectively. In the simple segmentation image ilastik exports, pixel (px, py) will then have value 2.

Processing new images in headless mode

Actually, after the classifier is trained, you don’t need the GUI anymore. If you’d rather run without it, ilastik has a special headless mode. This can be convenient for running on a cluster or on a remote machine.