Visualization of parameter space for image analysis.

Analysis of user requirements

We interviewed experienced assay developers at the Broad Institute of MIT and Harvard. These experts regularly use CellProfiler and frequently assist and train less experienced users. We also observed novice users at a hands-on introductory workshop on CellProfiler. Requirement 1 Complex biomedical research questions involving image analysis can usually be reduced to finding biological objects - such as cell nuclei - and to deriving simple statistics for these - such as the area occupied by each cell. To do so, users need support to effectively and efficiently identify a suitable set of parameter settings to get high-quality results from a representative test image. Once users have confidence in these settings, they will proceed to run a pipeline with these values to automatically analyze a large collection (hundreds to millions) of similar images (high-throughput analysis). The results, when analyzed, allow them to answer their real research questions. Typically, this involves identifying cases where cells deviate from the norm in a particular way. The above analysis enabled us to characterize the conventional approach to parameter optimization shown in Figure 1 . Our assay developer collaborators immediately recognized the linear nature of the description and readily agreed that it was an accurate depiction of how users proceed to identify suitable parameter settings for CellProfiler. They confirmed that this approach presents a usability challenge (see Section 1) and explained that parameter optimization can typically require anything from tens of minutes to two days’ work. It is generally not possible to generate image analysis output in real-time. Algorithms are often complex and rarely have linear complexity. In practice, algorithms incur a delay of a few seconds, in the best case, to several hours, in the worst case. Image size also has an influence on the delay users can expect and as image resolution increases - for example, histology images can measure 80,000 × 100,000 pixels [ 23 ] - time delays will be part of image analysis for the foreseeable future. This presents two problems. First, while users wait for algorithms to execute, they are not adding value. Second, waiting for algorithms to finish interrupts their train of thought when optimizing parameters. Our analysis of the parameter optimization process and the algorithms involved led us to formulate a first user requirement: R1. Turn the linear and restricted process of parameter optimization into a more unconfined real-time analysis experience. Support users by separating the time-consuming image analysis processing - where their attention is not required - from the specification of input parameters and the inspection of output. Requirement 2 We next turned our attention to the input parameters. Our analysis revealed three parameter classes: Aliases. Users are required to supply text labels for images or identified objects. Such aliases are used to refer to images or objects to operate on in downstream modules. Aliases are typically input once, are rarely changed, and do not require much time or thought to configure. Nominal parameters. In most modules, users are presented with lists of options from which to pick one. Examples include picking an alias from the list of those introduced in upstream modules or picking a particular variant of an operation such as thresholding. There are usually only a few options and once set, they are rarely changed. Nominal parameters do not add significant time costs or memory load. Continuous parameters. The parameters that users do struggle with require them to input metric values from continuous ranges. For example, users may have to supply a thresholding correction factor. Users experience tuning continuous parameters as difficult and extremely time consuming, but essential to ensure high quality output. When we suggested this to our assay developer collaborators they agreed wholeheartedly: parameters with continuous ranges are the ones that users repeatedly change. The input parameter settings of some modules are harder to optimize than others. Users spend a lot of time tuning the parameters of those modules responsible for detecting objects in images. The assay developers we interviewed hence expressed a strong conviction that users would prefer to optimize individual modules, as opposed to an entire pipeline. Furthermore, current CellProfiler users - novice and experienced alike - particularly appreciate the fact that a modular arrangement offers a conceptually convenient way to chunk the larger parameter optimization problem into meaningful and tractable sub-problems. Based on this preference for a modular approach and our characterization of input parameters, we formulated a second user requirement: R2. Enable users to optimize the input parameters of individual modules of their choosing. Focus on continuous parameters and support users in arriving at suitable settings for these. Requirement 3 To identify suitable settings, users currently proceed stepwise, starting with the first module and working toward the last. For each module they tweak the parameters in the iterative fashion described previously until they are satisfied with its output. They then move on to the next under the assumption that output of all modules that precede it are of sufficient quality. Users’ goal is to discover and understand the relationships between input parameter settings and output. They want to find parameter values that lead to high-quality outcomes and want to avoid values that lead to low-quality outcomes. Users generally know what they expect from a module. For example, if their aim is to identify cell nuclei, they are able to do so manually by referring to the input image. Such expert knowledge is often used as ground truth with which to quality check the results of image analysis modules. By studying users’ workflow we defined a third and final user requirement: R3. Support users in analyzing image-based outcomes in the context of input parameter settings and a reference image. Enable users to identify parameter settings that lead to high- and low-quality outcomes in order to optimize parameters.

Visual analysis

To address Requirement 3, we developed Paramorama, a prototype for the visual analysis of relationships between input parameter space and image-based outcomes. We use hierarchical clustering of the sampled parameters as starting point. As Figure 4 shows, and detailed below, the user interface consists of three views. First, a compact node-link representation provides an overview of the clustering hierarchy and serves as the primary navigation device ( Figure 4(a) ). Second, there is a refinement view for the analysis of user-selected areas of interest in the clustering hierarchy ( Figure 4(b) ). For this, we have developed a novel layout algorithm that provides perceptual support for investigating and comparing the image-based outcomes in the context of parameter space. Finally, we provide a reference image view to directly compare image-based outcomes with an input image ( Figure 4(c) ). Views are coordinated and are augmented with interactive features such as dynamic querying, user-defined parameter ordering, and user-specified tagging ( Figure 4(d)–(j) ).

Hierarchical clustering

Although we are considering parameters with continuous ranges, due to sampling, their domains have been discretized. An obvious approach is to represent the different combinations of parameter values as a table. As shown in Table 1 , columns represent the parameters p k while rows, or tuples, contain all the parameter values generated by sampling. The numerical values that have been sampled have a natural ordering. To respect this in a tabular representation, the tuples t i can be ordered first for p 1 , then for p 2 , and so forth (like column sorting in a spreadsheet). The first column will contain | p 1 | blocks of identical values (with | p 1 | the cardinality of p 1 ). The i th block will contain all tuples where p 1 assumes the i th value in its domain. For every block in the first column there will be | p 2 | blocks in column 2, and so on. Sorting tuples in this fashion produces a hierarchical arrangement of parameter values, which we took as the basis for defining a hierarchical clustering on tuples. As shown in Figure 5 , level k in the hierarchy corresponds to parameter p k −1 (the root, at level 1, contains all tuples). We argue that hierarchical clustering offers a number of advantages over a tabular representation. A tree constructed as above respects the ordering of parameter domains. It also provides an accurate overview of all combinations of sampled values that is convenient to navigate. Instead of sorting columns in a table, the ordering of leaves can be changed intuitively by reordering parameters. As shown in Section 5, this enables users to identify robust combinations of parameters by looking for contiguous regions in the clustering hierarchy that correlate with high-quality outcomes. Finally, it has been shown that representing combinatorial data in a hierarchical fashion enables users to answer queries rapidly and intuitively across multiple data dimensions if provided with sufficient interactive capabilities [ 18 ].

Overview

This presents a structured overview of the sampled parameter space by showing the clustering tree as a directed node-link diagram (see Figure 4(a) ). The overview serves as the primary navigation aid with which users can select areas of interest in parameter space to view in more detail. As users move the mouse cursor over nodes the subtree below the node in focus is highlighted in red. When a subtree is selected, the highlighting changes to blue and it is shown in more detail in the refinement view (see next section). Following standard practice, users can add multiple areas of interest by selecting further subtrees and remove selections by clicking on them a second time. We based our choice of tree visualization on evidence suggesting that node-link representations are often better suited for showing and navigating hierarchical relationships than other representations such as icicle plots and treemaps [ 1 , 12 ]. We considered different node-link layouts, including radial, but settled on a directed node-link diagram because it preserves the ordering of data values more explicitly by encoding them in only one direction (top-to-bottom in our case). Another consideration was the aspect ratio and compactness of the representation to maximize real-estate for the refinement view (see next section). To be space efficient, a radial layout requires an aspect ratio of 1:1. In practice, as here, visualization designers often wish to place compact overviews in an elongated area, which a directed layout utilizes better. We orient the node-link diagram from left to right to label nodes horizontally with their associated data values. Nodes are ordered from top to bottom as their associated values increase. Parameter values are encoded by a perceptually ordinal and colorblind safe color scale [ 3 ]. Luminance is high and saturation low to not clash with other color cues (see later sections). The directed layout also enables us to label each level of the tree with the parameter to which it corresponds. Refinement view The path from the root to each leaf in the clustering hierarchy represents a unique combination of sampled parameter values. Each leaf node has a unique image-based outcome associated with it. To enable users to view and compare these results, we provide a refinement view (see Figure 4(b) ). This view uses a novel layout algorithm to provide users with perceptual support for the analysis of the image-based outcomes produced by sets of parameter values. The algorithm has the following properties: It preserves the top-to-bottom ordering found in the overview, so there is a direct mapping between contiguous regions of parameter space in the overview and their associated outputs in the detail view. It allows users to specify at which parameter level to position outcomes side-by-side. By being able to change this cutoff threshold, users can dynamically change the layout to suit their review and comparison requirements. It shows the outputs of the selected subtrees in adjacent, but visually distinct areas, in the order they appear in the overview. This helps users to map each selected subtree in the overview with its representation in the detail view. These properties of our algorithm assist users to understand how outcomes vary with respect to different parameter values. Our layout algorithm is outlined below. For each subtree T that has to be displayed, LayoutAllTrees() assigns an adjacent rectangular area with width proportional to the number of subtrees that T contains at the user-specified cutoff level t . Layout Tree() then lays out each subtree T on the area it has been assigned by recursively traversing the nodes in T and nesting them in a manner inspired by treemaps [ 11 ]. Nodes at depth t are positioned side-by-side by dividing the width of the area assigned to a parent node between siblings. Other nodes are positioned top-to-bottom by division of the parent’s assigned height. procedure LayoutAllTrees( L T , A,t ) ▷ L T the list of subtrees to layout ▷ A the rectangular area available for layout ▷ t the cutoff threshold at which to wrap subtrees for i ← 1, L T . size () do T ← L T . get ( i ) c ← c + T.getNumberSubtreesAtLevel ( t ) end for anchor ← 0 for i ← 1, L T . size () do T ← L T . get ( i ) c prime; ← T.getNumberSubtreesAtLevel ( t ) A′.x ← anchor A′.y ← A.getY () A′.width ← ( c′ / c ) × A.getWidth () A′.height ← A.getHeight () anchor ← anchor + A′.getWidth () return LayoutTree( T , A′ , t ) end for end procedure procedure LayoutTree( T,A,t ) ▷ T the subtree to layout ▷ A the rectangular area available for layout ▷ t the cutoff threshold at which to wrap subtrees T.x ← A.getX () T.y ← A.getY () T.width ← A.getWidth () T.height ← A.getHeight () c ← T.getChildCount () for i ← 1, c do C ← T.getChild ( i ) if C.level () = t then w ← A.getWidth ()/ c A′.x ← A.getX () + ( i − 1) × w A′.y ← A.getY () A′.width ← w A′.height ← A.getHeight () else h ← A.getHeight () /c A′.x ← A.getX () A′.y ← A.getY () + ( i − 1) × h A′.width ← A.getWidth () A′.height ← h end if return LayoutTree( C,A′,t ) end for end procedure As an example of the results that our algorithm produces, consider the following. In Figure 4(a) , 11 subtrees of parameter space have been selected and these are mapped to 11 distinct regions in the refinement view in Figure 4(b) . One of the subtrees contains 64 combinations of parameter values ( Figure 4(d) ). Of the corresponding outcomes 48 are visible and 16 have been suppressed ( Figure 4(e) ). Image-based outcomes are embedded directly in the layout. As noted, the cutoff threshold t is user-specified and enables direct comparison of selected subtrees with respect to values assumed for the parameter with index equal to t . The value of t is shown in the overview by a vertical blue line that intersects the hierarchy at the specified level ( Figure 4(f) ). The above pseudocode provides the gist of our layout algorithm. It is easily extended to add framed borders at each level of the hierarchy. It is also straightforward to extend the algorithm to assign leaf nodes equal height and to align all subtrees to the top of the display as we have done in our implementation. As with the overview, users can interact with the refinement view directly using their mouse cursor. Reference image Requirement 3 calls for a reference image. This is typically the image that was originally fed into and processed by CellProfiler to detect objects. For example, if users start with an image containing individual cells, a collection of images containing cell outlines will be generated. Users are interested in comparing such outlines with the original image to judge the quality of results and to relate this to parameter values. Our prototype enables users to load a reference image and shows this in the reference image view (see Figure 4(c) ). When users move the mouse cursor over scaled images in the refinement view, the thumbnail that has the focus is overlaid on the reference image in this view. Users can also tear out the reference view to display it in a popup window that can be moved closer to a specific set of results in the refinement view. Interaction Our prototype consists of multiple coordinated views and it provides a number of ways to interact with the visualizations using the widget panel or direct manipulation. This includes brushing, tagging, parameter reordering, and filtering (see Figure 4(d)–(j) ). Brushing is used to highlight the current focus of a particular view in all other views. When the focus is on a single leaf, the image associated with that result is shown at a higher resolution in the reference image view. Color schemes are consistently applied across views: data values are encoded with a sequential color map, selections are highlighted in blue (see Figure 4(d) ) and the current focus in red ( Figure 4(g) ). We show users the values of all parameters that correspond to their current focus in a text area below the refinement view. We found that traditional tooltips obscured too much of the display when a number of parameters and the values they assume have to be listed. Our overview and refinement mechanism enables users to select subsets of the data and to view these in more detail while retaining context. To address Requirement 3, our prototype supports two high-level modes of selection, which are seamlessly integrated and do not require an explicit mode switch. The first facilitates drill-down and has already been discussed: users can select areas of interest in the overview, consider these in greater detail in the refinement view, and view individual results in even greater detail using the reference image view. To make selection less arduous, we also provide a facility to step through selections at a user-defined level in the clustering hierarchy. Using buttons or keyboard shortcuts, this tool selects the next unselected subtree and updates the refinement view accordingly. To aid selection, the overview distinguishes between nodes that have been considered (solid outlines) and those that have not (dashed outlines). The second mode of selection was designed so that users can keep track of and relate high- and low-quality outcomes to parameter values. This is achieved by tagging results as positive or negative using mouse clicks in the refinement view (see Figure 4(h) and (i) ). Results tagged as positive are color-coded in a green and those tagged as negative in magenta in both the overview and refinement view. These colors were chosen to be colorblind safe [ 3 ]. Users can toggle between tagging positive or negative using shortcut keys or the widget panel. To change the levels of the clustering hierarchy, users can interactively reorder the parameters. The visualizations are automatically updated and all previous selections and tags remain intact. As mentioned before, users can specify the cutoff level at which selected subtrees are displayed side-by-side in the refinement view. Finally, we offer users interactive filtering, also called dynamic querying [ 19 ], in two distinct modes. For parameter-based filtering users define an active range using the range sliders in the widget panel. Those parts of the clustering hierarchy that fall outside this range are filtered out in both the overview and refinement view. Users can specify whether to gray out filtered data - as shown in Figure 4(j) - or to hide them completely. For filtering based on user-assigned tags, users can filter out any combination of positive, negative or untagged data using checkboxes in the widget panel. Again, these can be grayed out or hidden from view.

Visual analysis

We now describe how our technique enabled a novice CellProfiler user (first author) to systematically analyze those regions of parameter space that yielded low- and high-quality results. The user started with a clustering hierarchy based on the parameter order described above ( p 1 − p 5 ). The cutoff threshold for the refinement view was set to four to easily compare the nucleus outlines for adjacent outcomes in parameter space. The user began by using the step-through selection feature for the hierarchy shown in the overview: the next previously unseen subtree at level three (see Figure 6(a) ) was accessed with a single mouse click to show a 4 × 4 matrix of outcomes ( Figure 6(b) ). By comparing these images and by moving the mouse over individual images to superimpose them on the reference image, the user easily identified poor results and tagged these ( Figure 6(c) ). For the negatively tagged outcomes, note the missing outlines for the two nuclei just above the nucleus in the lower right ( Figure 6(d) ). The nested hierarchical representation in the refinement view enabled the user to save time by tagging entire subtrees, as opposed to tagging individual outcomes. After identifying a few poor result sets the user suspected that these poor outcomes occurred when parameter p 4 = 13, its highest sampled value. The user was able to quickly confirm this by moving p 4 to the first level in the clustering hierarchy and by scanning all outcomes clustered under the last value for p 4 in the refinement view, none of which identified the two nuclei mentioned in the previous paragraph. All outcomes for this value of p 4 were then tagged as negative, as shown in the bottom subtree of Figure 6(e) . To reduce the complexity of subsequent analysis, the user next used the filtering facility of our prototype to hide all these negatively tagged outcomes. Using an approach similar to the one described above, the user again starting scanning the parameter space by stepping through subtrees at level three of the clustering hierarchy. The user quickly saw that when p 2 = {0.009, 0.09} the nucleus at the bottom left of the image was correctly detected (see Figure 7(a) ). The user tagged as positive those outcomes where this nucleus had been identified (compare Figure 7(b) with Figure 7(c) ). Next, the user reordered the parameters by moving p 2 to the first position. This enabled the user to quickly select all outcomes where p 2 = {0.009, 0.09} in the overview and to visually inspect them in the refinement view. The user was able to quickly see that high-quality outcomes were independent of the three unfiltered values that were sampled for p 4 . At this point, the user could have accepted the parameter values and proceeded with automatically analyzing a large collection of similar images. Alternatively, it would have been possible to reapply the sampler to sample more finely within this region and to visually inspect the outcomes again. Using the above approach, the novice user was able to identify in approximately 10 minutes a contiguous part of the parameter space that yields high-quality results. This is comparable to the time required by an expert CellProfiler user to identify optimal parameter values using the conventional parameter optimization approach and, in our experience, far quicker than a novice taking this approach.