
\chapter{Exploratory and Confirmatory Analyses\label{chapBB:ExpConfAnal}}

This chapter discusses the second and third phases of OODA (\index{exploratory analysis|(}exploratory
analysis and confirmatory analysis) that were introduced at the beginning
of Chapter \ref{chapBA:DataObjDefn}. 

\section{Exploratory Analysis - Discover Structure in Data \label{secB2:Data-Visualization}}

Data visualization\index{visualization|(}, as illustrated for example
using the Spanish Mortality data\index{Spanish Mortality data} in
Figures \ref{fig:Mortality-Raw}-\ref{fig:Mortality-Scatterplot},
is a very important part of exploratory data analysis. A personal
opinion is that it should represent a larger part of statistical training,
and of funded research, than it currently does. The present state
seems to be driven by statistical models and goals (for example analyzing
causality) becoming increasingly complex, which led to a tendency
to co-opt a large share of attention in the field. However visualization
is not only important for exploratory analysis and understanding how
data objects\index{data objects} relate to each other as demonstrated
in Figures \ref{fig:Mortality-Raw}-\ref{fig:Mortality-Scatterplot},
it is frequently also important for the effective choice of data object
(e.g. whether or not to transform), and further provides important
reality checks.

Important references on data visualization\index{visualization!references}
include \citet{tufte1983visual}, \citet{Cleveland1985Elements,Cleveland1993Visualizing}
and \citet{Tukey1990VisualDisplay}. These works contain many useful
ideas and discussion of what comprises good graphics, although they
can sometimes be overly prescriptive. The rest of this section considers
two specific types of data visualization that are critical to OODA.

A perhaps too often ignored, but frequently critical, step in OODA
is the study of marginal distributions. Visualizations of marginal
distributions, e.g. by histograms or QQ plots\index{QQ plots}\index{},
are common when there is time for careful analysis of classical small
scale data sets. This often proves very useful in handling variables
with strong natural skewness\index{skewness}, indicating a potential
benefit from transformation\index{transformation} (see Section \ref{secD2:Preprocess-Trans}
for much more on this), and also in the case of strong outliers, which
depending on the context can either be deleted or handled through
the use of robust methods (see Chapter \ref{chapO:Robust-Methods}).

A reason that this step seems challenging in high dimensional contexts
is that there are generally just too many variables (i.e. features
or traits) to humanly comprehend the population structure of all of
them. A careful analyst will try to look at some representatives,
but it may not be obvious how to choose those. This problem is addressed
using the graphical device of \emph{marginal distribution plot\index{marginal distribution plot}s}
in Section \ref{secD1:Preprocess-MargDists}, where case studies on
the Spanish Mortality data (from Section \ref{sec:Curves-as-Data})
and the Drug Discovery data\index{Drug Discovery data} set are provided. 

As illustrated in Section \ref{sec:Curves-as-Data}, PCA\index{Principal Component Analysis|(}
is an effective and commonly used tool for exploring modes of variation\index{modes of variation|(}.
These give insights into how data objects relate to each other, such
as exploring potential clusters. \citet{RamsaySilverman2002,RamsaySilverman2005}
made it clear that PCA is a powerful tool for understanding variation
in FDA, i.e. curves as data object contexts. Less well known is that
this insightful idea was first published in \citet{Rao1958PCAgrowthCurves},
in the context of analysis of growth curves. Basics of PCA are described
in Chapter \ref{chapP:PCA-Details}. Important ideas discussed there
include the fact that the good idea of PCA has been rediscovered (and
generally given different names) a number of times, and that the misconception
that PCA is only useful for Gaussian data sets (because one motivation
of it is via Gaussian likelihood ideas) is seriously misleading. The
latter point is also clear from several of the examples given in this
section.

\subsection{Example: Tilted Parabolas FDA \label{subsecB.B.1:ExampleParabolas}}

Figure \ref{fig:10d-ToyPCA} shows an FDA\index{functional data analysis|(}
toy example to illustrate the concept of decomposition into modes
of variation, in the spirit of Figure \ref{fig:2dToy-Decomp}. The
$n=50$ input raw data curves are shown in the top left panel. These
are simulated to have an approximately parabolic shape, but variation
of several types is included as well, hence the name \emph{Tilted
Parabolas}.\index{Tilted Parabolas data|(} Each curve is really just
a parallel coordinates plot (as discussed in Section \ref{secBA.1:Foundations_Terminology})
of a collection of 10 dimensional vectors, but conceptually it makes
sense to think of a bundle of smooth curves.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigG1CMYK}

\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigG2CMYK}

\caption{Tilted Parabolas 10-d toy example to illustrate concept of modes of
variation in FDA. Top row has the raw data curves on the left, mean
curve in the center and mean residuals on the right. Remaining rows
show PC components, with mode of variation plots (projection\index{projection|(}s)
on the left, residuals center and distribution of scores (projection
coefficients) on the right. This demonstrates insights available by
decomposing data into modes of variation. \label{fig:10d-ToyPCA}}
\end{figure}
The object space\index{object space|(} - feature space\index{feature space|(}
concept illustrated in Figures \ref{fig:2dToy-Mean} - \ref{fig:2dToy-PC2}
is useful here, except that explicit visualization of the feature
space is not done because that space is $\mathbb{R}^{10}$ for this
data. Nonetheless, it is still useful to think of statistical analysis
as being done in that space on the cloud of points that represents
the bundle of curves, while looking at the corresponding object space
(i.e. curves) view. Colors are based on the Matlab default rotating
color palette, with the same colors used in the other panels for visual
correspondence.

The top center panel of Figure \ref{fig:10d-ToyPCA} shows the first
natural statistical summary: the sample mean. Again, it is useful
to think of that curve as the object space representation of the mean
of the cloud of points in the feature space ($\mathbb{R}^{10}$).
The mean curve can also be considered as the point-wise mean of the
curves in the top left panel. The top right panel shows the mean residuals,
which are a visualization of the curves that correspond to shifting
the point cloud in $\mathbb{R}^{10}$ so that it is mean centered
at the origin. These residuals already highlight an interesting aspect
of the data: the parabolic shape of the curves is driven entirely
by the mean, and not the variability about the mean. 

The next three rows show the results of a PCA decomposition into modes
of variation, of the same type shown in Figures \ref{fig:Mortality-PC1}
- \ref{fig:Mortality-PC2} and \ref{fig:2dToy-PC1} - \ref{fig:2dToy-PC2}. 

The first mode of variation is the left plot in the second row. As
discussed in Section \ref{sec:Curves-as-Data}, this is based on finding
the direction in the feature space ($\mathbb{R}^{10}$) that maximizes
the projected variation (in the sense illustrated in Figure \ref{fig:2dToy-PC1}),
projecting each mean residual curve onto that direction, and then
showing the resulting set of curves (as the projection coefficient
multiplied by the direction vector). Recall that this set of curves
are columns of a rank one matrix. In particular, they are all multiples
of the same curve (which is the curve representation of the direction
vector in the feature space). This clearly shows that the first mode
of variation is essentially a vertical shift. With this knowledge
in hand, that mode can clearly be seen also in the mean residuals
on the top right, as well as in the raw data on the left. 

The right panel in the second row, shows the distribution of the projection
coefficients, i.e. the \emph{scores\index{scores|(}}, again with
corresponding colors (e.g. the gold followed by gold and red on the
right correspond to the same colored curves on the bottom of the left
hand panel). The format of these scores distribution plots\index{scores!distribution plot}
is the same as that used in Figures \ref{fig:Mortality-PC1}, \ref{fig:Mortality-PC2},
\ref{fig:Mortality-MargDistPlot} and \ref{fig:MortalityLog-MargDistPlot},
where each score is represented with a symbol, and the black curve
is a smooth histogram. Because there is no special ordering in this
data set, the height of the points in the PC scores plots can be considered
to be random. Using such displays with simulated random heights is
the \emph{jitter plot\index{jitter plot}} idea proposed by \citet{TukeyTukey1990JitterPlots}
as a device for visualizing one dimensional data sets. In many displays
here (also called jitter plots) order in the data is used for heights
as that can reveal aspects of the data which would disappear using
random heights. Specific examples of this include Section \ref{subsecD.1.A:SpanishMortalityData}
and the discussion of Figure \ref{figJ.C:HighDClass}. The center
panel shows the corresponding PC1 residual curves, each of which is
just the centered residual minus its PC1 projection. Note that these
are also the projections of the mean residuals onto the hyperplane
orthogonal to the PC1 direction.

The third row shows the second mode of variation. The left hand panel
is the object space representation of the projections of the PC1 residuals
(2nd row middle panel) onto the 2nd PC direction in $\mathbb{R}^{10}$.
Note that this also shows a very interpretable mode of variation,
a random tilt. This mode is much harder to see in either the raw data
curves, or the mean residuals, demonstrating the ability of PCA to
find interesting modes that are not visually apparent in the raw data.
The PC2 scores (i.e. the projection coefficients) in the right panel
show much less variation than for PC1, and the PC2 residuals in the
center panel also show relatively less variation.

The fourth row shows PC3, i.e. third mode of variation. The loadings
plot in the left panel looks rather random. This is because the data
were simulated as 
\begin{equation}
\left(x-6\right)^{2}+4Z_{1,j}+0.5Z_{2,j}\left(x-5\right)+Z_{x,j},\label{eqB.A:Generate10dEg}
\end{equation}
for $j=1,\cdots,50$, where $x=0.5,1.5,\cdots,9.5$, and where the
$Z_{x,j}$ are independent standard Gaussian random variables. Note
the coefficients are deliberately chosen to make these components
correctly ordered in PCs 1,2,3. Since the noise terms $\left(Z_{0.5,j},\cdots,Z_{9.5,j}\right)^{t}\in\mathbb{R}^{10}$
follow an isotropic Gaussian distribution, the PC3 direction is random.
The relatively small scale of the noise is also clear from the tightness
of the PC3 scores shown in the right panel. Also the PC3 residuals
in the center panel show that PC3 explains relatively little of the
variation in the PC2 residuals above, again because the noise is isotropic,
and thus evenly distributed among the remaining directions in $\mathbb{R}^{10}$.
Again these PC3 residuals are the PC2 residuals above minus the PC3
projections to the left.

As discussed in Section \ref{subsecBA.A.C:ScreePlots}, a useful viewpoint
on these issues comes from various sums of squares (in the spirit
of Analysis of Variance). The fact that the PC1 projections explain
most of the variance is quantified by the sum of squares of the PC1
projections (left, 2nd row) representing 86\% of the sum of the mean
residual sum of squares. The visual impression that the PC2 projections
(left, 3rd row) contain less variation is clear from the sum of squares
being only 10.4\%. The remaining sum of squares (i.e. summed over
all remaining PC components, which is also the sum of the residuals
shown center, 4th row) is only 3.6\%, confirming that all of the remaining
variation is quite small. The spherical nature of the remaining variation
is confirmed by the PC3 variation explained being only 0.7\%. Graphical
comparison of these numbers is provided by the scree plot\index{scree plot}
in the lower left panel of Figure \ref{figB-Y:Screeplots}.

Figure \ref{fig:10d-ToyPCA} also provides an additive decomposition
of variation as highlighted in Figure \ref{fig:2dToy-Decomp}. In
particular, the raw data in the top left panel is the sum of the mean
in the top center, the modes of variation in the remaining left panels,
plus the residuals in the bottom center panel.

As discussed in detail in Section \ref{subsecP-A-B:SVD}, the PC direction
vectors used in the above data decomposition are easily computed,
using either an eigen analysis\index{eigen analysis} of the covariance
matrix, or equivalently a \index{Singular Value Decomposition}Singular
Value Decomposition of the mean residual matrix.

A graphical point worth discussion here is the axes used in Figure
\ref{fig:10d-ToyPCA}. In particular (except for the first row) the
vertical axes in the first two columns, as well as the horizontal
axes in the third column, are deliberately taken to be the same (even
across the rows). Such a view is quite nonstandard for most graphics
packages, which generally adhere to the goal of trying to use as much
of the graphics space as efficiently as possible, in particular minimizing
\emph{white space}. While the minimization of white space is generally
a sensible default, in this context it does have an intuitive cost
as demonstrated in Figure \ref{fig:10d-ToyPCA-MinWhiteSpace}, which
is a replotting of the bottom 3 rows of Figure \ref{fig:10d-ToyPCA},
but this time using axes that minimize white space. The difference
between these two figures is perhaps most strong in the bottom row,
which is easily understood as small scale noise artifacts in Figure
\ref{fig:10d-ToyPCA}, but visually appear to be equal players in
Figure \ref{fig:10d-ToyPCA-MinWhiteSpace}. In particular the important
decrease in variation for the higher PC components become much harder
to see (only discernible by carefully studying the axis labels). The
relative \emph{shapes} (horizontal shift in the first mode, tilt in
the second, noise in the third) are now highlighted, at the cost of
it being harder to interpret \emph{relative variation}. However, this
effect can be easily mitigated by also including a scree plot, as
introduced in Figure \ref{figB-Y:Screeplots}, which includes this
data set in its lower left panel. Understanding variation using a
scree plot has already been illustrated in Figures \ref{tab2-B:ObjectFeature}
and \ref{figB-U:DensCDFQuant}.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigH2CMYK}

\caption{Same FDA of \index{Tilted Parabolas data|)}Tilted Parabolas data
as in Figure \ref{fig:10d-ToyPCA}, showing more typical axis choice
to minimize white space. Note this loses the intuitive illustration
of relative amount of variation in the modes.\label{fig:10d-ToyPCA-MinWhiteSpace}}
\end{figure}
One more issue about white space, is that when trying to put a number
of plots on a single page, it can make sense to also eliminate the
white space \emph{between} plots. The \emph{trellis graphic\index{trellis graphic}}
ideas of \citet{BeckerCleveland1996Trellis} provide appropriate ways
to do this. Examples appear in Figure \ref{PCA-DNA-scores}, in the
context of the DNA Molecule data, and in Figures \ref{figO.H:GWAS-PCA}-\ref{figO.J:GWAS-VL1PCA}
for the Cystic Fibrosis GWAS data.

\subsection{Example: Twin Arches FDA \label{subsecB.B.2:ExampleTwinArches}}

Another FDA toy example is the \emph{\index{Twin Arches data|(}Twin
Arches} data shown in Figure \ref{fig:50d-ToyPCA}, whose format is
very similar to Figure \ref{fig:10d-ToyPCA}. This time the $n=50$
data curves shown in the upper left panel, are parallel coordinate
plot\index{parallel coordinate plot}s of vectors in $\mathbb{R}^{50}$.
Details of the construction are given below, but at this point consider
the data in an exploratory spirit. Apparent is a somewhat higher background
noise level than for the Twin Arches data, and also some strong structure
in the data. The center panel shows the sample mean which this time
is essentially constant, so the mean residuals in the left panel are
very similar to the original data curves (again using the same axes
makes this visually apparent, which would be much harder to see with
white space minimizing axes, as in Figure \ref{fig:10d-ToyPCA-MinWhiteSpace}).

Again the second row shows the PC1 mode of variation. The PC1 loadings
plot on the left seems to capture the main twin arch structure, but
note that some of these projection coefficients are essentially zero.
The PC1 scores in the right panel makes it clear that there are actually
three strong clusters in the PC1 direction, i.e. this is a very non-Gaussian
mode of variation. Perhaps more surprising is the object space\index{object space|)}
representation of the PC1 residuals in the center panel. While some
of the residual curves seem to be 0 plus noise, others seem to retain
the same arched structure, for reasons discussed below. 

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigI1CMYK}

\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigI2CMYK}

\caption{Twin Arches toy FDA example in 50-d, illustrating ability of PCA to
find insightful modes of variation. This time the mean is negligible
and the strong arched structure are artifacts of clustering, i.e.
non-Gaussian structure in the data. \label{fig:50d-ToyPCA}}
\end{figure}
The PC2 mode of variation plot, on the left in the third row, may
also at first be surprising. This is because it looks similar to the
PC1 mode just above. But they cannot be similar since these direction
vectors in $\mathbb{R}^{50}$ (recall each such plot consists of multiples
of a single vector) must be orthogonal. A careful look at the colors
reveals what is happening. Notice for PC1, the generally gold color
goes up at the first arch and down at the second. Suggesting this
function is roughly a sine wave. In PC2 the mostly blue color goes
upwards for both arches, while the more red curves go downwards for
each, which is a direction orthogonal to the PC1 eigenvector. By the
way, these colors have not been deliberately assigned, but are just
artifacts of the random generation of the curves, together with over-plotting
effects, where the color tends to be dominated by the last plotted
curves.

As suggested by the PC2 residual curves, the next components are pure
Gaussian noise, with PC directions looking quite random, which thus
are not shown here. 

As noted in the survey paper by \citet{febrero2012statistical}, a
number of FDA software packages aim to integrate PCA with noise reduction
in a single step, as pioneered by \citet{RiceSilverman1991SmoothPCA}.
These include perhaps most notably the FDA package accompanying \citet{RamsaySilverman2002,RamsaySilverman2005},
and the PACE package started by \citet{YaoMuellerWang2005PACE}. While
this process is critically important in many high noise cases, as
well as in the case of uneven and sparse sampling (of the horizontal
coordinates of the curves), in perhaps surprisingly many cases such
as these two examples, it can be enough to simply do naive PCA on
the data. The reason seems to be that when the true underlying direction
is smooth, the noise is averaged out by PCA. In \citet{GaydosKingsolver2013SmoothDirections}
a variation of PCA, which maximizes smoothness instead of variation,
is proposed and integrated with PCA in an interesting way.

An important principle of multivariate analysis is that joint distributions
can contain much richer structure than is apparent from the marginals.
This concept can be used for a more clear understanding of the structure
of the Twin Arches toy data illustrated in Figure \ref{fig:50d-ToyPCA}
by studying bivariate projections in addition to the univariate scores
distributions\index{scores!distribution plot} shown in the right
hand column. Such a view is the\index{scores!scatterplot|(} \emph{scatterplot
matrix\index{scatterplot matrix|(}} shown in Figure \ref{fig:50d-ToyPCAScatMat},
which is in the same format as shown in Figure \ref{fig:Mortality-Scatterplot}
for the Spanish Mortality data\index{Spanish Mortality data}. This
view shows the distributions of the 1-d projection coefficients (PC
scores) along the diagonal, with in particular the first two being
the same as the lower two in the right column of Figure \ref{fig:50d-ToyPCA}.
The off diagonal plots show corresponding two dimensional plots. For
example, the top center panel is the scatterplot of the PC1 versus
PC2 scores. Note this is closely linked with the panel below (the
horizontal axes are the same, so e.g. the left cluster in the PC2
scores is the same as the left cluster in the scatterplot), and with
the plot to the left (where the PC1 score axis becomes the vertical
axis so the left cluster in the PC1 scores is the bottom cluster in
the scatterplot). This PC1 versus PC2 scatterplot gives a clear view
of the underlying structure in this case, there are actually 4 clusters,
which project down to 3 clusters in each of the PC1 and PC2 directions.
Note that the center left plot is just the transpose of the top center
plot. Since this does not convey much new information, the below diagonal
plots are sometimes replaced by other graphics.

The remaining column and row all show the 3rd component. The univariate
PC3 scores distributions appear to be Gaussian, which is consistent
with how the data were generated. Note that a more conventional white
space minimizing choice of axes is used this time, so a careful look
at the axis labels is helpful to see that this mode contains much
less variation than PC1 and PC2. Again the scatterplots in the right
column share the same horizontal axis, so the actually spherical clusters
have been strongly stretched by this axis choice. Similarly, for the
scatterplots in the bottom row, which are just transposes.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigJCMYK}

\caption{Scatterplot matrix view of the Twin Arches data from Figure \ref{fig:50d-ToyPCA}.
Shows clusters apparent in 1-d marginal scores distribution plots
come from marginalizing 4 actual clusters. \label{fig:50d-ToyPCAScatMat} }
\end{figure}
 Scatterplot matrix views tend to be very insightful, and are recommended
for most situations where relationships between\index{data objects!relationships between|(}
data objects are relevant. A natural question is: if 2-d projections
show more than 1-d projections, why not also consider higher dimensional
projections? While potential improvements appear to be obvious, and
implementation is fairly straightforward for 3-d, it does come with
substantial overhead, such as the need for dynamic graphics. These
take substantial energy to both implement and to visually explore,
which can be a large drawback for routine data analysis tasks. For
projections of dimension higher than 3, visualization becomes much
more challenging, and is thus not frequently done.

\subsection{Case Study: Lung Cancer Data\label{subsecB.2.A:LungCancerData} }

While\index{Lung Cancer data|(} toy examples, such as those in Figures
\ref{fig:10d-ToyPCA} - \ref{fig:50d-ToyPCAScatMat} can give many
insights, it is also important to consider real data sets. An interesting
example is the set of curves (called the \emph{Lung Cancer RNAseq\index{RNAseq data}}
data) shown in Figure \ref{fig:LungCancerLogCurves}. These come from
a study of lung cancer, and in particular was an early data set collected
as part of The Cancer Genome Atlas (TCGA), \citet{Weinstein2013TCGA}.
The focus here is on the gene CDKN2A, which has long been known to
be involved in many types of cancer. The horizontal axis represents
the region on the chromosome that is used to produce the RNA measured
here (using the RNAseq technology described in \citet{Wang2009RNAseq}).
For each of the $d=1709$ locations, the vertical axis shows the counts
(on the $\log_{10}$$\left(\cdot+1\right)$ scale\index{scale!log|(})
of pieces of amplified RNA molecules that match the chromosome at
this location. There are $n=180$ such curves. The $\log_{10}$ scale
is useful, and is the data object representation choice used in the
rest of this section, since these counts range over 3 orders of magnitude.
An artifact of this $\log_{10}$ scaling for this data set is that
very small counts, such as 1 and 2 occupy a large chunk of the bottom
of the plot, since $\log_{10}\left(1+1\right)\approx0.301$ and $\log_{10}\left(2+1\right)\approx0.477$.
The same rotating palette of seven colors used in Figures \ref{fig:10d-ToyPCA}
- \ref{fig:50d-ToyPCAScatMat} is used here as well. The curves seem
somewhat chunky in nature, in particular being substantially lower
over some intervals, because these coding regions do not appear in
a contiguous region on the chromosome, but instead are separated into
intervals called \emph{exons}. The union of these exonic regions are
used as the horizontal axis in Figure \ref{fig:LungCancerLogCurves}.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigKCMYK}

\caption{Raw $log_{10}$ count curves of Lung Cancer RNAseq data. Colors are
standard rotating palette used for good contrast of curves. It is
not easy to discern population structure. \label{fig:LungCancerLogCurves}}
\end{figure}
While there is a lot of variation in these curves, it is hard to discern
much structure, although they vary over several orders of magnitude.
As illustrated above a PCA scores scatterplot, shown for the Twin
Arches data\index{Twin Arches data|)} in Figure \ref{fig:LungCancerPCA},
is useful for understanding relationships between data objects. Even
the 1-d scores distributions\index{scores!distribution plot} on the
diagonal already show multi-modal structure, which is quite apparent
in the black smooth histograms. But, as illustrated in Figure \ref{fig:50d-ToyPCAScatMat},
the 2-d off diagonal scatterplots do a much better job of highlighting
the multi-modal structure of the data, where three clusters are immediately
apparent. Statistical significance of these clusters is formally explored
in Section \ref{subsecL-B-A:HiDimSigClust}. Note that in the spirit
of the phenomenon illustrated in Figure \ref{fig:50d-ToyPCAScatMat}
the top two clusters are combined in the PC1 scores, and the right
two clusters combine in the PC2 scores. Only the first 2 components
are shown in Figure \ref{fig:LungCancerPCA}, because the 3rd and
4th components are driven by a few outlying cases, which are not further
studied here. In particular, the first two components explain 82\%
and 8\% respectively, while the 3rd and 4th explain 3\% and 2\%. Another
interesting pattern is the points in the second through fourth quadrants
appear to be bounded below by a line. An open question is what generates
this apparent structure in the data.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigLCMYK}

\caption{Lung Cancer RNAseq PCA scatterplot matrix. Colored circles represent
feature space\index{feature space|)} view of corresponding curves
in Figure \ref{fig:LungCancerLogCurves}. Shows three clear clusters
in the data. \label{fig:LungCancerPCA}}
\end{figure}
Insight into the drivers of these clusters comes from a technique
called \emph{brushing\index{brushing|(}} in \citet{BeckerCleveland1987Brushing}.
The idea is to use colors to keep track of subsets of the data in
multiple graphics. This is illustrated in Figure \ref{fig:LungCancerPCABrushed},
which shows the same distribution of points as in Figure \ref{fig:LungCancerPCA},
with colors that have now been manually chosen to highlight the three
clusters. Automatic versions of clustering can also be done using
a variety of methods as discussed in Chapter \ref{chapK:Clustering}.

Note that the visual impression of these \index{cluster}clusters\index{clustersing}
in the 1-d distributions shown on the diagonal is now enhanced with
appropriately colored versions of the smooth histogram which focus
on each cluster. These are \emph{sub-densities} in the sense that
the area under the main black curve is 1, and the areas under each
colored curve, which are proportional to the cluster sizes, sum to
1. Note that in regions where one cluster is dominant, the colored
sub-density is the same as the black overall density, and thus overplots
it (e.g. red on the left side of the PC1 scores and blue on the left
side of the PC2 scores). In other regions, the relative curve heights
give a clear visual impression of the corresponding cluster proportions.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigMCMYK}

\caption{Brushed PCA scatterplot matrix of Lung Cancer RNAseq data, using colors
to highlight the three clusters. Also shows sub-density estimates,
for visual separation, in the 1-d distributions on the diagonal. \label{fig:LungCancerPCABrushed}}
\end{figure}
This brushing technique is often especially insightful when used across
a variety of graphical displays. An example of this appears in Figure
\ref{fig:LungCancerLogCurvesBrushed}. These are the same curves shown
in Figure \ref{fig:LungCancerLogCurves}, but now the color scheme
developed in Figure \ref{fig:LungCancerPCABrushed} is used. Note
that the red curves are all substantially lower than the others. In
the early days of gene expression these cases would have been labeled
as \emph{unexpressed} (which actually means expression at a much lower
level, recall the log \index{scale!log|)}scale on the vertical axis).
Note that for most exons the blue and brown curves both have high
expression values. There is an exon to the left of center where all
cases seem unexpressed, which is reasonably labeled an annotation
error (an on-going issue with such biological data sets). The most
interesting issue is the exon right of center, where the brown cases
are high, but the blue cases are essentially unexpressed. This is
an event called \emph{alternate splicing} where actually different
versions of mRNA are produced from this chromosome region by different
people. This is very important to the development of new treatments
for cancer, because such phenomena can be targeted by appropriate
drugs.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigNCMYK}

\caption{Same Lung Cancer RNAseq curves as in Figure \ref{fig:LungCancerLogCurves},
now using brush\index{brushing|)}ed colors from Figure \ref{fig:LungCancerPCABrushed}.
Shows clusters are very important, in particular representing alternate
splicing. \label{fig:LungCancerLogCurvesBrushed}}
\end{figure}
While the alternate splicing present in this gene CDKN2A has been
well known for some time, the success in finding it with this type
of visualization motivated \citet{Kimes2014SigFuge} to use this type
of idea to scan the whole genome in search of unknown alternate splices.
A key challenge was that, as noted in Chapter \ref{chapK:Clustering},
most automatic clustering methods always find many clusters, whether
they represent important biological structure as in Figure \ref{fig:LungCancerLogCurvesBrushed},
or not. Hence confirmatory analysis, as discussed in Section \ref{secB3:Confirmatory-Analysis}
and Chapter \ref{chapL:Confirmatory-Analysis} is essential, and was
key to the SigFuge method developed in \citet{Kimes2014SigFuge}.

Another important aspect of data representation\index{data representation}
is \emph{\index{scale}scale} and \emph{normalization\index{normalization}}
issues. These important parts of OODA preprocessing\index{preprocessing}
are discussed in detail in Section \ref{secD4:Standardization}. As
seen in several examples above PCA can be a powerful visualization
device for finding interesting structure in data. But because PCA
is driven by finding directions of maximal variation, it can lose
effectiveness in situations where differing variables (i.e. features
or traits) have different scalings. In particular, PCA will tend to
be driven by those variables with the most variation, while ignoring
those with smaller scale variation. This challenge can be particularly
acute in situations where different variables even measure non-commensurate\index{commensurate}
quantities, such as having different units. Approaches to standardization\index{standardization}
are discussed in Section \ref{secD4:Standardization}.

Another useful response to the tendency of PCA to focus on variation
is to replace PCA directions in high dimensional space with directions
aimed at other aspects of the data. The case of directions highlighting
differences between subgroups is studied in Section \ref{subsecB.2.B:PanCancerExample}.\index{functional data analysis|)}

\subsection{Case Study: Pan Cancer Data\label{subsecB.2.B:PanCancerExample}}

Figure \ref{figB-Q:PanCanPCA} shows a PCA scatterplot view of part
of the \emph{Pan Cancer} data\emph{\index{Pan Cancer data|(}} set,
from \citet{Hoadley2014TCGA-PanCan}, who explored many contrasts
between 12 cancer types, based on a variety of measurements. That
data set was another product of TCGA discussed in Section \ref{subsecB.2.A:LungCancerData}.
The raw data are counts indicating expression of $d=12478$ genes,
again measured using RNAseq. Substantial preprocessing, including
log transformation\index{transformation!log} as discussed in Section
\ref{secD2:Preprocess-Trans} have been done by \citet{Hoadley2014TCGA-PanCan}.
Studied here is a subset of $n=300$ cases (this number gives clear
visualization of the main point about the limitations of PCA) with
50 from each of the cancer types Bladder Cancer (BLCA, magenta), Kidney
Renal Cancer (KIRC, blue), Ovarian Cancer (OV, cyan), Head and Neck
Squamous Cell Cancer (HNSC, green), Colon Adenocarcinoma (COAD, yellow)
and Breast Cancer (BRCA, red). While each of the six cancer types
can be clearly seen, there is substantial overlap of the classes in
this view. This is because the PCA directions only maximize variance,
and they ignore class labels.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigQPanCanPCACMYK}

\caption{PCA scatterplot matrix view of the Pan Cancer gene expression data,
symbols are tissue samples, colored according to cancer type. Note
substantial overlap of several cancer types. \label{figB-Q:PanCanPCA}}
\end{figure}
Figure \ref{figB-R:PanCanDWD} shows an alternate scatterplot view
of the same Pan Cancer gene expression data shown in Figure \ref{figB-Q:PanCanPCA}.
The symbols and colors are the same, but instead of using PC directions
for the axes, the directions used in the projections are designed
to deliberately separate pairs of cancer types. Each direction is
based on the DWD\index{Distance Weighted Discrimination|(} (Distance
Weighted Discrimination) method, trained on pairs of cancer types.
DWD was also used in Section \ref{sec:Images-as-Data} to visually
separate the male and female faces, and is discussed in more detail
in Section \ref{secJ-D:DWD}. The projection direction used in the
first row and column is DWD trained on only the Kidney (blue) versus
the Head and Neck (green) cancer types. The projections of the full
data set onto that direction (although DWD was trained on just those
two) are shown in the upper left and on the same horizontal axis in
the other first column plots, as well as the vertical axis of the
other plots in the top row. Note that both cancer types stand out
as distinct clusters in these views, so DWD has succeeded in separating
out the expected biological differences. 

Similar excellent separation happens for Colon Cancer (yellow) in
the second DWD direction in Figure \ref{figB-R:PanCanDWD}, although
the Bladder Cancer (magenta) stands out less well. In the third DWD
direction the Ovarian Cancer (cyan) stands out clearly, while Breast
Cancer does not. The latter is not surprising because Breast Cancer
is well known, see e.g. \citet{perou2000molecular}, to have several
subtypes which are quite distinct from each other. Presumably each
of these would be clearly different from the others, but because of
their diversity the union fails to be very distinct in this sense.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigRPanCanDWDCMYK}

\caption{Same Pan Cancer data from Figure \ref{figB-Q:PanCanPCA}, with PC
directions replaced by directions deliberately aimed at separation
of types (highlighted with colors). Shows much better distinction
of cancer types, demonstrating that PCA directions may not find all
interesting structure in data. \label{figB-R:PanCanDWD}}
\end{figure}
In\index{Pan Cancer data|)} earlier scatterplot matrix\index{scatterplot matrix|)}
views the axes were orthogonal, as that is a consequence of PCA. However,
there is no guarantee of orthogonality otherwise, as can be seen in
several off-diagonal panels of Figure \ref{figB-R:PanCanDWD}. These
views should be considered as projections onto the two dimensional
subspaces generated by the pair of non-orthogonal\index{non-orthogonal axes}
directions. In each case the horizontal axis shows that direction,
and the vertical axis is just its orthogonal complement (in the 2-d
subspace). The other line shows the second direction in that subspace.
For example, in the first row second column, the nearly vertical line
is approximately a transport of the line between the means of the
dark blue (KIRC) and green (HNSC) clusters. More detail of this type
of visualization appears around Figure \ref{figF.H:NonOrthoAxes}. 

One might wonder how these particular pairings of cancer types (on
which the DWD directions were trained) were chosen. This was done
by considering all pairings and deliberately choosing a set of three
on the basis of good visual distinction of types. But the main point
of these figures is that there can be visualization directions giving
much different visual insights than those available from PCA.\index{Principal Component Analysis|)}

Another striking example of PCA providing not the best separation
of cancer classes can be found in \citet{LiuPerouMarron2009CrossPlatVisualization}.

The methods and examples studied in this section provide a somewhat
non-standard way of thinking about high dimensional data. The currently
fashionable notion in much of statistics is that when faced with high
dimensional data, one must use approaches such as \emph{sparsity\index{sparsity}},
i.e. treating most variables as negligible, to reduce the data to
a ``manageable dimensionality''. While sparsity is a useful approach
in some cases and has been tackled effectively using a very large
range of methods starting with the LASSO approach of \citet{Tibshirani1996Lasso},
there seem to be many more OODA contexts where the fundamental sparsity
assumptions are far from being reasonably well satisfied. These include
almost all of the examples discussed in Chapter \ref{chap1B:Breadth-of-OODA},
and also the rich genetic data discussed in Figures \ref{fig:LungCancerLogCurves}
- \ref{fig:LungCancerLogCurvesBrushed}. Yet sparsity ideas seem to
be currently both over used and over studied in the statistics literature,
perhaps because most statisticians tend to think about high dimensional
data in a too \emph{variable centric} way. The OODA viewpoint demonstrated
in this section allows taking a more \emph{object centric} approach,
where the primary focus is more usefully placed on the data objects
and the relationships between\index{data objects!relationships between|)}
them, not the variables. Of course variables are important, but they
should be playing the role of representer\index{representer}s of
the objects, as opposed to being the focus of the analysis. 

However variables often contain useful insights about the \emph{drivers}
of the relationships displayed in the above object oriented views.
This is often usefully done through visualization of loadings\index{loadings!plots|(},
i.e. the entries of the direction vectors of each mode of variation\index{modes of variation|)}.
One type of loading visualization was done in the left panels of Figures
\ref{fig:Mortality-PC1} and \ref{fig:Mortality-PC2}, where the curves
are multiples of the direction vectors. That curve visualization is
less useful in other settings, such as the gene expression data shown
in Figures \ref{figB-Q:PanCanPCA} and \ref{figB-R:PanCanDWD}, where
there is no natural insightful ordering of the variables (critical
to the display of curves) and often simply too many variables ($d=12478$
in the present case) for a useful curve display. Some solutions to
this are demonstrated in Figure \ref{figB-X:PanCanDWDLoadings}. The
top panel shows indices (horizontal axis) of the sorted loadings (vertical
axis) from the DWD direction vectors trained to separate the KIdney
Renal clear cell Carcinoma (KIRC) patients from those with Head and
Neck Squamous cell Carcinoma (HNSC). Note there are just a few very
large both positive and negative values. Those strong loadings are
studied in the bottom panel, with bars showing the top twenty (in
absolute value) variables that are labeled with gene names. Validation
of the relevance of these genes comes from the \citet{NCBI2019} website.
In particular, the most strongly expressed (in the HNSC direction)
gene LYPD3 is commonly expressed in esophagus and skin, while the
genes that are strongest in the KIRC direction, CUBN and RBP5, are
involved in activity in the intestine and kidney, and kidney and liver,
respectively.

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigXPanCanDWDLoadingsCMYK}

\caption{Loadings of first DWD direction (KIRC vs. HNSC) of cancer data in
Figure \ref{figB-R:PanCanDWD}. Top panel shows sorted loadings for
all $d=12478$ genes, bottom panel zooms in on top 20 (absolute value)
gene names. \label{figB-X:PanCanDWDLoadings}}
\end{figure}
Further discussion of various roles played by loadings,\index{loadings!plots|)}
especially for PCA, can be found in Section \ref{subsecP-A-B:SVD}.
\index{exploratory analysis|)}

\section{Confirmatory Analysis - Is It Really There? \label{secB3:Confirmatory-Analysis}}

The\index{confirmatory analysis|(} visualization methods discussed
in Section \ref{secB2:Data-Visualization} are very good at providing
useful insights and at finding population level structure in data.
However an important aspect to keep in mind is that they also have
the potential for finding useless artifacts of sampling variation. 

This point is illustrated using the Overlapping Classes\index{Overlapping Classes data|(}
simulated data sets in Figure \ref{fig:HiDimToyPCA }. Here two classes
of data were generated in $d=1000$ dimensions, with $n_{1}=n_{2}=50$
data points in each class. It is hard to see much difference between
the red class (shown as circles) and the blue class (shown as plus
signs) in the PCA scatterplot view shown in Figure \ref{fig:HiDimToyPCA }.
However, an important lesson from the Section \ref{secB2:Data-Visualization}
is that for high dimensional data, PCA may not find all interesting
aspects of the overall distribution because it focuses only on variation
in the data.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigSCMYK} 

\caption{PCA scatterplot view of the Overlapping Classes data in 1000 dimensions.
Shows no apparent difference between classes. \label{fig:HiDimToyPCA } }
\end{figure}
Figure \ref{fig:HiDimToyDWD} more deliberately targets the class
difference between the red plus signs and the blue circles, using
the DWD direction which was previously used in Figures \ref{fig:Faces-DWD}
and \ref{figB-R:PanCanDWD}. Projections on this DWD direction appear
in the upper left panel. Note that this shows a very clear and distinct
separation of the two classes which is visually comparable to some
of those seen for the Pan Can data in Figure \ref{figB-R:PanCanDWD}.
The other two directions are orthogonal PC directions, which are computed
using PCA based on the projection of the data onto the $d=999$ dimensional
subspace orthogonal to the DWD direction. This is useful because it
allows the directions used in the scatterplot matrix to be orthogonal,
which generally makes the view more interpretable. These issues are
discussed in more detail in Chapter \ref{chapF:Directions-for-Visualization}.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterB/OODAbookChpBFigTCMYK}

\caption{DWD and Orthogonal PC view of the Overlapping Classes data from Figure
\ref{fig:HiDimToyPCA }. Now shows strong inter-class difference.
\label{fig:HiDimToyDWD}}
\end{figure}
While\index{scores!scatterplot|)} the DWD separation looks very seductive,
it is important to keep in mind that DWD is very efficient at finding
directions which separate groups of data in high dimensions. But there
is another side to this, which is that DWD can be in some sense too
good. That is an issue in the Overlapping Classes\index{Overlapping Classes data|)}
example, because both the red and blue classes were simulated from
the $d=1000$ dimensional standard normal distribution, $N_{d}\left(\boldsymbol{0}_{d,1},\boldsymbol{I}_{d}\right)$,
where 
\begin{equation}
\boldsymbol{0}_{d,n}=\left[\begin{array}{ccc}
0 & \cdots & 0\\
\vdots & \ddots & \vdots\\
0 & \cdots & 0
\end{array}\right],\label{eqB.G:0sMatrix}
\end{equation}
denotes the $d\times n$ matrix of zeros\index{matrix of zeros},
and where 
\begin{equation}
\boldsymbol{I}_{d}=\left[\begin{array}{cccc}
1 & 0 & \cdots & 0\\
0 & 1 & \ddots & \vdots\\
\vdots & \ddots & \ddots & 0\\
0 & \cdots & 0 & 1
\end{array}\right]\label{eqB.F:IdentityMatrix}
\end{equation}
 is the $d\times d$ identity matrix. This highlights a fact which
is discussed in detail in Chapter \ref{chapM:HDasy}: high dimensional
data can frequently exhibit perhaps surprising behavior. Actually,
the specific behavior observed here, and even the amount of visual
separation apparent in the DWD direction will be directly explained
and even predicted using considerations developed in Section \ref{secM-B:HDLSS}.
Another perhaps surprising aspect of Figure \ref{fig:HiDimToyDWD}
is the large standard deviations of the PC scores, clearly far larger
than the standard normal value of 1. This will also be explicitly
studied and quantified in Section \ref{secM-A:RandomMatrix}. The
fact that this apparent difference between the red and blue classes
is spurious is confirmed in Section \ref{secL1:DiProPerm}, where
it is seen that the DiProPerm\index{DiProPerm|(} hypothesis test
(described below) for the difference of the means between these two
datasets gives a quite non-significant p-value of 0.82.

Figure \ref{fig:HiDimToyDWD} makes a very important point about data
visualization in general. While it can be very useful at finding important
population structure in data, it is also quite capable of finding
things which are just natural artifacts of the sampling variation
(which can appear in unexpected ways). For this reason it is critical
to combine any exploratory visual analysis with \emph{confirmatory
analysis}, as introduced here and studied more deeply in Chapter \ref{chapL:Confirmatory-Analysis}.

In the more complicated areas of OODA, e.g. many of those illustrated
in Chapter \ref{chap1B:Breadth-of-OODA}, confirmatory analysis is
still in a relative state of infancy (compared to other parts of statistics).
One reason for this is that in some of those areas, such as tree\index{tree|(}-structured
or \index{manifold|(}manifold data objects, it can be quite challenging
to develop appropriate null probability distributions, which underlie
much of classical statistical inference. This has motivated permutation
and bootstrap solutions, although careful investigation of their properties
remains as a wide open research area in mathematical statistics.

Some existing confirmatory analysis methods for OODA are discussed
in Chapter \ref{chapL:Confirmatory-Analysis}. Section \ref{secL1:DiProPerm}
discusses a generally useful high dimensional permutation type of
test, called \emph{DiProPerm}. The key steps are:
\begin{itemize}
\item Find a DIrection in the data space, such as the DWD\index{Distance Weighted Discrimination|)}
direction used in Figures \ref{fig:Faces-DWD}, \ref{figB-R:PanCanDWD}
and \ref{fig:HiDimToyDWD}, although any other systematic direction
can used as well.
\item PROject the data onto that direction to focus on representative univariate
components, e.g. the numbers whose distribution is shown in the top
left panel of Figure \ref{fig:HiDimToyDWD}. Then summarize the projections
with an appropriate summary statistic. A natural choice might be the
2 sample t-statistic. However a surprising result of the careful mathematical
analysis of \citet{Wei2016DiProPerm} is that the simple difference
of sample means provides a more stable hypothesis test in high dimensions.
\item PERMute the data to assess statistical significance. In particular
randomly reassign the group labels (e.g. red and blue for the data
in Figures \ref{fig:HiDimToyPCA } and \ref{fig:HiDimToyDWD}), recompute
the separating direction, the projection\index{projection|)}s and
the summary statistic, to generate one element of a simulated null
distribution. Repetition generates a simulated null population and
comparison with the original summary statistic provides statistical
inference such as p-values.
\end{itemize}
In Section \ref{secL1:DiProPerm} it is seen that while it may not
always be the most powerful mean hypothesis test, the DiProPerm test
is generally useful because it provides direct confirmation (or not)
of visually observed effects, such as the difference between the red
and blue groups in Figure \ref{fig:HiDimToyDWD}. Further examples
exploring these issues, together with real data examples highlighting
the importance of this type of confirmatory analysis appear in Section
\ref{secL1:DiProPerm}.

While the DiProPerm test provides a very useful reality check for
confirming visualized differences between previously defined groups,
care must be taken in the comparison of groups discovered say by \emph{clustering}.
The useful operation of clustering can be done in an informal visual
way, as for the Lung Cancer\index{Lung Cancer|)} RNAseq data\index{RNAseq data}
in Figure \ref{fig:LungCancerPCABrushed}. It can also be carried
out in many more mathematically motivated ways, as discussed in Chapter
\ref{chapK:Clustering}. It is seen in Section \ref{secL2:SigClust}
that application of DiProPerm in clustering contexts can be seriously
misleading. Yet the method of clustering has led to many important
discoveries in data, so it will continue to be an important tool.
In parallel to the challenge of spurious visualization illustrated
in Figure \ref{fig:HiDimToyDWD}, is the question of ``which clusters
are really there?'' as opposed to being spurious artifacts of the
sampling variation. An answer to this question, which becomes particularly
challenging in the high dimensional case is the SigClust\index{SigClust}
approach motivated in Section \ref{secL2:SigClust}. 

There is one more important aspect of confirmatory analysis in OODA.
This is that carefully working from the OODA viewpoint can yield much
more powerful and insightful analyses than are available from naive
implementation of classical methods. An example of this is the study
of osteoarthritis and its impact on knee shape carried out in \citet{AnMarronNelson2016OsteoArthritis}.
The shape data objects were represented by a set of 60 two dimensional
landmark\index{landmark}s, collected from standard x-ray images,
analyzed using Procrustes methods as discussed in Section \ref{sec:Landmark-Shapes}
and in \citet{DrydenMardia2016}. Earlier work in this area, such
as \citet{Gregory2007OsteoArth} and \citet{Nelson2014OsteoArth},
used PCA to summarize the population structure and then did 2 sample
t-tests on the resulting sets of scores\index{scores|)}.

There are 2 ways in which OODA offers improvement over this approach.
First is the concept, illustrated in Figures \ref{figB-Q:PanCanPCA}
and \ref{figB-R:PanCanDWD}, that important information in terms of
class differences may not show up strongly in any chosen low rank
PCA direction. The second is that the multiple testing requires some
type of adjustment, such as a Bonferroni\index{Bonferroni adjustment}
correction or \index{False Discovery Rate}False Discovery Rate calculation,
which entails additional loss of power. This issue was shown to be
serious in the relatively small scale ($n=65$) study of \citet{AnMarronNelson2016OsteoArthritis},
where an exploration of knee shape in the development versus non-development
of osteoarthritis in African American females resulted in a DiProPerm
p-value of 0.033, while the first 5 PC based p-values are reported
in the first row of Table \ref{tabB.C:PvalsOsteoArth}. In this case
PC3 happens to give a significant result at the 5\% level. 

However, since multiple tests are considered some adjustment is needed.
The second row shows a Bonferroni adjustment based upon the 5 components
shown, and the third component no longer gives a significant result.
Parallel conclusions follow using the more sophisticated False Discovery
Rate adjustment, \citet{BenjaminiHochberg1995FDR}, in the bottom
row. Of course the number 5 (of components) is arbitrary, but consideration
of more components will result in a less significant result. Also,
the separation in the 3rd component of these data may be something
of a fluke, as demonstrated by some of the strong cancer type separations
shown in Figure \ref{figB-R:PanCanDWD} being clearly not available
in the PCA of the same data shown in Figure \ref{figB-Q:PanCanPCA}.
The main lesson here is that thinking in an object oriented way, i.e.
developing the inference in terms of the data objects, instead of
doing a naive piecemeal inference on the components (that had been
standard in this area), results in a more powerful statistical inference.
See \citet{Nelson2017OsteoArth} for related applications.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|}
\hline 
PC no. & 1 & 2 & 3 & 4 & 5\tabularnewline
\hline 
\hline 
p-val & 0.223 & 0.505 & 0.013 & 0.350 & 0.204\tabularnewline
\hline 
Bonferroni p-val & 1.000 & 1.000 & 0.065 & 1.000 & 1.000\tabularnewline
\hline 
FDR p-val & 0.372 & 0.505 & 0.065 & 0.437 & 0.372\tabularnewline
\hline 
\end{tabular}

\caption{Naive and Bonferroni adjusted p-values for the osteoarthritis data.
Shows adjusted results are not significant in contrast to the significant
p-value of 0.033 from DiProPerm\index{DiProPerm|)}. \label{tabB.C:PvalsOsteoArth}}

\end{table}

\section{Further Major Statistical Tasks\label{secB4:Further-Statistical-Tasks}}

While data visualization\index{visualization|)}, as illustrated in
Section \ref{secB2:Data-Visualization}, and confirmatory analysis\index{confirmatory analysis|)}
as discussed in Section \ref{secB3:Confirmatory-Analysis} are important
components of OODA, there are also a number of important analytic
methods that are used as well. These include:
\begin{itemize}
\item \emph{Distance based analysis}. A number of OODA situations involve
data objects which lie in spaces where statistical analysis\emph{
}can be challenging. A straightforward general strategy is to first
find a metric\index{metric} on the space and then to compute the
matrix of pairwise distances. Chapter \ref{chapE:Distance-Based-Methods}
discusses various methods for data analysis whose input is only a
matrix of distances between data objects. Perhaps most important among
these is an analog of PCA called Multi-Dimensional Scaling\index{Multi-Dimensional Scaling},
studied in Section \ref{secE.2:Multi-dimensional-scaling}. A crucial
issue in metric based analysis is choice of metric, which is essentially
a data object representation issue, that is explored in Section \ref{secI-C:ImportantDists}
and other places. 
\item \emph{Statistics on manifolds.} Chapter \ref{chapG:Manifold-Data-Analysis}
discusses data objects lying in \emph{manifolds}, essentially smooth
curved surfaces. Relatively simple examples of data objects that are
usefully thought of as lying on a manifold include \emph{directional}
data\emph{\index{directional data}}, where angles (e.g. wind or magnetic
field directions) are the data objects, see \citet{MardiaJupp2000Directional}
and \citet{Fisher1987Spherical,Fisher1995Circular}. More complicated
manifold data objects arise in the study of shape, for example the
various type of shape data objects representation discussed in Section
\ref{sec:Shapes-as-Data}. Statistical analysis of data objects lying
on a manifold remains a controversial topic, as there are a number
of ways to approach it, with no clear consensus on issues even as
to how population centers should be computed. 
\item \emph{Tree\index{tree|)} structured\index{tree structured data}
data objects\index{data objects!tree structured}.} Even more challenging
than manifold data are data objects having a tree structure, in the
sense of mathematical \emph{graph\index{graph} theory}. This area
is studied in Chapter \ref{chapI:Tree-Structured-Data}, motivated
by a data set where each data object is a representation of the arteries
in a human brain that was introduced in Section \ref{sec:Tree-Structured-Data}.
As for manifold\emph{\index{manifold|)}} data, a number of different
analytic methods have been proposed, and it is even less clear which
approaches are most natural. A perhaps exotic, but quite successful
approach has been \emph{Topological Data Analysis\index{Topological Data Analysis}},
done in \citet{Bendich2016TDA}. 
\item \emph{Classification\index{classification}} (also sometimes called
\emph{discrimination\index{discrimination}} or \emph{pattern recognition\index{pattern recognition}}).
This is a large field and in fact has become a very important component
of the field called \emph{machine learning\index{machine learning}}.
A good overview is available in \citet{DudaHartStork2001PattRecog}
and \citet{HastiTibsFried2009ElStatLearn}. This area is reviewed
briefly in Chapter \ref{chapJ:Classification}. 
\item \emph{Clustering\index{clustering}}. This is another very large field,
with again just some discussion in Chapter \ref{chapK:Clustering}.
A classic reference in this field is \citet{Hartigan1975Clustering}.
Another good source is \citet{KaufmanRousseeuw2009Cluster}. In machine
learning clustering is often called \emph{unsupervised learning}\index{unsupervised learning},
to provide useful contrast with classification being called \emph{supervised
learning\index{supervised learning}}, since the goals are related,
although in the latter class labels are given, while in the former
they are derived from the data.
\item \emph{Statistical smoothing\index{smoothing}}. This is one more field
with a large literature and many proposed approaches, often with substantial
controversy, as reviewed in Chapter \ref{chapN:Smoothing}. It includes
\emph{density estimation\index{density estimation}}, essentially
a smoothed version of histograms, and \emph{nonparametric\index{nonparametric}
regression\index{nonparametric regression}} which is essentially
scatterplot smoothing\index{scatterplot smoothing}. While smoothing
methods are commonly used in exploratory data analysis\index{exploratory analysis},
less well known is the confirmatory method SiZer\index{SiZer}, proposed
by \citet{ChaudhuriMarron1999SiZer}.
\item \emph{Robust Methods.} Once again this is a very widely studied area
of statistics. The main idea is to develop statistical methodologies
which focus on methods with reduced sensitivity to violation of assumptions.
Much of that effort has gone towards dealing with outliers, which
can be very important in OODA, as discussed in Chapter \ref{chapO:Robust-Methods}.
Major references in this area include \citet{HuberRonchetti2009RobustStat},
\citet{HampelEtAl2011Robust} and \citet{StaudteSheather1990Robust}.
See  \citet{Clarke2018RobustnessBook} for a more recent overview
of this area.
\item \emph{Data Integration\index{Data Integration}}. This relatively
new statistical area is driven by the desire in many research areas
to make multiple types of measurements and to integrate those in a
meaningful way in statistical analyses. In OODA terms, the data objects
are typically multiple vectors, which could be merely concatenated
into a single vector, but there is often interest in understanding
how these relate to each other. This is commonly done using regression
methods, which makes sense when the goal is prediction, but not when
the goal is a non-directional understanding of the relationship. The
latter is accomplished by methods such as Canonical Correlation Analysis\index{Canonical Correlation Analysis},
\index{Partial Least Squares}Partial Least Squares and the more general
JIVE approach discussed in Sections \ref{subsecP-B-A:PLS} - \ref{subsecP-B-C:JIVE}.
\end{itemize}

