
\chapter{OODA Preprocessing\label{chapD:Preprocessing}}

An\index{preprocessing|(} acronym going back at least to the early
days of computer programming was GIGO for ``Garbage In - Garbage
Out''. That principle certainly applies to modern data analysis,
yet seems to be all too frequently ignored. This chapter on OODA preprocessing
describes some useful ways for understanding sometimes hidden data
problems and some remedies, that scale\index{scale|(} in a reasonable
way to larger data sets, even those with many variables (i.e. trait\index{trait|(}s).
Section \ref{secD1:Preprocess-MargDists} gives examples demonstrating
the importance of a careful study of marginal distributions and how
they can be used to guide data object choice. The often useful approach
of normalization\index{normalization} (usually shifting and scaling
of variables, but with some perhaps non-obvious variations) is discussed
in Section \ref{secD4:Standardization}. Another data representation\index{data representation}
point is transformation\index{transformation} of variables which
is considered in Section \ref{secD2:Preprocess-Trans}. Finally Section
\ref{secD3:Preprocess-Register} studies registration, which is one
more data object representation issue that is relevant to image and
shape analysis, as well as to phase variation in Functional Data Analysis.

A general term that encompasses all of these issues is \emph{data
provenance\index{data provenance}}, which includes information about
the sources and processes that lead to the creation and representation
of data, \citet{Glavic2007DataProv}.

\section{Visualization of Marginal Distributions\label{secD1:Preprocess-MargDists}}

As noted in Section \ref{secB2:Data-Visualization} an important OODA
pre-processing step, which can often help to avoid unpleasant surprises
of many types is visualization\index{visualization|(} of marginal
distributions. This section recommends use of \emph{marginal distribution
plot\index{marginal distribution plot|(}s} to address the challenge
of doing this for a large number of variables (i.e. traits or feature\index{feature|(}s)
in a given data set. The key idea is to select a \emph{representative}
subset of the variables to actually look at. The idea of sorting on
a one dimensional summary statistic (e.g. the mean of each variable
as in Figures \ref{fig:Mortality-MargDistPlot} and \ref{fig:MortalityLog-MargDistPlot}),
is essentially that of Tukey's \emph{scagnostics}, see \citet{Wilkinson2005Scagnostics,Wilkinson2008Scagnostics}
for good overview and discussion. The difference is that scagnostics
use numerical summaries (e.g. correlation) to find interesting scatterplots
(two dimensional views) from a large collection, while in contrast
these marginal distribution plots similarly use summaries to understand
a large collection of one dimensional marginal distributions.

\subsection{Case Study: Spanish Mortality Data\label{subsecD.1.A:SpanishMortalityData}}

An example of a marginal distribution plot is shown in Figure \ref{fig:Mortality-MargDistPlot},
for the Spanish Mortality data\index{Spanish Mortality data|(} studied
in Section \ref{sec:Curves-as-Data}. This is based on the same data
matrix that was used in the left panel of Figure \ref{fig:Mortality-Raw}.
Recall that the columns of that matrix (the data objects studied there)
were indexed by years. The rows of that matrix are viewed as variables
(i.e. features or traits) and correspond to ages. The upper left panel
in the marginal distribution plot shows the mean mortality of ages,
sorted into increasing order. Note that the first half of these averages
all appear to be quite small, with much larger values appearing among
the second half. This is consistent with the visual impression from
the left half of Figure \ref{fig:Mortality-Raw} that around half
of the ages have mortality orders of magnitude smaller than the rest.
This set of sorted variable means is also the key to finding a \emph{representative}
set of variables (ages in this case) to actually visualize. One notion
of representative is to look at an equally spaced subset (among the
sorted mean ages), as indicated by the vertical dashed lines. The
remaining panels show the 8 marginal distributions of ages corresponding
to those 8 lines, using the same format as in the right panels of
Figures \ref{fig:Mortality-PC1} and \ref{fig:Mortality-PC2}. In
particular, the circles correspond to the years (i.e. the data objects,
colored using the same year rainbow pattern from Figure \ref{fig:Mortality-Colored})
using mortality as the horizontal coordinate, with the vertical coordinate
(and color, magenta 1908 - red 2002) indicating order in the data
set (thus the year). An important point is that this use of order
in the data set as vertical coordinates reveals far more structure
than would be available from the random ordering of the original jitter
plot\index{jitter plot}s as discussed in Section \ref{secB2:Data-Visualization}.
The black curve is a smooth histogram, i.e. kernel density estimate\index{kernel density estimate},
as discussed in Chapter \ref{chapN:Smoothing}.

Note that the first two shown ages, 11 and 19, all have very small
mortalities on the order of $10^{-3}$. The ages in the middle row,
32, 40 and 59, have medium mortalities on the order of $10^{-2}$.
On the bottom row, all mortalities are larger. An important issue
is that data sets having variables with such diverse scales can be
problematic for many forms of statistical analysis. This motivates
using one of a number of approaches to data adjustment, discussed
in detail in this chapter. 

\begin{figure}[H]
\includegraphics[scale=0.8]{ChapterB/OODAbookChpBFigECMYK}

\caption{Marginal distribution plot of the Spanish Mortality data from Section
\ref{sec:Curves-as-Data}. Variable (trait) means are the blue curve
in the upper left panel. Marginal distributions of a representative
(equally spaced) set of variables, indicated as vertical dashed lines,
appear in the remaining panels. As in Figure \ref{fig:Mortality-Colored}
colors indicate years using a rainbow from 1908 (magenta) to 2002
(red). Shows large variation in variable scaling and many variables
have strong skewness\index{skewness|(} and presence of outliers.\label{fig:Mortality-MargDistPlot}}
\end{figure}
These marginal distribution plots show additional challenges to classical
analysis methods, such as skewness appearing in most plots, and also
the presence of one or more outliers, usually the violet pandemic
year 1918 discussed above. These challenges can also be addressed
using methods discussed in Chapter \ref{chapD:Preprocessing} on preprocessing.

Given the above described variation across orders of magnitude, log
transformation\index{transformation!log|(} is a natural type of data
adjustment to consider for the present data set. Figure \ref{fig:MortalityLog-MargDistPlot}
shows the variable mean sorted marginal distribution plots for the
$\log$ transformed mortalities. While there is still natural variation
in the means, it no longer spans over several orders of magnitude.
This is the reason that the visual impression of variation in the
right panel of Figure \ref{fig:Mortality-Raw} is much more insightful
than in the left panel. An added benefit of this transformation is
that the skewed distributions above are now transformed into mostly
bimodal distributions, which is again very consistent with the fairly
rapid overall improvement in mortality, observed in the discussion
of Figure \ref{fig:Mortality-PC1}. Note that the impact of the outlying
year 1918 is also substantially diminished.

\begin{figure}[H]
\includegraphics[scale=0.8]{ChapterB/OODAbookChpBFigFCMYK}

\caption{Marginal distributions similar to those shown in Figure \ref{fig:Mortality-MargDistPlot},
but for $\log_{10}$ Spanish Mortalities. Shows strong beneficial
effects of log transformation.\label{fig:MortalityLog-MargDistPlot}}
\end{figure}
Finding a representative set of variables (ages) by sorting on the
variable means was very effective for understanding critical aspects
of this mortality data set, as shown in Figures \ref{fig:Mortality-MargDistPlot}
and \ref{fig:MortalityLog-MargDistPlot}. As demonstrated in the following
chemometric example, other summary statistics can highlight different,
and very insightful, notions of representative variables as well.
An issue is the number 8, of representative variables shown in Figures
\ref{fig:Mortality-MargDistPlot} and \ref{fig:MortalityLog-MargDistPlot}.
This was chosen purely for graphical convenience, in the present format.
In other situations $15=\left(4\times4\right)-1$ allows simultaneous
viewing of more representatives. A much larger number results in each
marginal distribution being too small for easy viewing.\index{Spanish Mortality data|)}

\subsection{Case Study: Drug Discovery Data\label{subsecD.1.B:DrugDiscoveryData}}

The next example comes from the area of drug discovery, or more precisely
Quantitative Structure Activity Relationships as discussed in \citet{CherkasovTropsha2014QSAR}.
This particular \emph{Drug Discovery} data\emph{\index{Drug Discovery data|(}}
set is from \citet{Borysov2016Binaries}. There are $n=262$ chemical
compounds, that are represented by $d=2489$ chemical features (i.e.
variables or traits). The primary goal is to distinguish \emph{inactive}
compounds shown as blue circles, from \emph{active} ones shown as
red plus signs in the coming graphics.

Visualizing the data as curves, e.g. for the Spanish mortality data
in Figures \ref{fig:Mortality-Raw} and \ref{fig:Mortality-Colored},
is not insightful as there is no natural ordering of the variables.
A PCA scatterplot matrix\index{scatterplot matrix}\index{scores!scatterplot|(},
using the same format as Figures \ref{fig:50d-ToyPCAScatMat}, \ref{fig:LungCancerPCA},
\ref{fig:LungCancerPCABrushed} and \ref{figB-Q:PanCanPCA}, is shown
in Figure \ref{figD-A:DrugPCARaw}. This view of the data is dominated
by relatively few of the data objects. Almost all of the $n=262$
data points are tightly clustered near the origin, which seems to
be where any meaningful differences between the actives and inactives
may be found. However, as indicated in Figures \ref{figB-Q:PanCanPCA}
and \ref{figB-R:PanCanDWD}, there can be a large amount of interesting
structure in data which is not apparent from merely looking at PC
scores. There are many potential causes of such behavior. One of these,
that is frequently worth checking, is the behavior of the marginal
distributions. For example highly skewed marginal distributions (such
the log normal) can frequently generate such data views. 

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigA-DrugPCARawCMYK}

\caption{PCA scatterplot of raw Drug Discovery data. This view is driven by
a few outliers and gives very poor separation of the active (red)
and inactive (blue) compounds. \label{figD-A:DrugPCARaw} }
\end{figure}
A number of marginal distribution analyses (based on various sorts
of trait summary statistics) for this Drug Discovery data will now
be presented to demonstrate the usefulness of that approach. As in
Figure \ref{fig:Mortality-MargDistPlot}, a reasonable starting point
is based on a sort of the sample means, with visualization of an equally
spaced set of distributions. The upper left panel shows the sorted
variable means as a blue curve. Note that most of the means appear
to be around 0 with a few relatively huge values on the far right.
Because PCA finds directions of maximal variation, these very few
variables are potential drivers of the unfortunate population structure
observed in Figure \ref{figD-A:DrugPCARaw}, and there may yet be
useful population structure that will emerge when those variables
are properly handled. Note also the rather small downturn in the blue
variable mean curve on the far left. 

As above, eight representative marginal distributions (that number
again chosen merely for convenience of plotting) corresponding to
the vertical dashed lines are shown in the remaining panels. These
show huge heterogeneity in the variables present in this data set.
The first few show no variation at all, i.e. all values are exactly
the same. In the first marginal distribution, they are all equal to
-999 (this perhaps surprising value is explained below). For the next
three variables, all values are 0. The center right variable is all
0's except for a single 1. The variables on the bottom row are also
wildly different from each other, with a discrete distribution on
the left, and a clearly skewed distribution, with values that are
four orders of magnitude larger, on the right. 

That value of -999 is sometimes used to code missing values\index{missing values|(}
(in fact this is the case here), perhaps with the idea that it is
so different from all the others that it would be easily noticed and
properly dealt with during an analysis. However, that idea failed
in this data set, because there are some variables that are so much
bigger in magnitude (which may have been added to this combined data
set by a different analyst). In particular, because -999 is a number,
it would be easy to make the big mistake of treating them as meaningful
data. This type of effect easily arises in Big Data contexts where
there is a lot of merging of diverse data sets in contexts for which
no individual has a complete understanding of all aspects. This marginal
distribution type of visualization is often effective in discovering
such anomalies.

Figure \ref{figD-B:DrugMargDistMean} makes it clear that a number
of variables with no variation (and thus no information about active
vs. inactive compounds) can be deleted from the data set with no loss
of information. It also indicates that careful attention should be
paid to the missing values, coded as -999, and finally that both the
relative magnitude and skewness of other variables will need careful
consideration.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigB-DrugMargDistRawCMYK}

\caption{Marginal distribution plot view of the Drug Discovery data, sorted
on sample means. Shows great diversity of variation over variables.
Many have no variation and a very few are orders of magnitude bigger.
\label{figD-B:DrugMargDistMean}}
\end{figure}
Figure \ref{figD-C:DrugMargDistSD} shows the marginal distributions,
this time sorted on the variables sample \emph{standard deviation}
(SD). This summary statistic gives a more clear focus on those variables
with no variation, revealing that they are nearly half of the $d=2489$
variables. This also sustains several important lessons from Figure
\ref{figD-B:DrugMargDistMean} such as there are a few variables that
are several orders of magnitude larger, and the distributional shapes
are very heterogeneous.

An aspect not yet discussed is that the text below each distributional
subplot is the name of the variable (i.e. chemical feature). This
can be very useful for identification of important features in data.
Note that the variable with largest SD, labeled ww, is different from
the variable SRW10, which has the largest mean (bottom right in Figure
\ref{figD-B:DrugMargDistMean}). In addition the variable nHBonds,
which was seen to have the smallest mean (with each value being -999)
in Figure \ref{figD-B:DrugMargDistMean}, does not appear in Figure
\ref{figD-C:DrugMargDistSD}. The reason is that in the latter graphics,
there are a large number of variables with SD = 0, and the large number
of ties are broken by simply using the original variable ordering.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigC-DrugMargDistSDCMYK}

\caption{Standard deviation sorted marginal distribution view of the Drug Discovery
data. Reveals many variables with no variation. \label{figD-C:DrugMargDistSD}}
\end{figure}
As seen in Figures \ref{figD-B:DrugMargDistMean} and \ref{figD-C:DrugMargDistSD},
much can be learned from marginal distribution plots using equally
spaced (with respect to the variable summary statistic) representative
distributions. However, in some situations it is very useful to focus
in on particular parts of the collection, often those with the smallest
and/or the biggest variable summary measures. For example, with the
goal of taking a more careful look into the missing values coded by
-999 in the Drug Discovery data, Figure \ref{figD-D:DrugMargDistMeanSmall}
again considers distributions sorted by variable mean, but now shows
the 8 smallest mean values. This is reflected by all eight of the
vertical dashed lines being on the far left (although very hard to
see because they are so close to the vertical axis). It reveals that
there are six variables that are all missing (i.e. all values are
-999) and at least two more with some missings. 

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigD-DrugMargDistMeanSmallCMYK}

\caption{Drug Discovery data explored using the eight smallest mean marginal
distributions. Shows six variables are all -999 and others have a
few -999s. \label{figD-D:DrugMargDistMeanSmall}}
\end{figure}
Figure \ref{figD-E:DrugMargDistMin} highlights a different set of
variables, this time by sorting on the minimum value of each marginal
distribution. In this situation, this view is less informative than
those above, again because of how ties are handled in the sorting.
For example, instead of seeing all values at -999 as in several of
the above plots, the shown variable, IC4, only takes on -999 once
(and does not appear in Figure \ref{figD-D:DrugMargDistMeanSmall}).
The next 6 variables all have a minimum of 0, so they appear in an
arbitrary order, which is not particularly useful for understanding
specifics in this data set, although it does happen to show the wide
heterogeneity present here.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigE-DrugMargDistMinCMYK}

\caption{Minimum value sorted marginal distributions for the Drug Discovery
data. Again shows wide range of variation, but less informative because
of arbitrary handling of many ties. \label{figD-E:DrugMargDistMin}}
\end{figure}
Based on the above insights, straightforward calculation verified
that there were 1315 variables with no variation, and 16 that had
at least one value of -999. Removal of these variables resulted in
a cleaned data set of $d=1164$ variables, which is further studied
in the following. In other situations, much more care may be needed
to deal with missing values. A usually important issue is whether
all variables with missing values should be deleted (which is sensible
here, since there are so few missings), or else imputed in one of
various ways. See \citet{GelmanHill2007HierArchModels} and \citet{Enders2010AppliedMissingData}
for good overview of many possible approaches to data \index{imputation}imputation.
This type of data object choice can be very challenging, and is often
most effectively done in consultation with collaborators having domain
knowledge.

A PCA view of the cleaned data is not shown here because it looks
exactly like Figure \ref{figD-A:DrugPCARaw} above. The reason is
that both views are driven by the large magnitude variables, so of
course eliminating those variables with no variation changes nothing.
Also removing the -999s has no visible impact on the PCA scores view
because the larger variables are so much larger, as revealed by a
careful look at the axis labels in Figure \ref{figD-A:DrugPCARaw}.

Figure \ref{figD-F:DrugMargVDSD} studies SD for this cleaned data
set. This shows that in addition to a few variables with extremely
large variation, there are also some with extremely small variation
and seems to indicate the presence of some binary variables (taking
on only the values 0 and 1). This very wide range of variation suggests
that some type of normalization\index{normalization}, say rescaling
each variable by its standard deviation, will give a much different
result, perhaps revealing other types of structure in the data, as
discussed in Section \ref{secD4:Standardization}. Note that the lower
right (maximal SD) variable, ww, is the same as the lower right variable
in Figure \ref{figD-C:DrugMargDistSD}. The others are all different,
because they are equally spaced in a smaller set of variables.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigF-DrugMargDistVDSDCMYK}

\caption{Drug Discovery data, after cleaning, viewed with marginal distribution
plots sorted on variable standard deviation. Shows widely differing
variation. \label{figD-F:DrugMargVDSD}}
\end{figure}
Several of the above plots, e.g. those in the lower right panels of
Figures \ref{figD-C:DrugMargDistSD}, \ref{figD-E:DrugMargDistMin}
and \ref{figD-F:DrugMargVDSD}, suggest that skewness is a serious
issue for at least some of these variables. This is explicitly studied
in Figure \ref{figD-G:DrugMargVDSkew} by sorting the variables this
time on skewness. The upper left blue curve indicates more right variable
skewness than left skewness, although both types are present. It also
once again shows a strong presence of binary variables. 

Note in addition that the variables with strongest skewness (typified
here by F05|O-Ci| in the lower right panel) take on just a single
value of 1, with all other $n-1=261$ values being 0. The large flat
spot on the upper right of the blue curve shown in the upper left
panel indicates that in fact there are around one hundred such variables.
Conventional wisdom is that those variables contain little useful
information and thus should also be eliminated from consideration.
However, the large number of them suggested this issue may be worth
a second look. This was done in \citet{Borysov2016Binaries} who showed
that in this case, there actually is useful information in these variables
and incorporating them in data analyses actually gave improved classification\index{classification!in drug discovery}.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigG-DrugMargDistVDSkewCMYK}

\caption{Marginal distribution view sorted on skewness, for the clean Drug
Discovery data. Shows many variables with strong skewness, including
binary distributions. \label{figD-G:DrugMargVDSkew}}
\end{figure}
When there is a combination of discrete and continuous variables in
a data set, as suggested in the above analyses, there are other sortings
of marginal distributions that are also quite useful. One of these
is the number of unique values for each variable, as shown in Figure
\ref{figD-H:DrugMargVDNumUniq}. This clearly highlights binary variables
(which have only two unique values) by putting them first, with the
blue curve in the upper left panel showing nearly 500 such. In addition
to a number of clearly quite discrete variables, there are variables
such as ZM1 in the middle left that appear to be continuous and yet
contain a large number of exact replicates. The upper right part of
the blue curve reveals that there are very few truly continuous variables,
for which the number of uniques is $n=262$, as for AEigp in the lower
right panel.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigH-DrugMargDistVDNumUniqCMYK}

\caption{Clean marginal distributions of Drug Discovery data, sorted on number
of uniques. Shows wide range, from binary (nearly 500 such) to completely
continuous variables. \label{figD-H:DrugMargVDNumUniq}}
\end{figure}
A different way to study discreteness versus continuity of variables
is to sort on \emph{number most frequent} as in Figure \ref{figD-I:DrugMargVDNumMFreq}.
This measure counts the number of times each value appears in the
distribution and reports the largest count. This number is 1 for continuous
variables, because all values are different. Note that the shown continuous
variable, TIE, is different from the continuous variable AEgip shown
in the lower right panel of Figure \ref{figD-H:DrugMargVDNumUniq}.
Hence there are at least two truly continuous variables (namely AEgip
and TIE), which appear in different orders in the two sorts. The question
of how many continuous variables are present could be more carefully
studied by looking at the smallest variables in this sort, or the
largest number of uniques using the ordering in Figure \ref{figD-H:DrugMargVDNumUniq}.
In this case there are only 9 variables which are truly continuous
in the sense of having $nuniq=262$. In this view, the binary variables
appear on the right, in the order of how many times the majority values
appear. This is another way of seeing that hundreds of variables have
just a very few nonzero values. Because relatively few statistical
methods (with the important exception of \emph{zero inflated models},
going back at least to \citet{Lambert1992ZeroInflatedPoisson}) are
designed to handle such a mix of continuous and discrete variables
(i.e. traits), the need to use methods such as these visualization\index{visualization|)}s
is becoming increasingly important in Big Data contexts.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigI-DrugMargDistVDNumMFreqCMYK}

\caption{Drug Discovery data, after cleaning. MargDistPlot sorted on number
most frequent. Another way of contrasting discrete and continuous
variables, again showing a wide range of both types. \label{figD-I:DrugMargVDNumMFreq}}
\end{figure}
Given the large number of binary variables, a question arises as to
how much information for separating active versus inactive compounds
is present in those variables only. To study this, the data are reduced
to only the $d=364$ binary variables, with the resulting PCA scatterplot
shown in Figure \ref{figD-J:DrugVDBinaryPCA}. This shows some perhaps
surprisingly rich structure in this data with most of the active cases
focused in just a few regions and larger regions with essentially
no active cases. As noted above, \citet{Borysov2016Binaries} gives
a more detailed analysis of this binary data\index{binary data} set.
Improvements in statistical power available from the various versions
of this Drug Discovery data considered above (and some others below),
for distinguishing the actives from inactives, are given using the
DiProPerm hypothsis test in Table \ref{tabL.A:DrugDiProPermZscores}.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigJ-DrugVDBinaryPCACMYK}

\caption{Drug Discovery data, after cleaning. PCA based on binary variables
only. Shows these variables contain red-blue separation information.
\label{figD-J:DrugVDBinaryPCA}}
\end{figure}
In the PCA in Figure \ref{figD-J:DrugVDBinaryPCA} the binary data
(0-1) are treated as real numbers. An alternate analysis, that stays
within the binary domain by using entirely Boolean operations\index{Boolean operations}
is Binary Matrix Factorization\emph{\index{Binary Matrix Factorization}},
proposed by \citet{Zhang2007bBMF}, and further discussed in Section
\ref{sec F.D:AlternativeDirns}.

This case study has shown that \index{marginal distribution plot|)}marginal
distribution plots can discover many important aspects of data sets.
In many situations, these can be essential in indicating strategies
such as variable deletion, scaling (as studied in Section \ref{secD4:Standardization})
and/or transformation\index{transformation|(} (see Section \ref{secD2:Preprocess-Trans})
that can be very important to arriving at an effective data analysis.
A few commonly used summary statistics have been demonstrated here,
but many others can be equally revealing in other situations. For
example small values of kurtosis can easily find variables with strong
bimodal structure. Data sets with endemic outliers can adversely affect
conventional moment based summaries, in which case robust\index{robust methods}
summaries, such as the robust skewness-like measure of \citet{Bowley1920book}
and the robust kurtosis-like measure of \citet{Ruppert1987Kurtosis},
can be very useful. \citet{An2017PhDDissertation} developed some
L-statistics based methods that are quite effective at finding important
genes in the context of cancer research.

\section{Standardization - Appropriate Linear Scaling\label{secD4:Standardization}}

The point from Section \ref{secB2:Data-Visualization}, that scaling
and normalization\index{normalization|(} issues can be very important
in OODA, is explicitly illustrated in Figures \ref{fig:ToyPCAnScaling}
and \ref{fig: ToyPCAnScalingCorr} in this section. As noted there,
the need for scaling is obvious when different variables (i.e. trait\index{trait|)}s
or feature\index{feature|)}s) are not \emph{commensurate\index{commensurate}},
e.g. when they use different units. But it is also critical when variables
have scalings that are different by orders of magnitude (as seen for
the Drug Discovery data in Figure \ref{figD-C:DrugMargDistSD}). Since
PCA seeks directions of maximal variation, this issue can have a very
strong impact. As noted in many classical texts, such as \citet{MardiaKentBibby1979MultivarAnal},
\citet{Muirhead1982Multivar}, \citet{Jolliffe2002} and \citet{Anderson2003Multivar},
this can be handled by \emph{pre-whitening\index{pre-whitening|(}},
i.e. standardizing each variable by subtracting the mean and dividing
by the standard deviation. That operation followed by PCA is equivalent
to replacing the usual sample covariance matrix input with the sample
correlation matrix in PCA\index{Principal Component Analysis|(}.

\subsection{Example: Two Scale Curve Data \label{subsecD4A:ExampleTwoScaleCurves}}

A toy example designed to explore this issue is the \emph{Two Scale
Curves} data\index{Two Scale Curves data|(} set shown in Figure \ref{fig:ToyPCAnScaling},
whose format is quite similar to Figure \ref{fig:10d-ToyPCA}. The
$n=200$ raw data curves in $d=100$ dimensions appear in the upper
left panel, using a rainbow color scheme. Note that the first 20 variables
(as indexed on the horizontal axis) exhibit a much higher amount of
variation than the remaining 80. The mean in the top center panel
is essentially 0, so the mean residuals in the top right are nearly
the same as the raw data. The first PC mode of variation\index{modes of variation|(}
in the second row is clearly driven by the first 20 variables, and
is a mode reflecting all 20 variables moving up or down together.
Similarly the second PC shown in the third row, has a mode of variation
which is a contrast between the first and second 10 variables, which
is orthogonal to the PC1 mode, and of course reflects less total variation.
The last row shows some remaining variation of much smaller scale.

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigO1CMYK}

\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigO2CMYK}

\caption{Two Scale Curves toy FDA example, illustrating challenge of differing
variable scaling. The first 20 variables have much more variation
and thus drive the first two principal components. \label{fig:ToyPCAnScaling}}
\end{figure}
The same Two Scale Curves data set as in Figure \ref{fig:ToyPCAnScaling}
is re-analyzed in Figure \ref{fig: ToyPCAnScalingCorr}, shown in
the same format. This time the data are pre-whitened by standardizing
each variable (mean subtracted, divided by standard deviation). The
similar variation of each variable is immediately clear in the input
data plot in the upper left. Unlike Figure \ref{fig:ToyPCAnScaling}
the last 80 variables are now as prominent. Very different are the
discovered modes of variation, as now the first two modes focus mostly
on structure in the second 80 variables, while the variation that
dominated the analysis in Figure \ref{fig:ToyPCAnScaling} now shows
up only in PC3 and its residuals (which would thus show up in PC4
had that been plotted). The reason that variables 21-100 now drive
the analysis is that the per-variable magnitude of the signal is now
comparable with variables 1-20 and there are simply more of them,
giving more overall variation (and they are simulated to be independent,
thus the variation goes in essentially orthogonal directions) so these
are now the dominant modes in the PCA.

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigP1CMYK}

\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigP2CMYK}

\caption{Two Scale Curves data set from Figure \ref{fig:ToyPCAnScaling}, with
pre-whitening based on correlations. Now the last 80 variables drive
the variation, i.e. appear in the first two components, leading to
much different conclusions. This shows data scaling and normalization
have a critical impact on this type of analysis. \label{fig: ToyPCAnScalingCorr}}
\end{figure}
Quantification of these ideas is given in Table \ref{tab:ToyPCAnScalingTableSS},
which contains the percentages of sums of squares of each mode of
variation (as illustrated in Figure \ref{figB-Y:Screeplots}), with
respect to the residuals about the mean. These are the numbers that
are graphically displayed in the scree plot\index{scree plot}s of
Figure \ref{figB-Y:Screeplots}. Because the raw data curves in the
upper left of Figure \ref{fig:ToyPCAnScaling} have almost all their
variability on the left side of the range, it is not surprising that
part of the range drives two very large PC components explaining almost
all the variation in the data (with the first having about 3 times
the energy of the second), as seen in the top row. The bottom row
shows a more even spread of variation, which is consistent with the
visual impression of the standardized data in the top left of Figure
\ref{fig: ToyPCAnScalingCorr}. In particular, the table together
with the figure show that it is variation on the right part of the
range which has become dominant, containing about 80\% of the variation.
The remaining 20\% is split at the same 3:1 ratio between the third
and fourth modes of variation.

\begin{table}[H]
\begin{tabular}{|c|c|c|c|c|}
\hline 
 & PC 1 & PC 2 & PC 3 & PC 4\tabularnewline
\hline 
\hline 
Raw PCA & 76\% & 24\% & 0.1\% & 0.03\%\tabularnewline
\hline 
Standardized PCA & 53\% & 27\% & 15\% & 5\%\tabularnewline
\hline 
\end{tabular}

\caption{Percent sum of squares explained by each PC component for the Two
Scale Curves data in Figures \ref{fig:ToyPCAnScaling} and \ref{fig: ToyPCAnScalingCorr}.
Quantifies how raw data components focus on structure on the left,
while standardization shifts the focus to the right.\label{tab:ToyPCAnScalingTableSS}}
\end{table}

\subsection{Overview of Standardization\label{subsecD4B:OverviewStandardization}}

It is worth considering which of the two very divergent analyses in
Figures \ref{fig:ToyPCAnScaling} and \ref{fig: ToyPCAnScalingCorr}
is more appropriate. As noted above, most classical texts on multivariate
analysis will recommend doing the analysis based on the correlation
matrix (i.e. pre-whitening as in Figure \ref{fig: ToyPCAnScalingCorr}).
This is often a sensible default, especially in situations where different
variables are measured in different units. However it is important
to realize that in other situations the original data scaling may
be most appropriate and thus should be preserved. For example, in
the Lung Cancer data\index{Lung Cancer data} in Figure \ref{fig:LungCancerLogCurvesBrushed}
pre-whitening by standardization\index{standardization} will result
in the small exon starting at exonic nt number 500 playing too large
a role in the analysis. Clearly this is an important data object choice,
deserving careful consideration (and discussion with knowledgeable
collaborators) in data analysis. More discussion of the tendency of
PCA to focus on large scale variation can be found in Chapter \ref{chapP:PCA-Details}.

The effect of pre-whitening\index{pre-whitening|)} on the cleaned
Drug Discovery data from Section \ref{secD1:Preprocess-MargDists}
is demonstrated in Figure \ref{figD-K:DrugVDnonBinaryPCA}. A quantitative
measure of the impact is also given in Table \ref{tabL.A:DrugDiProPermZscores}.
Unlike the outlier driven PCA shown in Figure \ref{figD-A:DrugPCARaw},
this view shows much more in the way of interesting \index{data objects!relationships between}relationships
between the data objects, i.e. the chemical compounds. In particular,
it is clear that there are now very complex (e.g. highly nonlinear)
relationships between the active (red pluses) and inactive (blue circles)
compounds, which is why Drug Discovery has been a challenging problem
over the years. Note for example an indication of small regions where
interesting comparisons can be made, which motivates the idea of \emph{activity
cliffs}, (regions of abrupt transition between classes) as studied
in \citet{Maggiora2006ActivityCliffs}. 

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigK-DrugVDnonBinaryPCACMYK}

\caption{Drug Discovery data, after cleaning. PCA on non-binary variables only.
Shows these also contain red-blue separation information. \label{figD-K:DrugVDnonBinaryPCA}}
\end{figure}
There are many other aspects of data normalization that should be
kept in mind as needed. For example, while standardization of the
variables (i.e. of each row of the data matrix\index{data matrix})
can be very sensible in some cases as illustrated using the Two Scale
Curves data\index{Two Scale Curves data|)} in Figures \ref{fig:ToyPCAnScaling}
and \ref{fig: ToyPCAnScalingCorr}, column standardization can be
more useful in others. A canonical example of this is genetic molecular
measurements, which are based on amplification of DNA or RNA in a
way that is not easy to calibrate, resulting in columns of the data
matrix which tend to differ by scale factors. For gene expression
studies, e.g. \citet{Hoadley2014TCGA-PanCan}, this is commonly handled
by scaling each column appropriately. While averages could be used
for this, that would not allow for expected differing overall expression
across cases, so instead normalization\index{normalization|)} to
achieve a common third quartile is often used. That choice is driven
by the idea that the most actively different genes should not be involved,
and the third quartile is sufficiently stable to properly represent
overall variation. 

In other situations projection\index{projection|(} of each data object
(column vector) to the unit sphere, $S^{d}$ (recall the notation
(\ref{eqB.C:Spheres})) can be appropriate. This projection is computed
as $\frac{\boldsymbol{x}}{\left\Vert \boldsymbol{x}\right\Vert _{2}}$,
using the notation (\ref{eqB.D:LpNorms}). It is useful in situations
where vector length contains only nuisance variation\index{nuisance variation},
and the interesting variation is in the angles between vectors. See
the discussion around Figure \ref{figK.D:HierCos} for more in the
context of hierarchical clustering\index{hierarchical clustering}.
In such cases, another option is projection to the unit simplex, as
discussed in Section \ref{secC2:Compositional-Data-Analysis}. As
noted in Section \ref{secC2:Compositional-Data-Analysis}, \citet{XiongDittmerMarron2015}
found that (for virus detection using DNA data) projection to the
simplex gave better results than projecting to the sphere.

\section{Transformation - Appropriate Nonlinear Scaling\label{secD2:Preprocess-Trans} }

As seen above relative magnitude of variables is an important consideration
in OODA. Similarly the distributional shape of the marginal distributions
can also have a major effect as seen using the Spanish Mortality data\index{Spanish Mortality data}
in Figures \ref{fig:Mortality-MargDistPlot} and \ref{fig:MortalityLog-MargDistPlot}.
Another example that illustrates this point is shown in Figure \ref{figD-L:TCGA-Log2vsRaw}.
This is a different subset of the Pan Can data analyzed in Section
\ref{subsecB.2.B:PanCancerExample}. Here the focus is on the cancer
types Ovarian Cancer, which is labeled as OV in TCGA notation and
shown as magenta circles, and Uterine Cancer, labeled UCEC and indicated
using green plus signs. Only the 1000 most variable genes, among genes
having no missing values\index{missing values|)}, are considered.
The full data set (with a few hundred cases of each type) shows a
strongly statistically significant difference between these two cancer
types for almost any type of analysis, so for good contrast between
statistical methods, randomly chosen subsets of size $n_{1}=n_{2}=30$
cancer patients of each type are analyzed here for illustration.

The top row of Figure \ref{figD-L:TCGA-Log2vsRaw} shows PCA scatterplot\index{scores!scatterplot|)}
views of the data. Unlike the scatterplot matrices shown above, e.g.
in Figures \ref{figD-A:DrugPCARaw}, \ref{figD-J:DrugVDBinaryPCA}
and \ref{figD-K:DrugVDnonBinaryPCA}, here each plot shows only the
PC2 vs. PC1 scores scatterplot (often the left plot in the second
row in matrix views). The upper left panel of Figure \ref{figD-L:TCGA-Log2vsRaw}
studies the distribution of raw counts. Note that PC1 is dominated
by a single very large case (about an order of magnitude bigger than
all others). PC2 is driven by a handful of other cases, but still
only a relatively few. While one might hope to see a large difference
between the OV and UCEC cases, if it can be seen in this scatterplot,
it can only be in the lower left part of the plot, but is very hard
to perceive due to over-plotting. For a closer view of potential class
differences, the top center panel shows a zoomed in (on the lower
left corner) version of that plot. This makes it even more clear that
this data set suffers from strong skewness (which can also be easily
seen using the Marginal Distribution views described in Section \ref{secD1:Preprocess-MargDists}),
with essentially no OV-UCEC difference visible. This does not mean
that there is no difference, only that it does not appear in the 2
dimensional subspace of the first 2 principal components. 

For such strongly skewed data, a log transformation of each variable
is often very useful, as it tends to strongly reduce the influence\index{influence}
of data points that are orders of magnitude larger than the others.
The top right panel shows the result of the $\log_{2}$ transformation
applied to each variable. That transformation is usual in this field,
where the doubling interpretation of that log base is commonly desired.
Note that these two modes of variation highlight a clear and strong
difference between the cancer types, appearing as mostly the dominant
mode of variation (i.e. the PC 1 Scores).

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigL-TCGAlog2vsRawCMYK}

\caption{Contrast of Ovarian (magenta circles) and Uterine (green plus signs)
Cancer gene expression, for the Pan Can data. Top row shows PC1 vs.
2 scores, for raw count data (left, with zoomed in version, center)
and $\log_{2}$ transformed data on the right. Middle row shows corresponding
MD projected distributions. Bottom row is ROC curve analyses. Shows
log transformed analysis gives much better contrast between cancer
types in all ways. \label{figD-L:TCGA-Log2vsRaw}}
\end{figure}
Table \ref{tab:=000025SSforLogvsNotCancerData} provides another way
of seeing that the $\log_{2}$ transformation provides a much better
scale on which to analyze this data set. In particular, the outlier
in the upper left panel of Figure \ref{figD-L:TCGA-Log2vsRaw} is
seen to dominate the raw data analysis, with the first PC containing
88\% of the total variation about the mean. In contrast, on the $\log_{2}$
scale the first PC explains 25\% while the second explains 10\%, which
are much more reasonable as there are a large number of diverse biological
processes whose presence should be reasonably represented in gene
expression measurements.

\begin{table}[H]
\begin{tabular}{|c|c|c|}
\hline 
 & PC1 & PC2\tabularnewline
\hline 
\hline 
Raw Data & 88.2 & 4.7\tabularnewline
\hline 
$\log_{2}$ Data & 24.8 & 9.9\tabularnewline
\hline 
\end{tabular}

\caption{Percent of sums of squares about the mean explained by the first two
principal components, for the raw gene expression (top) and the $\log_{2}$
transformed version of the same data. Shows that log transformation
gives more sensible distribution of variation in the data. \label{tab:=000025SSforLogvsNotCancerData}}
\end{table}
The middle row of Figure \ref{figD-L:TCGA-Log2vsRaw} more directly
targets the OV vs. UCEC contrast by showing the univariate distribution
of projections onto the \emph{\index{Mean Difference}Mean Difference}
(MD) direction vector, which is just the normalized (to have norm
1) difference between the sample mean vectors. This direction vector
is also called the \emph{centroid classifier}, see \citet{Tibshirani2002CentroidClass}.
See Section \ref{secJ-A:ClassicalMethods} for much more discussion
of the MD direction. The display format is the same as used several
times above, e.g. the marginal distribution plot\index{marginal distribution plot}s
in Section \ref{secD1:Preprocess-MargDists}, using symbols whose
x-coordinate reflects the value with y-coordinates simply providing
visual separation. Again, the black curve is a kernel density estimate\index{kernel density estimate},
with the colored curves representing proportional sub-densities for
each of the two data types. The middle left panel shows this distribution
for the raw counts data. As in the top left panel, the difference
is not easy to discern because the view is again dominated by a single
large outlier, which obscures how well separated the two groups are.
The zoomed view in the center panel shows that actually the MD direction
provides decent separation of the classes, with the green plus UCEC
cases tending to lie more to the left of the OV magenta circles, although
there is substantial overlap. This overlap is quantified using the
Receiver Operating Characteristic\index{Receiver Operating Characteristic curve}
(ROC) curve of \citet{Hanley1982ROCcurve}, in the lower left panel.
This curve is generated by sliding a cutoff point along the horizontal
axis of the left middle panel, and for each such point displaying
the proportion of UCEC (green) points that are smaller on the vertical
axis versus the proportion of smaller OV (magenta) points on the horizontal
axis. The fact that more UCEC points lie to the left is reflected
by the curve moving fairly steeply upwards. Once the cutoff point
includes all UCEC points the curve remains at height one. The fact
that this curve lies mostly towards the upper right of this plot shows
that the two populations are relatively well separated. A simple numerical
summary of ROC behavior is the \emph{Area Under the Curve\index{Area Under the Curve}}
(AUC), which in this case is 0.86.

The middle right panel studies the MD projections for the $\log_{2}$
transformed version of the data. Given the obvious group separation
in the PC 1-2 scatterplot in the upper right panel, it is not surprising
that there is a very strong separation between UCEC and OV that is
apparent in this projection\index{projection|)}. The strong visual
impression is confirmed in the lower right panel by the ROC plot in
the lower right following first the vertical axis, then the top horizontal
line, resulting in an AUC of 1.

A related contrast between the raw count data and the $\log_{2}$
transformed version is provided using the DiProPerm\index{DiProPerm}
confirmatory\index{confirmatory analysis} method, described in Section
\ref{secL1:DiProPerm}. That hypothesis test for exploring differences
between the UCEC and OV cases, using the mean difference direction
and mean difference summary statistics, for the raw counts gave a
non-significant p-value of $0.24$, while the $\log_{2}$ counts gave
a strongly significant p-value $\ll10^{-4}$. This is another way
of seeing that analyzing this data on the $\log_{2}$ scale\index{scale|)}
is very well worthwhile.

An important variation of the log transformation is the \emph{shifted
log transformation}\index{shifted log transformation}, of the form
$\log\left(\cdot-c\right)$, where the data are shifted by a constant
amount $c$ before application of the logarithm. This is useful both
for data which take on 0 or negative values, and in the case of $c<0$
is also useful as a typically less stringent version of the $\log$
\index{transformation!log|)}transformation, that is useful for data
with relatively mild skewness. This works in the same way as the skewness
of the log normal distribution is controlled by the mean parameter
of the underlying normal distribution. Good automatic choice of the
shifted log transformation has been developed by \citet{Feng2016AutotLogTrans}.
The usefulness of that method in the case of the Drug Discovery data\index{Drug Discovery data|)}
is explored in Table \ref{tabL.A:DrugDiProPermZscores}.

Another appealing and widely used family of transformations is the
\index{Box-Cox transformation} \index{transformation!Box-Cox}Box-Cox
family 
\[
f\left(x\right)=\begin{cases}
\begin{array}[t]{c}
\frac{x^{\lambda}-1}{\lambda}\\
\log x
\end{array} & \begin{array}[t]{c}
\mathrm{for}\;\lambda\neq0\\
\mathrm{for}\;\lambda=0,
\end{array}\end{cases}
\]
proposed by \citet{BoxCox1964Transfo}. This is essentially a family
of power \index{transformation!power}transformations coupled with
a linear transformation. Note that a careful calculation of the limit
as $\lambda\rightarrow0$ shows that this is a continuous function
of the tuning parameter $\lambda$. Hence this provides a broad and
flexible way of adapting to skewness\index{skewness|)} in data. One
more important general family of transformations is described in \citet{Johnson1949FamilyofCurves}. 

\section{Registration - Appropriate Alignment\label{secD3:Preprocess-Register}}

As noted in Section \ref{sec:Amplitude-and-Phase}, registration,
i.e. alignment issues, are often quite important in many types of
OODA. This point is illustrated using an FDA (curves as \index{data objects!curves}
data objects) toy example in Figure \ref{figD-M:ToyShiftsnFRReg},
which is similar to the Bimodal Phase Shift data\index{Bimodal Phase Shift data}
of Figure \ref{fig:Sim-Warps}. The raw data are shown in the left
hand panel. Each curve has two peaks, but there is now substantial
variation in both locations and heights of the peaks. In contrast
to Figure \ref{fig:Sim-Warps}, this time the curves are color coded
using the height of the left peak, with a rainbow color scheme ranging
from magenta (tallest) through green and yellow to red (shortest)
with the goal of highlighting the amplitude\index{amplitude!variation}
variation in this case. The varying locations of the peaks create
challenges for standard statistical analysis (which ignores the strong
phase\index{phase!variation} variation). For example, the (point-wise)
mean curve, shown as a thick black dashed line, is not at all representative
of the population. In particular, its peaks are substantially lower
than any peak in the data set, and the left peak actually appears
as two modes. In addition, it will seen be in Chapter \ref{chapH:FDA-Curve-Registration}
that PCA (e.g. as in Figures \ref{fig:Mortality-PC1}, \ref{fig:Mortality-PC2},
\ref{fig:2dToy-Decomp}, \ref{fig:10d-ToyPCA} and \ref{fig:50d-ToyPCA})
of data sets with such strong phase variation also provides very poor
low rank representations. 

The right panel of Figure \ref{figD-M:ToyShiftsnFRReg} shows the
results of a Fisher-Rao registration of these curves as described
in Section \ref{secH.A:Fisher-Rao-Reg}. The heights of each curve
are the same in both panels, but in the right panel the horizontal
axis for each curve has been appropriately warped (i.e. the horizontal
axis has been appropriately stretched and compressed separately for
each curve) to make the curves align very well. Note that the mean
of this set of curves, again shown as a thick black dashed curve is
now a quite sensible notion of center, as it lies clearly in the middle
of the data set, as well as typifying the general shape of each. As
seen in Chapter \ref{chapH:FDA-Curve-Registration}, PCA\index{Principal Component Analysis|)}
of such aligned sets of curves similarly provides a quite intuitive
and much more compact representation (in particular, requiring only
a single mode of variation\index{modes of variation|)}) of data sets
with strong phase variation.

\begin{figure}[H]
\includegraphics[scale=0.7]{ChapterD/OODAbookChpDFigM-ToyShiftsnFRRegCMYK}

\caption{Toy example, of functional data, each curve having two peaks, illustrating
usefulness of curve registration\index{curve registration}. Left
panel shows original curves, using rainbow color scheme on height
of left hand peak to highlight amplitude variation. Right panel shows
the result of a careful curve alignment. Dashed curve in each case
is the sample mean, which is much more representative of the data
curves after alignment.\label{figD-M:ToyShiftsnFRReg}}
\end{figure}
For quite similar reasons, registration is also critical in image
related tasks. For example, while collecting the Faces data\index{Faces data}
objects\index{data objects!face images} in Section \ref{sec:Images-as-Data},
little attempt was made during the initial photographs to ensure that
each face was in the same place in each picture, which presented a
challenge that is quite similar in spirit to what is seen in the left
panel of Figure \ref{figD-M:ToyShiftsnFRReg}. In particular, various
facial aspects (such as eyes, nose, mouth) were seriously misaligned
across images, resulting in a far fuzzier analog of Figure \ref{fig:Faces-DWD}
when the analysis was based on the unaligned data. As noted in Section
\ref{sec:Images-as-Data}, great improvement was made by a simple
affine transformation\index{transformation|)} based on landmarks
at the center of each eye and the mouth.

Generally in image analysis this type of consideration motivates the
study of shapes as \index{data objects!shapes}data objects. The mathematics
that allow convenient and rigorous definition and analysis of such
data is discussed in more detail in Chapter \ref{chapG:Manifold-Data-Analysis}.
Essentially that set of ideas is used to provide a fully automatic
approach to curve registration in Chapter \ref{chapH:FDA-Curve-Registration}.\index{preprocessing|)}
