
\chapter{Data Object Definition\label{chapBA:DataObjDefn}}

This chapter and the next together discuss basic aspects of OODA.
There are two main themes. The first theme is the three phases of
OODA (object definition, exploratory analysis\index{exploratory analysis}
and confirmatory analysis\index{confirmatory analysis}) that were
introduced at the beginning of Chapter 1. Object definition is discussed
here, while exploratory and confirmatory analysis are detailed in
Chapter \ref{chapBB:ExpConfAnal}. The second theme is \emph{modes
of variation}\index{modes of variation|(}, which have been informally
discussed through most of Chapters \ref{chap1A:What-is-OODA?} and
\ref{chap1B:Breadth-of-OODA}, and will be formally defined here in
Section \ref{subsecBA.1.1:FormalizingModes}. This chapter also provides
an overview of methods discussed in more detail in later chapters. 

\section{OODA Foundations\label{secBA.1:Foundations_Terminology}}

\subsection{OODA terminology \label{subsecBA-A-A:OODA-terminology}}

Any OODA starts with data object selection. This typically has two
main components, \emph{determination} of data objects, and their numerical
\emph{representation}. Determination involves choice of focus of the
analysis. One example is the Spanish Mortality data\index{Spanish Mortality data}
of Section \ref{sec:Curves-as-Data} and the choice between age indexed
curves over years and year indexed curves over age, as well as the
choice of $\log_{10}$ mortality. Another example is choosing whether
to focus on amplitude\index{amplitude!variation} and / or phase\index{phase!variation}
variation in Section \ref{sec:Amplitude-and-Phase}. It is useful
to consider the notion of \emph{object space\index{object space|(}}
as the conceptual space containing all potential data objects\index{data objects|(},
e.g. the space of curves for these two examples. Fundamental OODA
insights come from simultaneously considering the parallel notion
of \emph{\index{feature space|(}feature space}, which contains the
practical numerical representations, such as \emph{feature\index{feature|(}
vector}s, as illustrated in Section \ref{subsecBA.A.B:ObjFeatExample}.

Feature vectors are often aggregated into a \emph{\index{data matrix}data
matrix} which is a useful framework for organizing data analytic thoughts.
One of the matrix dimension\index{dimension}s typically represents
the \emph{cases}, i.e. the \emph{elements} of a statistical sample,
which are also sometimes called \emph{observations} or \emph{individuals}.
Some potentially confusing cross-cultural terminology has arisen in
bioinformatics, where a complex biological experiment is used to collect
each measurement, i.e. feature vector, which itself is frequently
called a \emph{sample} (by biologists, in stark contrast to the statistical
use of the term applying to the entire data set). The other matrix
dimension is used to index \emph{features} or numerical \index{descriptor}descriptors
of each data object, with \emph{variables} being a common synonym,
as in Table \ref{tab2-A:Synonyms}. Another synonym is \emph{trait\index{trait}s}
which is common in biological applications and is particularly useful
in Chapters \ref{chapF:Directions-for-Visualization} and \ref{chapP:PCA-Details}.

\begin{table}[H]
\begin{tabular}{|c|c|c|}
\hline 
 & Number & Synonyms\tabularnewline
\hline 
\hline 
Cases & $n$ & elements of a statistical sample, observations, \tabularnewline
 &  & individuals, biological samples, experimental units\tabularnewline
\hline 
Features & $d$ & variables, descriptors, traits\tabularnewline
\hline 
\end{tabular}

\caption{Commonly used synonyms for cases and features.\label{tab2-A:Synonyms}}
\end{table}
An important issue is that there is a distinct dichotomy in personal
preference as to which data matrix dimension is which. From the classical
linear algebraic point of view, where vectors are columns, it makes
the most sense to treat each data object as a column vector, and then
to horizontally concatenate these (i.e. bind the columns), resulting
in columns as data objects, with rows representing numerical features.
However, from the equally classical statistical tabulation viewpoint,
it is perhaps more natural to put variables (i.e. features or trait\index{trait}s)
in the columns and to hence use row vectors as the data objects.

Keeping this distinction in mind is critical to having meaningful
technical conversations, especially when linear algebra is involved.
OODA terminology makes this straightforward, by first agreeing whether
it will be rows or columns that are the data objects. This choice
is often closely connected with software preference. Much mainstream
statistical analysis is done using R and SAS, where rows as data objects
are the convention. More mathematically oriented work is often done
in Matlab where columns as data objects is the more natural choice.
Columns as data objects is typical in bioinformatics as well, although
this convention appears to be largely driven by the fact that typical
data sets tend to have many more variables (features or traits) than
cases, which were easiest to store in early versions of Excel in that
format. The convention in this book is columns as data objects.

Another point of varying conventions is the letters used to denote
the dimensions of the data matrix. Again this is context dependent,
with choices like $m$ and $n$ appearing in some areas. Statisticians
generally agree that $n$ should be used for sample size, i.e. for
the number of data objects. Quite common also is $p$ for the number
of variables (features, traits). Less clear is what $p$ might stand
for. Some say it stands for \emph{predictors}, but this seems limited
to mostly regression contexts. Others suggest \emph{parameters}, which
makes sense in contexts such as regression where the mean is the focus,
but not for consideration of covariance matrices (which typically
involve many more than $p$ parameters). The convention here is $d$
standing for \emph{dimension} of the data object vectors, and so with
columns as data objects the data matrix\index{data matrix} is a $d\times n$
matrix throughout this book. Further matrix notation appears at (\ref{eqB.E:Matrices}). 

\subsection{Object and Feature Space Example \label{subsecBA.A.B:ObjFeatExample}}

As noted in \citet{MarronAlonso2014} and mentioned above, a useful
framework for understanding relationships between\index{data objects!relationships between|(}
data objects is through the twin concepts of \emph{object} and \emph{feature}
spaces. The object space contains the raw curves, images, shapes or
tree\index{tree}s, while the feature space contains some sort of
numerical representation often in vector form. As pointed out in \citet{TelschowHuckemann2014OODAdisco},
the feature space can also be called \emph{descriptor space}\index{descriptor!space},
because that is where statistical description, such as the mean and
variation about the mean as in the following example, are naturally
computed. 

The interplay between these spaces is illustrated using the simple
\emph{\index{2-d Toy data}2-d Toy} FDA\index{Functional Data Analysis|(}
example shown in Figure \ref{fig:2dToy-Mean}. The data objects are
the $n=24$ very simple functions shown as black piece-wise lines
in the left panel of Figure \ref{fig:2dToy-Mean}, the conceptual
object space. This piece-wise linear functional form is used here
because it is \emph{two dimensional}, in the sense that each data
curve is entirely determined by heights of the two x signs plotted
on the vertical lines. Each curve has the constant values of height
$x_{1}$ on {[}0,1{]} and $x_{2}$ on {[}2,3{]}, and is piece-wise
linear between the x signs (on {[}1,2{]}). 

\begin{figure}[H]
\includegraphics[scale=0.8]{ChapterB/OODAbookChpBFigA2CMYK}

\caption{Simple 2-d Toy example illustrating object space (left panel) showing
data objects as (simple piece-wise linear) curves, and feature space
representing the same data objects as points (circles in the right
panel). The sample mean is shown in green in both panels.\label{fig:2dToy-Mean}}
\end{figure}
The relationships between these data objects is clearly illustrated
in the feature space view shown in the right panel, where each black
circle represents one piece-wise linear function, using a conventional
$\left(x_{1},x_{2}\right)$ scatterplot. Note that every point (not
just the data points) in the feature space has a representation as
a piece-wise curve and vice versa, i.e. there is a one to one correspondence
between these spaces. This view clearly shows two very distinct clusters,
which are also apparent in the left panel, at least after seeing the
right panel. In general higher dimensional cases, the feature space
is usually of at least somewhat higher dimension, and thus is more
challenging to visualize. However, it is often very useful to still
think in terms of such a point cloud in feature space as a device
for considering relationships between\index{data objects!relationships between|)}
data objects (e.g. the clusters apparent here). This concept was used
to explain the PCA modes of variation in Figures \ref{fig:Mortality-PC1}
and \ref{fig:Mortality-PC2} for the Spanish Mortality data\index{Spanish Mortality data}.
Graphical devices for visualizing such point clouds include matrices
of scatterplots\index{scores!scatterplot} as introduced in Figure
\ref{fig:Mortality-Scatterplot} and discussed further in Section
\ref{secB2:Data-Visualization}.

Figure \ref{fig:2dToy-Mean} also shows the sample mean in green.
This is computed as the conventional vector mean in the feature space,
shown as the green plus sign. The corresponding piece-wise curve is
shown in green in the object space in the left panel. Note that the
green curve is also the point-wise mean of the data curves. 

It was seen in Section \ref{sec:Curves-as-Data} that PCA can provide
an insightful decomposition into modes of variation. This is explored
in the context of this same 2-d toy example in Figure \ref{fig:2dToy-PC1}.
The first principal mode of variation is usefully understood in the
feature space as based on the direction from the mean (i.e. the red
line through the green plus sign), that maximizes projected variation.
The projection\index{projection|(} of each black data point onto
the red line is a magenta plus sign, which is the point on the line
closest to the data point, connected by a cyan line segment. The coefficients
of this projection (relative to the unit vector pointing in the direction
of the red line) are the scores\index{scores|(} (e.g. these quantities
were highlighted for the Spanish Mortality data in the right panels
of Figures \ref{fig:Mortality-PC1} and \ref{fig:Mortality-PC2}).
The entries of the direction vector are the loadings\index{loadings|(}
that determine the shape of the mode of variation shown in the left
panel of Figure \ref{fig:2dToy-Mean}. The red line has been carefully
chosen to maximize the sample variance of the scores, thus giving
the direction of maximal variation in the data. By the Pythagorean
theorem\index{Pythagorean theorem}, it is easily seen that this solution
is the same as minimizing the sum of the squared lengths of the cyan
line segments.

\begin{figure}[H]
\includegraphics[scale=0.8]{ChapterB/OODAbookChpBFigB4CMYK}

\caption{First principal component mode of variation for the 2-d Toy example.
Red line in the right panel is the direction of maximal variation,
cyan line segments show projections to magenta plus signs. Left panel
shows corresponding projected piece-wise lines in magenta. This plot
shows how PC1 is the one dimensional approximation which captures
most of the variation in the data. \label{fig:2dToy-PC1}}
\end{figure}
The left panel of Figure \ref{fig:2dToy-PC1} shows these projected
points as curves. As the magenta plus signs in the right panel are
close to the black circles, the magenta piece-wise lines in the left
panel are close to their corresponding black curves. These magenta
curves are the best one dimensional approximation of the data, in
that they are (signed) multiples of the same curve (with respect to
the mean) that has been determined by the loadings. Thus in the left
panel the Coordinate 1 ($x_{1}$) heights are negatively correlated
with the Coordinate 2 ($x_{2}$) heights. This negative correlation
is of course also reflected by the circles in the right panel. The
magenta curves are also usefully interpreted as a mode of variation,
with relatively small variation in the first coordinate ($x_{1}$)
and much larger negatively correlated variation in the second coordinate
($x_{2}$). 

Also insightful is the second mode of variation, defined in terms
of the second principal component, shown in Figure \ref{fig:2dToy-PC2}.
The yellow line in the right panel shows the second principal component
direction, which is orthogonal to the red line in Figure \ref{fig:2dToy-PC1}
(generally the orthogonal line with maximal variation, but in this
2-d example it is the only choice). Magenta line segments indicate
the operation of projection of each black data point onto the yellow
line, which results in a cyan plus sign. Note that lengths of these
magenta line segments are the same as the distances between the green
center point and the magenta plus signs in the right panel of Figure
\ref{fig:2dToy-PC1}. Furthermore the distances between the cyan plus
signs and the green center point on the right of Figure \ref{fig:2dToy-PC2}
are the lengths of the short cyan line segments on the right in Figure
\ref{fig:2dToy-PC1}. 

\begin{figure}[H]
\includegraphics[scale=0.8]{ChapterB/OODAbookChpBFigC4CMYK}

\caption{Second principal component mode of variation for the 2-d Toy example.
Yellow line in the right panel is the PC direction, with magenta line
segments showing cyan projections. Second mode of variation is shown
as cyan piece-wise lines in the left panel.\label{fig:2dToy-PC2}}
\end{figure}
The left panel of Figure \ref{fig:2dToy-PC2} shows the corresponding
mode of variation as the cyan piece-wise curves. Note that there is
far less visual variation present which is not surprising because
this is the direction of minimal variation. Also note that the $x_{1}$
heights (Coordinate 1) are positively correlated with the $x_{2}$
heights (Coordinate 2). This is consistent with the fact that the
cyan pluses in the right panel lie on an upward sloping line.

A useful summary of the decomposition into modes of variation shown
in the above plots appears in Figure \ref{fig:2dToy-Decomp}. The
raw data object piece-wise lines shown in black in the upper left
are the sum of the components shown in the other panels: the sample
mean in green in the upper right, magenta PC1 projections (first mode
of variation) in the lower left, cyan PC2 projection\index{projection|)}s
(second mode of variation) in the lower right. Note that the magenta
first mode focuses on the clustering aspect of the data, in addition
to the negative correlation of $x_{1}$ and $x_{2}$ and greater variability
in $x_{2}$ compared to $x_{1}$. The cyan second mode of variation
is much smaller in magnitude, and contains the smaller scale positive
correlation between $x_{1}$ and $x_{2}$ as well as being driven
more by $x_{1}$ than by $x_{2}$. Note that several effects have
been picked up by each mode of variation, as is typical in many applications.

\begin{figure}[H]
\includegraphics[scale=0.8]{ChapterB/OODAbookChpBFigD2CMYK}

\caption{PC decomposition of 2-d Toy example. Raw data objects in the upper
left, mean in the upper right (green), 1st mode in the lower left
(magenta), 2nd in the lower right (cyan). Raw curves are the sum of
others, showing insightful decomposition of variation.\label{fig:2dToy-Decomp}}
\end{figure}

\subsection{Scree Plots \label{subsecBA.A.C:ScreePlots}}

Another aspect of this data set that is clearly visible in Figure
\ref{fig:2dToy-Decomp} is that the first mode of variation (magenta
curves in the lower left) contains much more of the overall variation
in the data than the second mode (cyan curves, lower right). This
apportionment of variation is often worth numerically quantifying
using sums of squares and then graphically displaying the results
as a \emph{scree plot\index{scree plot|(}}, several of which are
shown in Figure \ref{figB-Y:Screeplots}. A scree plot summarizes
the variation (measured as sums of squares) of each PC mode, as a
function of the component index plotted as a solid red piece-wise
line. This is frequently (as here) scaled so the total is one, thus
displaying the proportion of variation for each mode. It is also frequently
useful to study the proportion of variation explained by the approximation
of the data made by each mode together with its predecessors. This
is the cumulative sum of the red values, shown here using a dashed
blue piece-wise line. A few of the values are also highlighted using
circles (for the red single mode points) and plus signs (for the blue
cumulatives). These represent modes that are explicitly displayed
in the respective analyses elsewhere in this book. 

The upper left panel of Figure \ref{figB-Y:Screeplots} shows the
scree plot for the 2-d Toy data set used in Figures \ref{fig:2dToy-Mean}
- \ref{fig:2dToy-Decomp}. This shows that the first PC mode of variation
has 98.3\% of the variation, which is consistent with the visual impression
in the lower panels of Figure \ref{fig:2dToy-Decomp}. Note that there
are only two components for this data set, so the dashed blue cumulative
curve is actually at height one starting at mode index 2. 

The scree plot for the Spanish Mortality data\index{Spanish Mortality data}
from Figures \ref{fig:Mortality-Raw} - \ref{fig:Mortality-Scatterplot}
is in the upper right panel of Figure \ref{figB-Y:Screeplots}. Once
again the first mode contains a very large share of the variation
(95.8\%), which is consistent with the fact that the vertical axis
for displaying the first mode of variation in the left panel of Figure
\ref{fig:Mortality-PC1} is an order of magnitude larger than for
the second mode in Figure \ref{fig:Mortality-PC2}. A careful look
(together with the overall wiggliness of the curves in Figure \ref{fig:Mortality-Colored})
suggests there may be additional variation in other modes of variation
although it is visually hard to distinguish from 0. When it is important
to understand such modes of variation that are orders of magnitude
smaller, a log transformation\index{transformation!log} of the scree
plot is useful (the same graphical principle illustrated in Figure
\ref{fig:Mortality-Raw}).

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigY2Scree} \caption{Scree plots, showing how variation is apportioned among PC modes of
variation for the 2-d Toy data from Figures \ref{fig:2dToy-Mean}
- \ref{fig:2dToy-Decomp} (upper left), the Spanish Mortality data
from Figures \ref{fig:Mortality-Raw} - \ref{fig:Mortality-Scatterplot}
(upper right), the 10-d toy data from Figures \ref{fig:10d-ToyPCA}
and \ref{fig:10d-ToyPCA-MinWhiteSpace} (lower left), and the cancer
gene expression data set from Figures \ref{figB-Q:PanCanPCA} - \ref{figB-X:PanCanDWDLoadings}.
Solid red curve shows individual proportions of variance and blue
dotted curve is the corresponding cumulative. \label{figB-Y:Screeplots}}
\end{figure}
The bottom left panel of Figure \ref{figB-Y:Screeplots} is the scree
plot for the Tilted Parabolas\index{Tilted Parabolas data} 10-d toy
data which will be studied in Figures \ref{fig:10d-ToyPCA} and \ref{fig:10d-ToyPCA-MinWhiteSpace}.
This shows a couple of large modes, followed by a number of smaller
ones. The large modes are classically thought of as ``signal'',
and the smaller ones as ``noise''. That conceptual ideal is the
basis of the use of the term ``scree'' here (apparently coined by
\citet{Cattell1966Scree}). In the natural world, as steep cliffs
erode, rocky rubble called scree tends to collect at their base. The
idea of the graphic is that the signal appears as a steep cliff, while
the noise is the scree collected next to it. It has been proposed
to try to find ``elbows'' (some say ``knees'') in the scree plot
to select the ``right'' number of principal components, but this
can be a very slippery operation. For example, the visual impression
of knees in graphics can change completely when transforming the data,
e.g. by looking at the log proportions. The lower left panel of Figure
\ref{figB-Y:Screeplots} also serves to illustrate problems with this
approach, as most would say that it suggests one signal mode of variation,
yet the analysis in Figure \ref{fig:10d-ToyPCA} shows \emph{two}
modes of variation representing clear underlying population structure. 

To make the point that not all scree plots have a large first mode,
the scree plot for the Pan Cancer gene expression data set which will
be studied in Figures \ref{figB-Q:PanCanPCA} - \ref{figB-X:PanCanDWDLoadings}
is shown in the bottom right panel. This time the first mode only
explains less than 20\% of the variation, and many components are
substantial players. Here the cumulative dashed blue curve shows that
the first 20 components together explain only around 60\% of the variation.
This is because gene expression is associated with very many diverse
biological processes in living cells, so its overall variation is
intrinsically high dimensional.

\subsection{Formalization of Modes of Variation\label{subsecBA.1.1:FormalizingModes}}

Figure \ref{fig:2dToy-Decomp} uses the \index{2-d Toy data} 2-d
Toy data to highlight the usefulness of the concept of \emph{modes
of variation} in OODA, which has been informally used in the above
discussion and also in Chapters \ref{chap1A:What-is-OODA?} and \ref{chap1B:Breadth-of-OODA}.
One candidate for a more formal definition is a direction in feature
space, such as that shown as a red line in the right panel of Figure
\ref{fig:2dToy-PC1} and a yellow line in the right panel of Figure
\ref{fig:2dToy-PC2}. These directions are the loadings vectors whose
entries determine the shape of the colored curves in the left panels
of those figures. However, the variation shown in these examples involves
more than just the direction in which it occurs. In particular, another
important aspect of that variation is the two cluster population structure,
which shows up in the PC1 scores (red plus signs) in the right panel
of Figure \ref{fig:2dToy-PC1}. That also determines the clear cluster
structure in the curves in the left panel. This suggests that a more
useful definition of mode of variation could be the union of the loadings
and the scores. However, consideration of the amplitude\index{amplitude!variation}
and phase variation through the Bimodal Phase Shift data\index{Bimodal Phase Shift data}
in Figure \ref{fig:Sim-Warps}, as well as the shape variation illustrated
in Figure \ref{fig:BladProsRect-PGA}, suggest that in general such
linear summaries of loadings and scores are not sufficiently general.
Instead the needed intuition comes more generally from consideration
of some type of one dimensional set of objects in the object space.
This motivates:
\begin{description}
\item [{Definition:}] A \emph{mode~of~variation} of a sample of \index{data objects|)}data
objects is a set of potential members of the object space that provides
a simple summary of one component of the variation. This summary is
in some sense one dimensional, e.g. reasonably representable by a
single real valued parameter.\label{Definition:Mode_of_Variation}
\end{description}
Such summarizing sets are bundles of curves representing two modes
of variation of the Spanish Mortality data\index{Spanish Mortality data}
in the left panels of Figures \ref{fig:Mortality-PC1} and \ref{fig:Mortality-PC2}.
Summarizing sets, representing modes of variation using the center
and two extreme shapes are shown for the Bladder-Prostate-Rectum data\index{Bladder-Prostate-Rectum data}
in Figure \ref{fig:BladProsRect-PGA}. Examples of nonlinear modes
of variation again displayed using bundles of curves approximating
the Bimodal Phase Shift data appear in Figure \ref{fig:Sim-Warps}.
A summarizing set illustrating a mode of variation connecting females
to males in image space appears for the Faces data\index{Faces data}
in the top part of Figure \ref{fig:Faces-DWD}. The notion of mode
of variation is revisited at several points in the following, perhaps
most notably in Chapters \ref{chapF:Directions-for-Visualization},
\ref{chapH:FDA-Curve-Registration} and \ref{chapP:PCA-Details}.

\section{Mathematical Notation\label{secBA.3:MathematicalNotation}}

Notation that is useful in the next section, as well as at many later
points includes:
\begin{itemize}
\item The set of real numbers, $\mathbb{R}$.
\item The standard $d$ dimensional Euclidean space of column vectors (denoted
as lower case bold letters), 
\begin{equation}
\mathbb{R}^{d}=\left\{ \boldsymbol{x}:\boldsymbol{x}=\left(\begin{array}{ccc}
x_{1} & \cdots & x_{d}\end{array}\right)^{t},\;x_{1},\cdots,x_{d}\in\mathbb{R}\right\} .\label{eqB.B:Vectors}
\end{equation}
.
\item Matrix transpose indicated by a superscript $t$.
\item The sphere whose surface dimension is $d$ (essentially points on
the surface of the solid unit ball in $\mathbb{R}^{d+1}$), 
\begin{equation}
S^{d}=\left\{ \boldsymbol{u}\in\mathbb{R}^{d+1}:\left\Vert \boldsymbol{u}\right\Vert _{2}=1\right\} .\label{eqB.C:Spheres}
\end{equation}
\item The $L^{p}$ norm on $\mathbb{R}^{d}$, 
\begin{equation}
\left\Vert \boldsymbol{x}\right\Vert _{p}=\left(\sum_{j=1}^{d}\left|x_{j}\right|^{p}\right)^{1/p}.\label{eqB.D:LpNorms}
\end{equation}
\item The Cartesian product, $\times$.
\item The set of $d\times n$ matrices (using bold capital letter notation),
\begin{equation}
\mathbb{R}^{d\times n}=\left\{ \boldsymbol{X}:\boldsymbol{X}=\left[\begin{array}{ccc}
x_{1,1} & \cdots & x_{1,n}\\
\vdots & \ddots & \vdots\\
x_{d,1} & \cdots & x_{d,n}
\end{array}\right],\;x_{11},\cdots,x_{dn}\in\mathbb{R}\right\} .\label{eqB.E:Matrices}
\end{equation}
\item Given a $d\times n$ \index{data matrix}data matrix $\boldsymbol{X}$,
the sample covariance matrix (using the notation $\widehat{var}_{i}$
and $\widehat{cov}_{i,i'}$ to denote the sample variance and covariance
of the $i$-th and $i'$-th rows of $\boldsymbol{X}$ formally defined
at (\ref{eqP.N:SampleVar}) and (\ref{eqP.M:SampleCov})), 
\begin{equation}
\widehat{\boldsymbol{\varSigma}}=\left[\begin{array}{cccc}
\widehat{var}{}_{1} & \widehat{cov}{}_{1,2} & \cdots & \widehat{cov}{}_{1,d}\\
\widehat{cov}{}_{2,1} & \widehat{var}{}_{2} & \ddots & \vdots\\
\vdots & \ddots & \ddots & \widehat{cov}{}_{d-1,d}\\
\widehat{cov}{}_{d,1} & \cdots & \widehat{cov}{}_{d,d-1} & \widehat{var}{}_{d}
\end{array}\right].\label{eqB.F:SampleCovariance}
\end{equation}
\end{itemize}

\section{Overview of Object and Feature Spaces\label{secBA.2:OverviewObjFeat}}

Figures \ref{fig:2dToy-Mean}-\ref{fig:2dToy-Decomp} and many others
focus on the display of curves as \index{data objects!curves}data
objects which can be thought of as visual representations of vectors.
The relevant plots are all piece-wise linear plots, where the heights
of the vertices are the entries of the vectors. Such plots have been
called \emph{parallel coordinate plot\index{parallel coordinate plot}s}
by \citet{Inselberg1985ParCoor,Inselberg2009ParCoorBook}, who advocated
them as a general multivariate analysis visualization\index{visualization!multivariate}
tool.

In FDA\index{Functional Data Analysis|)}, other representations of
curves besides digitization are also commonly used, often based on
mathematical \emph{basis} ideas. These include:
\begin{itemize}
\item \emph{Fourier\index{Fourier!basis}}. This orthogonal basis is very
useful for curves which are smooth and periodic. Insightful discussion
of Fourier methods in the context of time series analysis can be found
in \citet{Bloomfield2000} and \citet{Brillinger1981}.
\item \emph{Orthogonal Polynomials}. There are many such orthogonal bases
for curve space. Many useful facts can be found in the classical book
by \citet{Szego1975OrthoPolys}. A very useful, and easily accessible
summary of many important aspects can be found in \citet{GradshteynRyzhik2015}.
A specific example of orthogonal basis data object representation,
using a tensor product of polynomials and a Fourier basis, to analyze
the Cornea Curvature data\index{Cornea Curvature data} set is in
Section $\ref{secO.B.A:CorneaData}$.
\item \emph{B-splines}. There are many variants of these typically smooth
curves, which provide flexible and effective representations of smooth
data objects. See \citet{EilersMarx1996Bsplines}, \citet{StoneEtAl1997Splines}
and \citet{RuppertWandCarroll2003} for good overviews of statistical
aspects of this area. An important classical B-spline reference is
\citet{deBoor2001Splines}.
\item \emph{\index{Wavelet}Wavelets}. This orthonormal basis can give efficient
data object representation for curves with varying amounts of smoothness
in different locations. See the book \citet{Frazier2006WaveletBook}
for introduction to this area. Other important references include
\citet{Mallat1989WaveletTheory}, \citet{Daubechies1992Wavelets},
\citet{DonohoJohnstone1994Biometrika} and \citet{Donoho1995WaveletAsymptopia}.
As discussed in \citet{Mallat2009SparseWavelets}, a combination of
sparsity\index{sparsity} ideas and wavelets has been particularly
useful in image denoising\index{denoising}. Different types of useful
insights into wavelet curve estimation come from exact risk calculation
in \citet{MarronAdak1998ExactRiskWavelet}, and using spectral ideas
in \citet{Marron1999SpectralWavelets}.
\end{itemize}
The object - feature space concept is also useful for these curve
representations, where again the object space consists of curves,
but now the feature space is the space of basis coefficients. Data
analysis methods such as PCA still tend to work quite well when performed
on the vectors of basis coefficients in that feature space, together
with insightful visualization\index{visualization} of modes of variation
seen in the object space, in the spirit illustrated in Figures \ref{fig:2dToy-Mean}
- \ref{fig:2dToy-Decomp}. As noted above, a particularly deep example
of this type, where cornea curvature images\index{data objects!images cornea}
on a disk are represented by a carefully chosen orthonormal basis
is discussed in Section \ref{secO.B.A:CorneaData}.

Another very important aspect of data object representation is \emph{transformation\index{transformation}}.
The utility of this was illustrated in Figure \ref{fig:Mortality-Raw},
where it was seen that $\log_{10}$ mortality gave much clearer insights
than were available from the raw mortality. Data transformation is
further studied in Section \ref{secD2:Preprocess-Trans}. 

\citet{SangalliSecchi2014OODAdisco} gave an interesting discussion
of the importance of \emph{sufficiency\index{data objects!sufficiency}}
in data object choice. A related issue, very important to mathematical
statistical analysis of OODA is the choice of data object space, which
includes an appropriate metric\index{metric}. For example, in FDA,
there are many ways to measure distance between curves, e.g. there
is the whole family of $L^{p}$ norms. Much of the literature has
been dominated by the choice $p=2$ because of its close relationship
with classical least squares, and its tractability. However, when
robustness issues (discussed in Chapters \ref{chapE:Distance-Based-Methods}
and \ref{chapO:Robust-Methods}) are important $p=1$ can be very
useful. Furthermore \citet{DevroyeGyorfi1985L1DensityEst} offer good
reasons why $L^{1}$ is more natural in the case of probability densities
as data objects\index{data objects!probability distributions}. In
some cases, such as the occasional need to strongly penalize thin
spike departures, the choice $p=\infty$ can be more useful. In other
situations performance of derivatives are critical, so Sobolev type
norms are the most sensible choice. \citet{MarronTsybakov1995VisErr}
explored error criteria that correspond to human visual impression.
\citet{PanaretosZemel2019Wasserstein,ZemelPanaretos2019Wasserstein,PanaretosZemel2020invitation}
show that the Wasserstein \index{metric!Wasserstein}metric provides
those good visual properties in a natural way, using optimal transport
ideas. More discussion of these error criteria can be found in Sections
\ref{secI-C:ImportantDists} and \ref{secN-B:SmoothBasics}. However,
these OODA object space issues run deeper than just the mathematical
statistics. In particular, as seen in Chapter \ref{chapE:Distance-Based-Methods},
even simple data analytic notions such as population center can depend
critically on such choices. Piercesare Secchi nicely summarized this
set of ideas as: ``Experimental units only become data objects after
embedding in an appropriate space'', which is the role of the \index{object space|)}object
space. 

Table \ref{tab2-B:ObjectFeature} revisits the examples in Sections
\ref{sec:Curves-as-Data} - \ref{sec:Images-as-Data} and above, with
the goal of clarifying the \textquotedblleft data object\textquotedblright{}
terminology and the respective roles of the conceptual object and
feature spaces.
\begin{center}
\begin{table}[H]
\hskip -4cm\begin{scriptsize} %
\begin{tabular}{|c||c|c|c|c||c|c|}
\hline 
Example  & Choice of data objects  & Appropriate?  & Object space  & Feature space  & Methods used  & Results\tabularnewline
\hline 
Mortality & Raw Mortality vs Age curve  & $\times$ Obscures detail  & Curve space  & $\mathbb{R}_{+}^{99}$ &  & \tabularnewline
 & $\log_{10}$ Mortality vs Age curve  & $\checkmark$ Reveals detail  & Curve space  & $\mathbb{R}^{99}$  & Mean \& PCA  & Figs. \ref{fig:Mortality-Raw}-\ref{fig:Mortality-Scatterplot}\tabularnewline
\hline 
Prostate & Skeletal shapes  & $\checkmark$  & 3-d S-rep models & $\mathbb{R}^{k}\times\mathbb{R}_{+}^{l}\times(S^{2})^{m}$  & PGA  & Fig. \ref{fig:BladProsRect-PGA}\tabularnewline
\hline 
Proteomics  & Raw curves  & $\times$ Obscures detail  & Curve space  & $\mathbb{R}^{2001}$  &  & \tabularnewline
 & Aligned curves  & $\checkmark$  & Curve space  & $\mathbb{R}^{2001}$  &  & \tabularnewline
 & Warp functions  & $\checkmark$  & 1-d Diffeomorphisms  & $S^{2000}$  & Align true markers  & Fig. \ref{fig:TIC-spectra}\tabularnewline
 & Aligned \& warps  & $\checkmark$  & Curve $\times$ Diffeomorphisms  & $\mathbb{R}^{2001}\times S^{2000}$  &  & \tabularnewline
\hline 
Juggling  & Warping of acceleration  & $\checkmark$  & 1-d Diffeomorphisms  & $S^{d-1}$  & PNS  & Fig. \ref{fig:Juggling}\tabularnewline
\hline 
Speech  & Sound recording  & $\times$ Obscures detail  &  &  &  & \tabularnewline
 & Spectrogram  & $\times$ Obscures detail  &  &  &  & \tabularnewline
 & Covariance of spectrogram  & $\checkmark$  & Covariance functions & $PSD_{m}$ & Group means \& test & Fig. \ref{fig:Sound-Covs}\tabularnewline
\hline 
Faces  & Unregistered faces  & $\times$ Too blurred  & Set of all 2-d images  & $\mathbb{R}^{248\times186}$  &  & \tabularnewline
 & Registered faces  & $\checkmark$  & Set of all 2-d images  & $\mathbb{R}^{248\times186}$  & DWD  & Fig. \ref{fig:Faces-DWD}\tabularnewline
\hline 
2-d toy example & Curves & $\checkmark$  & Curves  & $\mathbb{R}^{2}$  & Mean \& PCA  & Fig. \ref{fig:2dToy-Mean}-\ref{fig:2dToy-Decomp}\tabularnewline
\hline 
\end{tabular}\caption{Specifics illustrating choices of conceptual object and feature spaces.\label{tab2-B:ObjectFeature}}

\end{scriptsize} 
\end{table}
\par\end{center}

This table emphasizes the respective roles of the object and feature
space. For example
\begin{itemize}
\item \emph{Mortality Example} (Section \ref{sec:Curves-as-Data}). Each
object curve is represented by a 99 dimensional feature vector (in
$\mathbb{R}^{99}$) for the 99 ages 0-98. As observed in Figure \ref{fig:Mortality-Raw},
the $\log_{10}$ variable choice reveals more useful data structure.
\item \emph{Proteomics Example} (Section \ref{sec:Amplitude-and-Phase}).
Here there are multiple reasonable choices of data objects, which
are curves or warps (i.e. diffeomorphism\index{diffeomorphism}s,
see Section \ref{subsecH.A.B:WarpIntro} for further discussion).
The curves are represented as 2001 dimensional feature vectors, while
the Fisher-Rao \index{warping}warping functions are usefully represented
in the feature space as points on the high dimensional sphere $S^{2000}$. 
\item \emph{Speech Example} (Section \ref{sec:Sounds-as-Data}). Choices
of conceptual data objects here include digital sound intensities
as shown in the top panel of Figure \ref{fig:Spectrogram}, spectrograms
in the bottom panel, and covariance functions of frequencies. As noted
in Section \ref{sec:Sounds-as-Data} the latter is the best choice
for revealing important aspects of human speech. Their discretization
results in the set of $m\times m$ symmetric \emph{positive semi-definite}
(also called \emph{nonnegative definite}) matrices, $PSD_{m}$, being
the natural feature space\index{feature space|)}.
\item \emph{Faces Example} (Section \ref{sec:Images-as-Data}). The conceptual
data objects are the $248\times186$ two dimensional images\index{data objects!face images}.
The feature representation came from reorganizing each image matrix
into a $\left(248\times186\right)=46,128$ dimension\index{dimension|)}al
feature vector.
\end{itemize}

\subsection{Example: Probability Distributions as Data Objects\label{subsecBA.2.1:ProbDistns}}

Another\index{data objects!probability distributions|(} data object
representation issue arises in the case of probability distributions
as data objects. Some commonly used representations of distributions
in probability theory are shown in Figure \ref{figB-U:DensCDFQuant},
for the bimodal mixture distribution $\frac{1}{3}N\left(120,1200\right)+\frac{2}{3}N\left(280,1200\right)$.
The left panel shows the \emph{probability density\index{probability density}
function} representation. This form is most useful for highlighting
important distributional aspects such as modality, as pointed out
for example by \citet{Silverman1986Density} and \citet{Scott2015MultivarDensity}
and also as is clear from this figure. The center panel shows the
\emph{cumulative distribution function} of the same distribution,
which is computed by integrating the density, and has strong utility
for the task of calculating probabilities such as p-values. The right
panel shows the corresponding \emph{quantile}\index{quantile}\emph{
function}, which is just the inverse function of the cumulative distribution
function. See \citet{Parzen2004Quantiles} for discussion of many
appealing properties of the quantile function.

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigUCMYK}

\caption{Different representations of the same bimodal normal mixture distribution.
Left panel is the probability density, center is the corresponding
cumulative, right is the quantile i.e. inverse cumulative. \label{figB-U:DensCDFQuant}}
\end{figure}
Which representation is preferable strongly depends on the context.
A perhaps not well understood point, illustrated in Figures \ref{figB-V:ProbDensObjects}
and \ref{figB-W:QuantFuncObjects}, is that for sets of distributions
where variation in the mean and/or variance are important modes, the
quantile function representation can easily give the most efficient
and insightful PCA decomposition. The upper left panel of Figure \ref{figB-V:ProbDensObjects}
shows a data set of $n=100$ curve data objects, each of which is
a Gaussian density, with a wide range of means and variances. The
data objects are colored, with more blue colors coding larger means,
and more red representing larger variances (thus lower curves). Hence
the small mean and variance densities (upper left) are black, with
the large mean and variance curves colored magenta (lower right).
The top right panel of Figure \ref{figB-V:ProbDensObjects} shows
the scree plot for the PCA of these curves, which indicates quite
poor signal compression, in the sense that the first few modes of
variation contain a fairly small fraction of the variation, i.e. the
signal power of this data set is spread widely across the PCA spectrum.
This is disappointing as there are in some sense only two modes of
variation (only differing means and variances). The reason for this
appears in the other left panels, which show the PCA loadings\index{loadings|)}
plots as introduced in the left panels of Figures \ref{fig:Mortality-PC1}
and \ref{fig:Mortality-PC2}. Unlike the analysis of the mortality
data shown in Section \ref{sec:Curves-as-Data} this decomposition
does not provide a very intuitive view of the variation in this data
set. In particular, neither of the first two modes reflect the two
intuitively obvious modes of variation (means and variances).

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigVCMYK}

\caption{Raw data (upper left panel) and PCA of a family of Gaussian densities.
Colors more blue for larger mean, and more red for larger variance.
Scree plot in the upper left panel, and the modes of variation (2
lower right panels), show very poor PCA summarization of this data
set. \label{figB-V:ProbDensObjects} }
\end{figure}
As shown in Figure \ref{figB-W:QuantFuncObjects}, the quantile\index{quantile}
representation of this set of probability distributions is far more
amenable to PCA. The upper left plot in Figure \ref{figB-W:QuantFuncObjects}
shows the quantile representations of the densities in the top left
panel of Figure \ref{figB-V:ProbDensObjects}, using the same colors.
The PCA scree plot\index{scree plot|)} of these curves, shown in
the upper right panel shows that the full data set is described completely
by only the first two PC components, which is a stark contrast to
the poor PCA signal compression seen for the density representation
of these data in Figure \ref{figB-V:ProbDensObjects}. Furthermore
the first component contains well over 95\% of the variation. 

\begin{figure}[H]
\includegraphics[scale=0.85]{ChapterB/OODAbookChpBFigWCMYK}

\caption{PCA of the quantile representations of the same set of distributions
as in Figure \ref{figB-V:ProbDensObjects}. Shows much more efficient
PCA summarization of variation in this data set, as well as more interpretable
modes of variation, using this improved choice of data objects. \label{figB-W:QuantFuncObjects}}
\end{figure}
Note that the PCA loading plots shown in the remaining left panels
of Figure \ref{figB-W:QuantFuncObjects} are quite interpretable.
PC1 is clearly a vertical shift mode of variation, which nicely captures
the shifting mean component of variation. That follows from the fact
that the vertical axis in the right (quantile) panel of Figure \ref{figB-U:DensCDFQuant}
is the same as the horizontal axes in the center (cumulative) and
left (density) panels. PC2 in Figure \ref{figB-W:QuantFuncObjects}
captures the variation in variances by reflecting differing slopes
which correspond to differing widths of the densities. The scores
distribution plots\index{scores!distribution plot} in the middle
and lower right panels of Figure \ref{figB-W:QuantFuncObjects} cast
additional insights into these modes of variation. Note that the PC1
scores show increasing blue towards the right, with black and red
on the left, and blue and magenta on the right, corresponding to the
left to right colors in the upper right panel of Figure \ref{figB-V:ProbDensObjects}.
Also, the PC2 scores\index{scores|)} show increasing red towards
the right, going from black and blue to red and magenta, similar to
the downward coloring in the density representations for increasing
variance.

Another approach to the dominant mean shift variation in this example
is the use of variation in warping\index{warping} functions to explicitly
target phase\index{phase!variation} variation, as seen in Section
\ref{sec:Amplitude-and-Phase}, and discussed in detail in Chapter
\ref{chapH:FDA-Curve-Registration}. \citet{HronMenafoglio2016Densities}
and \citet{MenafolgioSecchi2018Densities} propose other promising
approaches to the analysis of probability distributions as data objects,
which similarly result in modes of variation that are more insightful
than those captured by PCA.\index{modes of variation|)}

There are situations where explicit feature\index{feature|)} representation
of data objects can be side-stepped. An example is when only distances
between data objects are measured. There are many methods for handling
such situations, discussed in Chapter \ref{chapE:Distance-Based-Methods}.\index{data objects!probability distributions|)}
