Understanding High-Dimensional Spaces

High-dimensional spaces arise as a way of modelling datasets with many attributes. Such a dataset can be directly represented in a space spanned by its attributes, with each record represented as a point in the space with its position depending on its attribute values. Such spaces are not easy to work with because of their high dimensionality: our intuition about space is not reliable, and measures such as distance do not provide as clear information as we might expect. There are three main areas where complex high dimensionality and large datasets arise naturally: data collected by online retailers, preference sites, and social media sites, and customer relationship databases, where there are large but sparse records available for each individual; data derived from text and speech, where the attributes are words and so the corresponding datasets are wide, and sparse; and data collected for security, defense, law enforcement, and intelligence purposes, where the datasets arelarge and wide. Such datasets are usually understood either by finding the set of clusters they contain or by looking for the outliers, but these strategies conceal subtleties that are often ignored. In this book the author suggests new ways of thinking about high-dimensional spaces using two models: a skeleton that relates the clusters to one another; and boundaries in the empty space between clusters that provide new perspectives on outliers and on outlying regions. The book will be of value to practitioners, graduate students and researchers.