Traffic Measurement for Big Network Data

This book presents several compact and fast methods for online traffic measurement of big network data. It describes challenges of online traffic measurement, discusses the state of the field, and provides an overview of the potential solutions to major problems. The authors introduce the problem of per-flow size measurement for big network data and present a fast and scalable counter architecture, called Counter Tree, which leverages a two-dimensional counter sharing scheme to achieve far better memory efficiency and significantly extend estimation range. Unlike traditional approaches to cardinality estimation problems that allocate a separated data structure (called estimator) for each flow, this book takes a different design path by viewing all the flows together as a whole: each flow is allocated with a virtual estimator, and these virtual estimators share a common memory space. A framework of virtual estimators is designed to apply the idea of sharing to an array of cardinality estimation solutions, achieving far better memory efficiency than the best existing work. To conclude, the authors discuss persistent spread estimation in high-speed networks. They offer a compact data structure called multi-virtual bitmap, which can estimate the cardinality of the intersection of an arbitrary number of sets. Using multi-virtual bitmaps, an implementation that can deliver high estimation accuracy under a very tight memory space is presented. The results of these experiments will surprise both professionals in the field and advanced-level students interested in the topic. By providing both an overview and the results of specific experiments, this book is useful for those new to online traffic measurement and experts on the topic.