This research monograph summarizes a line of research that mapscertain classical problems of discrete mathematics and operationsresearch - such as the Hamiltonian Cycle and the Travelling SalesmanProblems - into convex domains where continuum analysis can be carriedout. Arguably, the inherent difficulty of these, now classical,problems stems precisely from the discrete nature of domains in whichthese problems are posed. The convexification of domains underpinningthese results is achieved by assigning probabilistic interpretation tokey elements of the original deterministic problems. In particular,the approaches summarized here build on a technique that embedsHamiltonian Cycle and Travelling Salesman Problems in a structuredsingularly perturbed Markov decision process. The unifying idea is tointerpret subgraphs traced out by deterministic policies (includingHamiltonian cycles, if any) as extreme points of a convex polyhedron in a space filled with randomized policies. The above innovative approach has now evolved to the point where thereare many, both theoretical and algorithmic, results that exploit thenexus between graph theoretic structures and both probabilistic andalgebraic entities of related Markov chains. The latter includemoments of first return times, limiting frequencies of visits tonodes, or the spectra of certain matrices traditionally associatedwith the analysis of Markov chains. However, these results andalgorithms are dispersed over many research papers appearing injournals catering to disparate audiences. As a result, the publishedmanuscripts are often written in a very terse manner and use disparatenotation, thereby making it difficult for new researchers to make use of the many reported advances. Hence the main purpose of this book is to present a concise and yeteasily accessible synthesis of the majority of the theoretical andalgorithmicresults obtained so far. In addition, the book discussesnumerous open questions and problems that arise from this body of workand which are yet to be fully solved. The approach casts theHamiltonian Cycle Problem in a mathematical framework that permitsanalytical concepts and techniques, not used hitherto in this context,to be brought to bear to further clarify both the underlyingdifficulty of NP-completeness of this problem and the relativeexceptionality of truly difficult instances. Finally, the material isarranged in such a manner that the introductory chapters require verylittle mathematical background and discuss instances of graphs with interesting structures that motivated a lot of the research in this topic.More difficult results are introduced later and are illustrated withnumerous examples.