Description : Markov processes are among the most important stochastic processes for both theory and applications. This book develops the general theory of these processes, and applies this theory to various special examples. The initial chapter is devoted to the most important classical example - one dimensional Brownian motion. This, together with a chapter on continuous time Markov chains, provides the motivation for the general setup based on semigroups and generators. Chapters on stochastic calculus and probabilistic potential theory give an introduction to some of the key areas of application of Brownian motion and its relatives. A chapter on interacting particle systems treats a more recently developed class of Markov processes that have as their origin problems in physics and biology. This is a textbook for a graduate course that can follow one that covers basic probabilistic limit theorems and discrete time processes.
Description : This textbook, now in its third edition, offers a rigorous and self-contained introduction to the theory of continuous-time stochastic processes, stochastic integrals, and stochastic differential equations. Expertly balancing theory and applications, the work features concrete examples of modeling real-world problems from biology, medicine, industrial applications, finance, and insurance using stochastic methods. No previous knowledge of stochastic processes is required. Key topics include: Markov processes Stochastic differential equations Arbitrage-free markets and financial derivatives Insurance risk Population dynamics, and epidemics Agent-based models New to the Third Edition: Infinitely divisible distributions Random measures Levy processes Fractional Brownian motion Ergodic theory Karhunen-Loeve expansion Additional applications Additional exercises Smoluchowski approximation of Langevin systems An Introduction to Continuous-Time Stochastic Processes, Third Edition will be of interest to a broad audience of students, pure and applied mathematicians, and researchers and practitioners in mathematical finance, biomathematics, biotechnology, and engineering. Suitable as a textbook for graduate or undergraduate courses, as well as European Masters courses (according to the two-year-long second cycle of the “Bologna Scheme”), the work may also be used for self-study or as a reference. Prerequisites include knowledge of calculus and some analysis; exposure to probability would be helpful but not required since the necessary fundamentals of measure and integration are provided. From reviews of previous editions: "The book is ... an account of fundamental concepts as they appear in relevant modern applications and literature. ... The book addresses three main groups: first, mathematicians working in a different field; second, other scientists and professionals from a business or academic background; third, graduate or advanced undergraduate students of a quantitative subject related to stochastic theory and/or applications." -Zentralblatt MATH
Description : Continuous-time Markov decision processes (MDPs), also known as controlled Markov chains, are used for modeling decision-making problems that arise in operations research (for instance, inventory, manufacturing, and queueing systems), computer science, communications engineering, control of populations (such as fisheries and epidemics), and management science, among many other fields. This volume provides a unified, systematic, self-contained presentation of recent developments on the theory and applications of continuous-time MDPs. The MDPs in this volume include most of the cases that arise in applications, because they allow unbounded transition and reward/cost rates. Much of the material appears for the first time in book form.
Description : Using a singular perturbation approach, this is a systematic treatment of those systems that naturally arise in queuing theory, control and optimisation, and manufacturing, gathering a number of ideas which were previously scattered throughout the literature. The book presents results on asymptotic expansions of the corresponding probability distributions, functional occupation measures, exponential upper bounds, and asymptotic normality. To bridge the gap between theory and applications, a large portion of the book is devoted to various applications, thus reducing the dimensionality for problems under Markovian disturbances and providing tools for dealing with large-scale and complex real-world situations. Much of this stems from the authors'recent research, presenting results which have not appeared elsewhere. An important reference for researchers in applied mathematics, probability and stochastic processes, operations research, control theory, and optimisation.
Description : Continuous time parameter Markov chains have been useful for modeling various random phenomena occurring in queueing theory, genetics, demography, epidemiology, and competing populations. This is the first book about those aspects of the theory of continuous time Markov chains which are useful in applications to such areas. It studies continuous time Markov chains through the transition function and corresponding q-matrix, rather than sample paths. An extensive discussion of birth and death processes, including the Stieltjes moment problem, and the Karlin-McGregor method of solution of the birth and death processes and multidimensional population processes is included, and there is an extensive bibliography. Virtually all of this material is appearing in book form for the first time.
Description : This book presents an algebraic development of the theory of countable state space Markov chains with discrete- and continuous-time parameters. A Markov chain is a stochastic process characterized by the Markov prop erty that the distribution of future depends only on the current state, not on the whole history. Despite its simple form of dependency, the Markov property has enabled us to develop a rich system of concepts and theorems and to derive many results that are useful in applications. In fact, the areas that can be modeled, with varying degrees of success, by Markov chains are vast and are still expanding. The aim of this book is a discussion of the time-dependent behavior, called the transient behavior, of Markov chains. From the practical point of view, when modeling a stochastic system by a Markov chain, there are many instances in which time-limiting results such as stationary distributions have no meaning. Or, even when the stationary distribution is of some importance, it is often dangerous to use the stationary result alone without knowing the transient behavior of the Markov chain. Not many books have paid much attention to this topic, despite its obvious importance.
Description : In this rigorous account the author studies both discrete-time and continuous-time chains. A distinguishing feature is an introduction to more advanced topics such as martingales and potentials, in the established context of Markov chains. There are applications to simulation, economics, optimal control, genetics, queues and many other topics, and a careful selection of exercises and examples drawn both from theory and practice. This is an ideal text for seminars on random processes or for those that are more oriented towards applications, for advanced undergraduates or graduate students with some background in basic probability theory.
Description : This book concerns continuous-time controlled Markov chains, also known as continuous-time Markov decision processes. They form a class of stochastic control problems in which a single decision-maker wishes to optimize a given objective function. This book is also concerned with Markov games, where two decision-makers (or players) try to optimize their own objective function. Both decision-making processes appear in a large number of applications in economics, operations research, engineering, and computer science, among other areas. An extensive, self-contained, up-to-date analysis of basic optimality criteria (such as discounted and average reward), and advanced optimality criteria (e.g., bias, overtaking, sensitive discount, and Blackwell optimality) is presented. A particular emphasis is made on the application of the results herein: algorithmic and computational issues are discussed, and applications to population models and epidemic processes are shown. This book is addressed to students and researchers in the fields of stochastic control and stochastic games. Moreover, it could be of interest also to undergraduate and beginning graduate students because the reader is not supposed to have a high mathematical background: a working knowledge of calculus, linear algebra, probability, and continuous-time Markov chains should suffice to understand the contents of the book. Contents:IntroductionControlled Markov ChainsBasic Optimality CriteriaPolicy Iteration and Approximation TheoremsOvertaking, Bias, and Variance OptimalitySensitive Discount OptimalityBlackwell OptimalityConstrained Controlled Markov ChainsApplicationsZero-Sum Markov GamesBias and Overtaking Equilibria for Markov Games Readership: Graduate students and researchers in the fields of stochastic control and stochastic analysis. Keywords:Markov Decision Processes;Continuous-Time Controlled Markov Chains;Stochastic Dynamic Programming;Stochastic GamesKey Features:This book presents a reader-friendly, extensive, self-contained, and up-to-date analysis of advanced optimality criteria for continuous-time controlled Markov chains and Markov games. Most of the material herein is quite recent (it has been published in high-impact journals during the last five years) and it appears in book form for the first timeThis book introduces approximation theorems which, in particular, allow the reader to obtain numerical approximations of the solution to several control problems of practical interest. To the best of our knowledge, this is the first time that such computational issues are studied for denumerable state continuous-time controlled Markov chains. Hence, the book has an adequate balance between, on the one hand, theoretical results and, on the other hand, applications and computational issuesThe books that analyze continuous-time controlled Markov chains usually restrict themselves to the case of bounded transition and reward rates, which can be reduced to discrete-time models by using the uniformization technique. In our case, however, the transition and the reward rates might be unbounded, and so the uniformization technique cannot be used. By the way, let us mention that in models of practical interest the transition and the reward rates are, typically, unboundedReviews:“The book contains a large number of recent research results on CMCs and Markov games and puts them in perspective. It is written in a very conscious manner, contains detailed proofs of all main results, as well as extensive bibliographic remarks. The book is a very valuable piece of work for researchers on continuous-time CMCs and Markov games.”Zentralblatt MATH