Description : The general theory of stochastic processes and the more specialized theory of Markov processes evolved enormously in the second half of the last century. In parallel, the theory of controlled Markov chains (or Markov decision processes) was being pioneered by control engineers and operations researchers. Researchers in Markov processes and controlled Markov chains have been, for a long time, aware of the synergies between these two subject areas. However, this may be the first volume dedicated to highlighting these synergies and, almost certainly, it is the first volume that emphasizes the contributions of the vibrant and growing Chinese school of probability. The chapters that appear in this book reflect both the maturity and the vitality of modern day Markov processes and controlled Markov chains. They also will provide an opportunity to trace the connections that have emerged between the work done by members of the Chinese school of probability and the work done by the European, US, Central and South American and Asian scholars.
Description : Measure-valued branching processes arise as high density limits of branching particle systems. The Dawson-Watanabe superprocess is a special class of those. The author constructs superprocesses with Borel right underlying motions and general branching mechanisms and shows the existence of their Borel right realizations. He then uses transformations to derive the existence and regularity of several different forms of the superprocesses. This treatment simplifies the constructions and gives useful perspectives. Martingale problems of superprocesses are discussed under Feller type assumptions. The most important feature of the book is the systematic treatment of immigration superprocesses and generalized Ornstein--Uhlenbeck processes based on skew convolution semigroups. The volume addresses researchers in measure-valued processes, branching processes, stochastic analysis, biological and genetic models, and graduate students in probability theory and stochastic processes.
Description : The modem theory of Markov processes has its origins in the studies of A. A. MARKOV (1906-1907) on sequences of experiments "connected in a chain" and in the attempts to describe mathematically the physical phenomenon known as Brownian motion (L. BACHELlER 1900, A. EIN STEIN 1905). The first correct mathematical construction of a Markov process with continuous trajectories was given by N. WIENER in 1923. (This process is often called the Wiener process.) The general theory of Markov processes was developed in the 1930's and 1940's by A. N. KOL MOGOROV, W. FELLER, W. DOEBLlN, P. LEVY, J. L. DOOB, and others. During the past ten years the theory of Markov processes has entered a new period of intensive development. The methods of the theory of semigroups of linear operators made possible further progress in the classification of Markov processes by their infinitesimal characteristics. The broad classes of Markov processes with continuous trajectories be came the main object of study. The connections between Markov pro cesses and classical analysis were further developed. It has become possible not only to apply the results and methods of analysis to the problems of probability theory, but also to investigate analytic problems using probabilistic methods. Remarkable new connections between Markov processes and potential theory were revealed. The foundations of the theory were reviewed critically: the new concept of strong Markov process acquired for the whole theory of Markov processes great importance.
Description : Markov processes are among the most important stochastic processes for both theory and applications. This book develops the general theory of these processes, and applies this theory to various special examples. The initial chapter is devoted to the most important classical example - one dimensional Brownian motion. This, together with a chapter on continuous time Markov chains, provides the motivation for the general setup based on semigroups and generators. Chapters on stochastic calculus and probabilistic potential theory give an introduction to some of the key areas of application of Brownian motion and its relatives. A chapter on interacting particle systems treats a more recently developed class of Markov processes that have as their origin problems in physics and biology. This is a textbook for a graduate course that can follow one that covers basic probabilistic limit theorems and discrete time processes.
Description : This book provides a rigorous but elementary introduction to the theory of Markov Processes on a countable state space. It should be accessible to students with a solid undergraduate background in mathematics, including students from engineering, economics, physics, and biology. Topics covered are: Doeblin's theory, general ergodic properties, and continuous time processes. Applications are dispersed throughout the book. In addition, a whole chapter is devoted to reversible processes and the use of their associated Dirichlet forms to estimate the rate of convergence to equilibrium. These results are then applied to the analysis of the Metropolis (a.k.a simulated annealing) algorithm. The corrected and enlarged 2nd edition contains a new chapter in which the author develops computational methods for Markov chains on a finite state space. Most intriguing is the section with a new technique for computing stationary measures, which is applied to derivations of Wilson's algorithm and Kirchoff's formula for spanning trees in a connected graph.
Description : The theory of Markov chains, although a special case of Markov processes, is here developed for its own sake and presented on its own merits. In general, the hypothesis of a denumerable state space, which is the defining hypothesis of what we call a "chain" here, generates more clear-cut questions and demands more precise and definitive an swers. For example, the principal limit theorem (§§ 1. 6, II. 10), still the object of research for general Markov processes, is here in its neat final form; and the strong Markov property (§ 11. 9) is here always applicable. While probability theory has advanced far enough that a degree of sophistication is needed even in the limited context of this book, it is still possible here to keep the proportion of definitions to theorems relatively low. . From the standpoint of the general theory of stochastic processes, a continuous parameter Markov chain appears to be the first essentially discontinuous process that has been studied in some detail. It is common that the sample functions of such a chain have discontinuities worse than jumps, and these baser discontinuities play a central role in the theory, of which the mystery remains to be completely unraveled. In this connection the basic concepts of separability and measurability, which are usually applied only at an early stage of the discussion to establish a certain smoothness of the sample functions, are here applied constantly as indispensable tools.
Description : Markov process theory is basically an extension of ordinary calculus to accommodate functions whos time evolutions are not entirely deterministic. It is a subject that is becoming increasingly important for many fields of science. This book develops the single-variable theory of both continuous and jump Markov processes in a way that should appeal especially to physicists and chemists at the senior and graduate level. A self-contained, prgamatic exposition of the needed elements of random variable theory Logically integrated derviations of the Chapman-Kolmogorov equation, the Kramers-Moyal equations, the Fokker-Planck equations, the Langevin equation, the master equations, and the moment equations Detailed exposition of Monte Carlo simulation methods, with plots of many numerical examples Clear treatments of first passages, first exits, and stable state fluctuations and transitions Carefully drawn applications to Brownian motion, molecular diffusion, and chemical kinetics
Description : Theory of Markov Processes provides information pertinent to the logical foundations of the theory of Markov random processes. This book discusses the properties of the trajectories of Markov processes and their infinitesimal operators. Organized into six chapters, this book begins with an overview of the necessary concepts and theorems from measure theory. This text then provides a general definition of Markov process and investigates the operations that make possible an inspection of the class of Markov processes corresponding to a given transition function. Other chapters consider the more complicated operation of generating a subprocess. This book discusses as well the construction of Markov processes with given transition functions. The final chapter deals with the conditions to be imposed on the transition function so that among the Markov processes corresponding to this function, there should be at least one. This book is a valuable resource for mathematicians, students, and research workers.
Description : This is a brief introduction to stochastic processes studying certain elementary continuous-time processes. After a description of the Poisson process and related processes with independent increments as well as a brief look at Markov processes with a finite number of jumps, the author proceeds to introduce Brownian motion and to develop stochastic integrals and Ito's theory in the context of one-dimensional diffusion processes. The book ends with a brief survey of the general theory of Markov processes. The book is based on courses given by the author at the Courant Institute and can be used as a sequel to the author's successful book Probability Theory in this series.
Description : The intent of this book is to present recent results in the control theory for the long run average continuous control problem of piecewise deterministic Markov processes (PDMPs). The book focuses mainly on the long run average cost criteria and extends to the PDMPs some well-known techniques related to discrete-time and continuous-time Markov decision processes, including the so-called ``average inequality approach'', ``vanishing discount technique'' and ``policy iteration algorithm''. We believe that what is unique about our approach is that, by using the special features of the PDMPs, we trace a parallel with the general theory for discrete-time Markov Decision Processes rather than the continuous-time case. The two main reasons for doing that is to use the powerful tools developed in the discrete-time framework and to avoid working with the infinitesimal generator associated to a PDMP, which in most cases has its domain of definition difficult to be characterized. Although the book is mainly intended to be a theoretically oriented text, it also contains some motivational examples. The book is targeted primarily for advanced students and practitioners of control theory. The book will be a valuable source for experts in the field of Markov decision processes. Moreover, the book should be suitable for certain advanced courses or seminars. As background, one needs an acquaintance with the theory of Markov decision processes and some knowledge of stochastic processes and modern analysis.
Description : Kiyosi Itô's greatest contribution to probability theory may be his introduction of stochastic differential equations to explain the Kolmogorov-Feller theory of Markov processes. Starting with the geometric ideas that guided him, this book gives an account of Itô's program. The modern theory of Markov processes was initiated by A. N. Kolmogorov. However, Kolmogorov's approach was too analytic to reveal the probabilistic foundations on which it rests. In particular, it hides the central role played by the simplest Markov processes: those with independent, identically distributed increments. To remedy this defect, Itô interpreted Kolmogorov's famous forward equation as an equation that describes the integral curve of a vector field on the space of probability measures. Thus, in order to show how Itô's thinking leads to his theory of stochastic integral equations, Stroock begins with an account of integral curves on the space of probability measures and then arrives at stochastic integral equations when he moves to a pathspace setting. In the first half of the book, everything is done in the context of general independent increment processes and without explicit use of Itô's stochastic integral calculus. In the second half, the author provides a systematic development of Itô's theory of stochastic integration: first for Brownian motion and then for continuous martingales. The final chapter presents Stratonovich's variation on Itô's theme and ends with an application to the characterization of the paths on which a diffusion is supported. The book should be accessible to readers who have mastered the essentials of modern probability theory and should provide such readers with a reasonably thorough introduction to continuous-time, stochastic processes.
Description : These volumes cover non-linear filtering (prediction and smoothing) theory and its applications to the problem of optimal estimation, control with incomplete data, information theory, and sequential testing of hypothesis. Also presented is the theory of martingales, of interest to those who deal with problems in financial mathematics. These editions include new material, expanded chapters, and comments on recent progress in the field.
Description : This book is an introduction to optimal stochastic control for continuous time Markov processes and the theory of viscosity solutions. It covers dynamic programming for deterministic optimal control problems, as well as to the corresponding theory of viscosity solutions. New chapters in this second edition introduce the role of stochastic optimal control in portfolio optimization and in pricing derivatives in incomplete markets and two-controller, zero-sum differential games.
Description : This work covers two topics in detail: Fourier analysis, with emphasis on positivity and also on some function spaces and multiplier theorems; and one-parameter operator semigroups with emphasis on Feller semigroups and Lp-sub-Markovian semigroups. In addition, Dirichlet forms are treated.
Description : Self-contained treatment covers both theory and applications. Topics include the fundamental role of homogeneous infinite Markov chains in the mathematical modeling of psychology and genetics. 1980 edition.
Description : This updated new edition introduces the reader to the fundamentals of queueing theory, including Markov processes and random walks. It contains an extended treatment of queueing networks and matrix analytic methods as well as additional topics like Poisson's equation, Palm theory and heavy tails.
Description : From observation to simulation -- Building the stochastic matrix -- Predictions by using 2-state Markov chains -- Predictions by using N-state Markov chains -- Absorbing Markov chains -- The average time spent in each state -- Discussions on different configurations of chains -- The simulation of an N-state Markov chain