In probability theory and statistics, the Markov property is the memoryless property of a stochastic process, which means that its future evolution is independent of its history. It is named after the Russian mathematician Andrey Markov. The term strong Markov property is similar to the Markov property, except that the meaning of "present" is defined in terms of a random variable known as a stopping time.
The term Markov assumption is used to describe a model where the Markov property is assumed to hold, such as a hidden Markov model.
A Markov random field extends this property to two or more dimensions or to random variables defined for an interconnected network of items. An example of a model for such a field is the Ising model.
A discrete-time stochastic process satisfying the Markov property is known as a Markov chain.
A stochastic process has the Markov property if the conditional probability distribution of future states of the process (conditional on both past and present values) depends only upon the present state; that is, given the present, the future does not depend on the past. A process with this property is said to be Markov or Markovian and known as a Markov process. Two famous classes of Markov process are the Markov chain and Brownian motion.
Note that there is a subtle, often overlooked and very important point that is often missed in the plain English statement of the definition: the statespace of the process is constant through time. The conditional description involves a fixed "bandwidth". For example, without this restriction we could augment any process to one which includes the complete history from a given initial condition and it would be made to be Markovian. But the state space would be of increasing dimensionality over time and does not meet the definition.
There are different formulations of the Markov property in the literature, depending on the assumed level of generality. In the following, some definitions are presented.
A sequence X 1 , X 2 , … {\displaystyle X_{1},X_{2},\dots } of random variables with values in N {\displaystyle \mathbb {N} } possesses the Markov property, if for all x 1 , … , x n + 1 ∈ N {\displaystyle x_{1},\dots ,x_{n+1}\in \mathbb {N} } , n ∈ N {\displaystyle n\in \mathbb {N} } , it holds
P ( X n + 1 = x n + 1 ∣ X n = x n , … , X 1 = x 1 ) = P ( X n + 1 = x n + 1 ∣ X n = x n ) , {\displaystyle P(X_{n+1}=x_{n+1}\mid X_{n}=x_{n},\dots ,X_{1}=x_{1})=P(X_{n+1}=x_{n+1}\mid X_{n}=x_{n}),}
if P ( X n = x n , … , X 1 = x 1 ) > 0 {\displaystyle P(X_{n}=x_{n},\dots ,X_{1}=x_{1})>0} . In other words, the distribution of X n + 1 {\displaystyle X_{n+1}} depend solely on the state of X n {\displaystyle X_{n}} and is independent of the state at any time previous to n {\displaystyle n} , which corresponds precisely to the intuition described in the introduction. The Markov property can be defined in the very same way for any finite or countable state space S {\displaystyle S} .
Let X = ( X t ) t ≥ 0 {\displaystyle X=(X_{t})_{t\geq 0}} be a stochastic process with values in N {\displaystyle \mathbb {N} } . Then X {\displaystyle X} possesses the Markov property, if for all 0 ≤ t 1 < ⋯ < t n + 1 {\displaystyle 0\leq t_{1}<\dots <t_{n+1}} and x 1 , … , x n + 1 ∈ N {\displaystyle x_{1},\dots ,x_{n+1}\in \mathbb {N} } , n ∈ N {\displaystyle n\in \mathbb {N} } , and it holds
P ( X t n + 1 = x n + 1 ∣ X t n = x n , … , X t 1 = x 1 ) = P ( X t n + 1 = x n + 1 ∣ X t n = x n ) , {\displaystyle P(X_{t_{n+1}}=x_{n+1}\mid X_{t_{n}}=x_{n},\dots ,X_{t_{1}}=x_{1})=P(X_{t_{n+1}}=x_{n+1}\mid X_{t_{n}}=x_{n}),}
if P ( X t n = x n , … , X t 1 = x 1 ) > 0 {\displaystyle P(X_{t_{n}}=x_{n},\dots ,X_{t_{1}}=x_{1})>0} .
The previous definitions can be significantly generalized to a wide class of state spaces and even to different notions of "time" (other than N {\displaystyle \mathbb {N} } and [ 0 , ∞ ) {\displaystyle [0,\infty )} ).
Let ( Ω , F , P ) {\displaystyle (\Omega ,{\mathcal {F}},P)} be a probability space with a filtration ( F s , s ∈ I ) {\displaystyle ({\mathcal {F}}_{s},\ s\in I)} , for some (totally ordered) index set I {\displaystyle I} ; and let ( S , Σ ) {\displaystyle (S,\Sigma )} be a measurable space. An ( S , Σ ) {\displaystyle (S,\Sigma )} -valued stochastic process X = { X t : Ω → S } t ∈ I {\displaystyle X=\{X_{t}:\Omega \to S\}_{t\in I}} adapted to the filtration is said to possess the Markov property if, for each A ∈ Σ {\displaystyle A\in \Sigma } and each s , t ∈ I {\displaystyle s,t\in I} with s < t {\displaystyle s<t} , it holds
In the case where S {\displaystyle S} is a discrete set with the discrete sigma algebra and I = N {\displaystyle I=\mathbb {N} } , this reduces to the discrete Markov property written above.
If I = [ 0 , ∞ ) {\displaystyle I=[0,\infty )} , then X {\displaystyle X} is called time-homogeneous if for all t , s ≥ 0 {\displaystyle t,s\geq 0} the weak Markov property holds:
The newly introduced probability measure P x ( X t ∈ ⋅ ) {\displaystyle P^{x}(X_{t}\in \cdot )} , x ∈ S {\displaystyle x\in S} , has the following intuition: It gives the probability that the process X {\displaystyle X} lies in some set at time t {\displaystyle t} , when it was started in x {\displaystyle x} at time zero. The function P t ( x , A ) := P x ( X t ∈ A ) {\displaystyle P_{t}(x,A):=P^{x}(X_{t}\in A)} , ( t , x , A ) ∈ R + × S × Σ {\displaystyle (t,x,A)\in \mathbb {R} _{+}\times S\times \Sigma } , is also called the transition function of X {\displaystyle X} and the collection ( P t ) t ≥ 0 {\displaystyle (P_{t})_{t\geq 0}} its transition semigroup.
There exists multiple alternative formulations of the elementary Markov property described above. The following are all equivalent:
For all t ≥ 0 {\displaystyle t\geq 0} the σ {\displaystyle \sigma } -algebras F t {\displaystyle {\mathcal {F}}_{t}} and F t ′ := σ ( X s : s ≥ t ) {\displaystyle {\mathcal {F}}_{t}':=\sigma (X_{s}:s\geq t)} are conditionally independent given X t {\displaystyle X_{t}} . In other words, for all A ∈ F t {\displaystyle A\in {\mathcal {F}}_{t}} , B ∈ F t ′ {\displaystyle B\in {\mathcal {F}}_{t}'} :
P ( A ∩ B ∣ X t ) = P ( A ∣ X t ) P ( B ∣ X t ) . {\displaystyle P(A\cap B\mid X_{t})=P(A\mid X_{t})P(B\mid X_{t}).}
For all t ≥ 0 {\displaystyle t\geq 0} , B ∈ F t ′ {\displaystyle B\in {\mathcal {F}}_{t}'} :
P ( B ∣ F t ) = P ( B ∣ X t ) . {\displaystyle P(B\mid {\mathcal {F}}_{t})=P(B\mid X_{t}).}
For all t ≥ 0 {\displaystyle t\geq 0} , A ∈ F t {\displaystyle A\in {\mathcal {F}}_{t}} :
P ( A ∣ F t ′ ) = P ( A ∣ X t ) . {\displaystyle P(A\mid {\mathcal {F}}_{t}^{'})=P(A\mid X_{t}).}
For all t ≥ 0 {\displaystyle t\geq 0} and Y : Ω → R {\displaystyle Y:\Omega \rightarrow \mathbb {R} } bounded and F t ′ {\displaystyle {\mathcal {F}}_{t}'} -measurable
E ⁡ [ Y ∣ F t ] = E ⁡ [ Y ∣ X t ] . {\displaystyle \operatorname {E} [Y\mid {\mathcal {F}}_{t}]=\operatorname {E} [Y\mid X_{t}].}
For all t ≥ s ≥ 0 {\displaystyle t\geq s\geq 0} and f : S → R {\displaystyle f:S\rightarrow \mathbb {R} } bounded and measurable
E ⁡ [ f ( X t ) ∣ F s ] = E ⁡ [ f ( X t ) ∣ X s ] . {\displaystyle \operatorname {E} [f(X_{t})\mid {\mathcal {F}}_{s}]=\operatorname {E} [f(X_{t})\mid X_{s}].}
For all t ≥ s ≥ 0 {\displaystyle t\geq s\geq 0} and f : S → R {\displaystyle f:S\rightarrow \mathbb {R} } continuous with compact support
E ⁡ [ f ( X t ) ∣ F s ] = E ⁡ [ f ( X t ) ∣ X s ] . {\displaystyle \operatorname {E} [f(X_{t})\mid {\mathcal {F}}_{s}]=\operatorname {E} [f(X_{t})\mid X_{s}].}
For all 0 ≤ s 1 < . . . < s n < s < t {\displaystyle 0\leq s_{1}<...<s_{n}<s<t} and f : S → R {\displaystyle f:S\rightarrow \mathbb {R} } continuous with compact support
E ⁡ [ f ( X t ) ∣ X s , X s n , . . . , X s 1 ] = E ⁡ [ f ( X t ) ∣ X s ] . {\displaystyle \operatorname {E} [f(X_{t})\mid X_{s},X_{s_{n}},...,X_{s_{1}}]=\operatorname {E} [f(X_{t})\mid X_{s}].} In particular, if S = N {\displaystyle S=\mathbb {N} } , then any function f {\displaystyle f} is continuous, and by choosing f = 1 { x } {\displaystyle f=1_{\{x\}}} the Markov property for continuous time and discrete state space follows. If there exists a so-called shift-semigroup ( θ t ) t ≥ 0 {\displaystyle (\theta _{t})_{t\geq 0}} , i.e., functions θ t : Ω → Ω {\displaystyle \theta _{t}:\Omega \to \Omega } such that
θ 0 = i d Ω {\displaystyle \theta _{0}=\mathrm {id} _{\Omega }} ,
θ t ∘ θ s = θ t + s ∀ s , t ≥ 0 {\displaystyle \theta _{t}\circ \theta _{s}=\theta _{t+s}\quad \forall s,t\geq 0} (semigroup property),
X t ∘ θ s = X t + s ∀ s , t ≥ 0 {\displaystyle X_{t}\circ \theta _{s}=X_{t+s}\quad \forall s,t\geq 0} ,
then the Markov property is equivalent to:
For all t ≥ 0 {\displaystyle t\geq 0} and Λ ∈ F 0 ′ {\displaystyle \Lambda \in {\mathcal {F}}_{0}'}
P ( θ t − 1 ( Λ ) ∣ F t ) = P ( θ t − 1 ( Λ ) ∣ X t ) . {\displaystyle P(\theta _{t}^{-1}(\Lambda )\mid {\mathcal {F}}_{t})=P(\theta _{t}^{-1}(\Lambda )\mid X_{t}).}
For all t ≥ 0 {\displaystyle t\geq 0} and Y : Ω → R {\displaystyle Y:\Omega \rightarrow \mathbb {R} } bounded and F 0 ′ {\displaystyle {\mathcal {F}}_{0}'} -measurable
E ⁡ [ Y ∘ θ t ∣ F t ] = E ⁡ [ Y ∘ θ t ∣ X t ] . {\displaystyle \operatorname {E} [Y\circ \theta _{t}\mid {\mathcal {F}}_{t}]=\operatorname {E} [Y\circ \theta _{t}\mid X_{t}].} Depending on the situation, some formulations might be easier to verify or to use than others.
Suppose that X = ( X t : t ≥ 0 ) {\displaystyle X=(X_{t}:t\geq 0)} is a stochastic process on a probability space ( Ω , F , P ) {\displaystyle (\Omega ,{\mathcal {F}},P)} with natural filtration { F t } t ≥ 0 {\displaystyle \{{\mathcal {F}}_{t}\}_{t\geq 0}} . Then for any stopping time τ {\displaystyle \tau } on Ω {\displaystyle \Omega } , we can define
Then X {\displaystyle X} is said to have the strong Markov property if, for each stopping time τ {\displaystyle \tau } , conditional on the event { τ < ∞ } {\displaystyle \{\tau <\infty \}} , we have that for each t ≥ 0 {\displaystyle t\geq 0} , X τ + t {\displaystyle X_{\tau +t}} is independent of F τ {\displaystyle {\mathcal {F}}_{\tau }} given X τ {\displaystyle X_{\tau }} . This is equivalent to
where 1 { τ < ∞ } {\displaystyle 1_{\{\tau <\infty \}}} denotes to indicator function of the set { τ < ∞ } {\displaystyle \{\tau <\infty \}} .
The strong Markov property implies the ordinary Markov property since by taking the stopping time τ = t {\displaystyle \tau =t} , the ordinary Markov property can be deduced. The converse is in general not true.
The strong Markov property only leads to non-trivial results in continuous time (i.e., results which do not hold with merely the Markov property), as in the discrete case the strong and the elementary Markov property are equivalent.
Although the strong Markov property is in general stronger than the elementary Markov property, it is fulfilled by Markov processes with sufficiently "nice" regularity properties.
A continuous time, homogeneous Markov process is said to have the Feller property, if its transition semigroup ( P t ) t ≥ 0 {\displaystyle (P_{t})_{t\geq 0}} (see above) fulfills
P t f := ∫ f ( x ) P t ( ⋅ , d x ) ∈ C 0 ( S ) {\displaystyle P_{t}f:=\int f(x)P_{t}(\cdot ,dx)\in C_{0}(S)} for all f ∈ C 0 ( S ) {\displaystyle f\in C_{0}(S)} ,
lim t → 0 ‖ P t f − f ‖ ∞ = 0 {\displaystyle \lim _{t\to 0}\,\lVert P_{t}f-f\rVert _{\infty }=0} for all f ∈ C 0 ( S ) {\displaystyle f\in C_{0}(S)} ,
where C 0 ( S ) {\displaystyle C_{0}(S)} denotes the set of continuous functions vanishing at infinity and ‖ ⋅ ‖ ∞ {\displaystyle \lVert \cdot \rVert _{\infty }} the sup norm.
In continuous time, the elementary Markov property is a rather weak condition and does not imply particularly interesting properties other than maybe the fact that for a Markov process X = ( X t ) t ≥ 0 {\displaystyle X=(X_{t})_{t\geq 0}} the reverse process ( X T − t ) t ≥ 0 {\displaystyle (X_{T-t})_{t\geq 0}} from a time T > 0 {\displaystyle T>0} has again the Markov property.
If X {\displaystyle X} is time-homogeneous and fulfills the weak Markov property, then already a few more statements can be made.
The law of the process on the path space ( R [ 0 , ∞ ) , B ( R ) ⊗ [ 0 , ∞ ) ) {\displaystyle (\mathbb {R} ^{[0,\infty )},{\mathcal {B}}(\mathbb {R} )^{\otimes [0,\infty )})} is uniquely determined by its transition semigroup ( P t ) t ≥ 0 {\displaystyle (P_{t})_{t\geq 0}} and the distribution of X 0 {\displaystyle X_{0}} .
For any α > 0 {\displaystyle \alpha >0} and bounded, non-negative, measurable function f {\displaystyle f} , the process
Z t = e − α t E X t ⁡ [ ∫ 0 ∞ e − α s f ( X s ) d s ] , t ≥ 0 , {\displaystyle Z_{t}=e^{-\alpha t}\operatorname {E} ^{X_{t}}{\Big [}\int _{0}^{\infty }e^{-\alpha s}f(X_{s})\mathrm {d} s{\Big ]},\qquad t\geq 0,} is a supermartingale and Y t = ∫ 0 t e − α s f ( X s ) d s + e − α t E X t ⁡ [ ∫ 0 ∞ e − α s f ( X s ) d s ] , t ≥ 0 , {\displaystyle Y_{t}=\int _{0}^{t}e^{-\alpha s}f(X_{s})\mathrm {d} s+e^{-\alpha t}\operatorname {E} ^{X_{t}}{\Big [}\int _{0}^{\infty }e^{-\alpha s}f(X_{s})\mathrm {d} s{\Big ]},\qquad t\geq 0,} is a uniformly integrable martingale.
However, the weak Markov property alone is still too general to imply much more than the above properties. Most statements on Markov processes require the Feller property (see above), which provides more regularity of the transition semigroup.
In the following, let X {\displaystyle X} be a Feller process, that is, a time-homogeneous Markov process with the Feller property, in a filtration ( F t ) t ≥ 0 {\displaystyle ({\mathcal {F}}_{t})_{t\geq 0}} with values in a Polish space S {\displaystyle S} and ( P t ) t ≥ 0 {\displaystyle (P_{t})_{t\geq 0}} being its transition semigroup.
If the filtration ( F t ) t ≥ 0 {\displaystyle ({\mathcal {F}}_{t})_{t\geq 0}} is augmented, then X {\displaystyle X} has a modification with right-continuous (even càdlàg) paths, and this modification has the strong Markov property. Thus, Feller processes are often assumed to be right-continuous strong Markov processes without further mention.
Let τ {\displaystyle \tau } be any stopping time such that E ⁡ [ τ ] < ∞ {\displaystyle \operatorname {E} [\tau ]<\infty } and let f ∈ C 0 ( S ) {\displaystyle f\in C_{0}(S)} be such that
A f ( x ) := lim t → 0 E x ⁡ [ f ( X t ) ] − f ( x ) t , x ∈ S , {\displaystyle Af(x):=\lim _{t\to 0}{\frac {\operatorname {E} ^{x}[f(X_{t})]-f(x)}{t}},\qquad x\in S,}
exists ( A {\displaystyle A} is the infinitesimal generator of X {\displaystyle X} ). Then, for all x ∈ S {\displaystyle x\in S} ,
E x ⁡ [ f ( X τ ) ] = f ( x ) + E x ⁡ [ ∫ 0 τ A f ( X t ) d t ] . {\displaystyle \operatorname {E} ^{x}[f(X_{\tau })]=f(x)+\operatorname {E} ^{x}{\Big [}\int _{0}^{\tau }Af(X_{t})\mathrm {d} t{\Big ]}.}
For a set A ⊂ S {\displaystyle A\subset S} , one might be interested in the question whether the process will return infinitely often to the set A {\displaystyle A} . Defining the set of paths under which the later holds as
R ( A ) := { lim sup t → ∞ 1 A ( X t ) = 1 } , {\displaystyle R(A):=\left\{\limsup _{t\to \infty }1_{A}(X_{t})=1\right\},}
the set A {\displaystyle A} is called recurrent, if P ( R ( A ) ) = 1 {\displaystyle P(R(A))=1} , and transient, if P ( R ( A ) ) = 0 {\displaystyle P(R(A))=0} .
In general, a set may be neither recurrent nor transient. For Feller processes, however, the following two conditions are equivalent.
Every bounded function f {\displaystyle f} (on Ω {\displaystyle \Omega } ) that is invariant with respect to the shift, that is, f ∘ θ t = f {\displaystyle f\circ \theta _{t}=f} for all t ≥ 0 {\displaystyle t\geq 0} , is already constant.
Every set is either recurrent or transient.
The strong Markov property is conserved under an h {\displaystyle h} -transform–a method introduced by J.L. Doob to manipulate Markov processes. For a function h : S → ( 0 , ∞ ) {\displaystyle h:S\to (0,\infty )} fulfilling
E ⁡ [ h ( X t ) ] = ∫ S h ( y ) P t ( x , d y ) = h ( x ) , ∀ x ∈ S , t ≥ 0 , {\displaystyle \operatorname {E} [h(X_{t})]=\int _{S}h(y)P_{t}(x,\mathrm {d} y)=h(x),\qquad \forall x\in S,\,t\geq 0,}
the h {\displaystyle h} -transformed process X h {\displaystyle X^{h}} is the (homogeneous) Markov process with transition function
P t h ( x , A ) = 1 h ( x ) ∫ A h ( y ) P t ( x , d y ) , ∀ x ∈ S , A ∈ B ( S ) , t ≥ 0. {\displaystyle P_{t}^{h}(x,A)={\frac {1}{h(x)}}\int _{A}h(y)P_{t}(x,\mathrm {d} y),\qquad \forall x\in S,\,A\in {\mathcal {B}}(S),\,t\geq 0.}
As mentioned above, if X ~ {\displaystyle {\tilde {X}}} is a right-continuous modification of the original process X {\displaystyle X} with the strong Markov property, then its h {\displaystyle h} -transform X ~ h {\displaystyle {\tilde {X}}^{h}} is a right-continuous strong Markov process as well.
Let X ~ {\displaystyle {\tilde {X}}} be a right-continuous modification of the original process X {\displaystyle X} with the strong Markov property and suppose X ~ {\displaystyle {\tilde {X}}} is killed at a random time ξ {\displaystyle \xi } with P ( ξ < ∞ ) > 0 {\displaystyle P(\xi <\infty )>0} (this means that for t ≥ ξ {\displaystyle t\geq \xi } the process is sent to a so-called cemetery point ∂ ∉ S {\displaystyle \partial \notin S} ) in such a way that the killed process is still a homogeneous Markov process. Then the reverse process
X t ← ( ω ) = { X ~ ξ − t ( ω ) , if 0 < t ≤ ξ ( ω ) , ∂ , if t > ξ ( ω ) or ξ ( ω ) = ∞ {\displaystyle {\overset {\leftarrow }{X_{t}}}(\omega )={\begin{cases}{\tilde {X}}_{\xi -t}(\omega ),&&{\text{ if }}0<t\leq \xi (\omega ),\\\partial ,&&{\text{ if }}t>\xi (\omega ){\text{ or }}\xi (\omega )=\infty \end{cases}}}
is a left-continuous homogeneous Markov process with semigroup ( P t ← ) t ≥ 0 {\displaystyle ({\overset {\leftarrow }{P_{t}}})_{t\geq 0}} . If P t ← {\displaystyle {\overset {\leftarrow }{P_{t}}}} and P t {\displaystyle P_{t}} are understood as operators P t : f ↦ ∫ S f ( x ) P t ( ⋅ , d x ) {\displaystyle P_{t}:f\mapsto \int _{S}f(x)P_{t}(\cdot ,\mathrm {d} x)} , then P t ← {\displaystyle {\overset {\leftarrow }{P_{t}}}} is the adjoint of P t {\displaystyle P_{t}} with respect to the scalar product induced by the measure
G ( A ) = ∫ 0 ∞ E ⁡ [ P t ( X ~ 0 , A ) ] d t , A ∈ B ( S ) , {\displaystyle G(A)=\int _{0}^{\infty }\operatorname {E} [P_{t}({\tilde {X}}_{0},A)]\mathrm {d} t,\qquad A\in {\mathcal {B}}(S),}
in the sense
⟨ P t ← f , g ⟩ G = ⟨ f , P t g ⟩ G = ∫ S f ( x ) P t g ( x ) d G ( x ) {\displaystyle \langle {\overset {\leftarrow }{P_{t}}}f,g\rangle _{G}=\langle f,P_{t}g\rangle _{G}=\int _{S}f(x)P_{t}g(x)\mathrm {d} G(x)}
for all continuous functions f , g ∈ C ( S ) {\displaystyle f,g\in C(S)} .
Moreover, if there exists a measure ν {\displaystyle \nu } such that ⟨ P t f , g ⟩ ν = ⟨ f , P t g ⟩ ν {\displaystyle \langle P_{t}f,g\rangle _{\nu }=\langle f,P_{t}g\rangle _{\nu }} (in this case X ~ {\displaystyle {\tilde {X}}} is called self-dual) and the measure G {\displaystyle G} is absolutely continuous with respect to ν {\displaystyle \nu } , then X ← {\displaystyle {\overset {\leftarrow }{X}}} has the same transition probabilities as the h {\displaystyle h} -transform X ~ h {\displaystyle {\tilde {X}}^{h}} with h = d G / d ν {\displaystyle h=\mathrm {d} G/\mathrm {d} \nu } . So in particular, if X ~ 0 h {\displaystyle {\tilde {X}}_{0}^{h}} and X 0 ← {\displaystyle {\overset {\leftarrow }{X_{0}}}} have the same distribution, then X ← {\displaystyle {\overset {\leftarrow }{X}}} and X ~ h {\displaystyle {\tilde {X}}^{h}} are the same process.
Assume that an urn contains two red balls and one green ball. One ball was drawn yesterday, one ball was drawn today, and the final ball will be drawn tomorrow. All of the draws are "without replacement".
Suppose you know that today's ball was red, but you have no information about yesterday's ball. The chance that tomorrow's ball will be red is 1/2. That's because the only two remaining outcomes for this random experiment are:
On the other hand, if you know that both today and yesterday's balls were red, then you are guaranteed to get a green ball tomorrow.
This discrepancy shows that the probability distribution for tomorrow's color depends not only on the present value, but is also affected by information about the past. This stochastic process of observed colors doesn't have the Markov property. Using the same experiment above, if sampling "without replacement" is changed to sampling "with replacement," the process of observed colors will have the Markov property.
Many prominent stochastic processes are Markov processes: The Brownian motion, the Brownian bridge, the stochastic exponential, the Ornstein-Uhlenbeck process and the Poisson process have the Markov property.
More generally, any semimartingale X {\displaystyle X} with values in R n {\displaystyle \mathbb {R} ^{n}} that is given by the stochastic differential equation
where B = ( B 1 , . . . , B d ) {\displaystyle B=(B^{1},...,B^{d})} is a d {\displaystyle d} -dimensional Brownian motion and f 1 , . . . , f d , g 1 , . . . , g d : R n → R n {\displaystyle f_{1},...,f_{d},g_{1},...,g_{d}:\mathbb {R} ^{n}\to \mathbb {R} ^{n}} are autonomous (i.e., they do not depend on time) Lipschitz functions, is time-homogeneous and has the strong Markov property. If f 1 , . . . , f d , g 1 , . . . , g d {\displaystyle f_{1},...,f_{d},g_{1},...,g_{d}} are not autonomous, then X {\displaystyle X} still has the elementary Markov property.
In the fields of predictive modelling and probabilistic forecasting, the Markov property is considered desirable since it may enable the reasoning and resolution of the problem that otherwise would not be possible to be resolved because of its intractability. Such a model is known as a Markov model.
An application of the Markov property in a generalized form is in Markov chain Monte Carlo computations in the context of Bayesian statistics.
Causal Markov condition
Chapman–Kolmogorov equation
Doob's h-transform
Hysteresis
Markov blanket
Markov chain
Markov decision process
Markov model
