MarkovChainsStationaryDistributions马尔可夫链的平稳分布

Markov Chains:Stationary Distributions
1.Stationary distribution:Let X n be a Markov Chain having state-space S and transition
function P.Ifπ(x)is a probability distribution such that:
x∈S
π(x)P(x,y)=π(y),y∈S,
thenπis a stationary distribution.This condition is often referred to as general balance and written down asπT P=πT.
2.What the above means:P maintainsπ.That is,ifπ0=π,thenπt=π0P t=πfor all t.
3.Our goal:Under certain conditions(irreducible and aperiodic-we will see these later),πis
unique andπt→πas t→∞,irrespective ofπ0.
4.Condition on transition probabilities:Suppose that a stationary distribution,π,exists
and that:
lim
n→∞
P n(x,y)=π(y),y∈S.
So,
P(X n=y)=
x∈S
π0(x)P n(x,y).
Therefore,
lim n→∞P(X n=y)=
x∈S
π0(x)π(y)=π(y).
Thus,X n L→π.
5.Observe:The stationary distribution,when X n L→π,is unique.
6.Example:For the2-state Markov Chain we saw that,if p+q>0,then the stationary
distribution is given by:
π(0)=
q
p+q
;π(1)=
p
p+q
.
We also saw that0<p+q<2then lim P n=πholds.
1
7.Example:Consider a Markov Chain having state-space S={0,1,2}and the transition
matrix:
P=
1/31/31/3
1/41/21/4
1/61/31/2
.
To check whether this chain has a stationary distribution or not,we try solving the general balance equations and check if the solution is a probability distribution.So,for the above:
πT(I−P)=0
which gives:
πT=
6
25
,
2
5
,
9
25
.
This is the unique stationary distribution.
8.HW:Consider an irreducible birth and death chain on{0,1,...,d}or on the non-negative
integers(here d=∞).Investigate thefinite and infinite cases for the existence of the stationary distribution.When it exists,give the form of the stationary distribution.
9.Example:Consider the Ehrenfest chain with d=3balls.Then,
P=
0100
1/302/30
02/301/3
0010
.
You may identify this as an irreducible birth and death chain and use your results from the above HW to get its stationary distribution.Otherwise,you may directly solve the general
balance equation to get:
πT=
1
8
,
3
8
,
3
8
,
1
8
.
Note,however,that lim P n π,since P n(x,x)=0for all odd n.
10.Modified Ehrenfest Chain:Suppose we have2boxes labelled1and2and d balls labelled
1,2,...,d.Initially some of the balls are in box1and the remainder are in box2.An integer is selected at random and the ball labelled by that integer is removed from its box.We now
2
select at random one of the two boxes and put the removed ball into this box.The procedure is repeated indefinitely,the selections being made independently.Let X n be the number of balls in box1after the n th trial.Set d=3.Then,the transition matrix is given by:
P=
1/21/200
1/61/21/30
01/31/21/6
001/21/2
.
The stationary distribution is again:
πT=
1
8
,
3
8
,
3
8
,
1
8
.
We will see later that here lim P n→π.
ying out the path in front:Consider an irreducible birth and death chain with stationary
distributionπ.Suppose that P(x,x)=r x=0,x∈S,as in the Ehrenfest chain.Then,at each transition the chain moves either one step to the right or one step to the left.So,the chain can return to its starting point only after an even number of transitions.Thus,P n(x,x)=0 for all n,and for such a chain the formula
lim
n→∞
P n(x,y)=π(y)
clearly fails to hold.But,there is a way to handle such situations.Recall our old friend Mr.
Cesaro from Real analysis:Let{a n}be a sequence of numbers such that,lim a n=L for some finite number L,then
lim 1
n
n
i=1
a i=L.
However,this formula can still hold even when lim a n fails to exist.For example,if a n=0for odd n and a n=1for even n,then a n has no limit,but the means still converge with L=1/2. We will see see that
lim n→∞1
n
n
k=1
P k(x,y)
will always exist for an arbitrary Markov Chain.Then we will use this limit to determine which Markov Chains have stationary distributions and when there is such a stationary distribution.
3。