第 13 章 贝叶斯统计
13.1 Basics
Definition 13.1. A set \(G\) of elements is called a group if it satisfies the following four conditions.
There is defined an operation, group multiplication, which with any two elements \(a,b\in G\) associates an element \(c\) of \(G\). The element \(c\) is called the product of \(a\) and \(b\) and is denoted \(ab\);
Group multiplication obeys the associative law, which means \((ab)c=a(bc)\);
There exists an element \(e\in G\), called the identity, such that \(ae=ea=a,\;\forall\;a\in G\);
For each element \(a\in G\), there exists an element \(a^{-1}\in G\) which called its inverse, such that \(aa^{-1}=a^{-1}a=e\).
Definition 13.2. A class \(G\) of transformations is called a transformation group if it is closed under both composition and inversion.
Theorem 13.1. A transformation group is a group.
13.1.1 Group Family
Definition 13.3. A group family of distributions is a family obtained by subjecting a random vector with a fixed distribution to a transformation group.
Property 13.1.1. A group family is independent of which of its members is takn as starting distribution.
证明. Let \(\mathbf{X}\) denote an \(n\)-dimensional random vector characterized by a specific distribution \(F\). The group family \(G\) is formed by applying a transformation group \(T\) to \(\mathbf{X}\). Let \(\mathbf{Y}\) be an element of \(G\) that is distinct from \(\mathbf{X}\). The distribution \(F\) can be derived by applying the transformation \(T\) to \(\mathbf{Y}\). For any \(\mathbf{Z} \in G\), there exists a transformation \(\mathcal{T}_1 \in T\) such that \(\mathbf{X} = \mathcal{T}_1 \mathbf{Z}\). Given that \(\mathbf{Y} \in G\), there also exists a transformation \(\mathcal{T}_2 \in T\) such that \(\mathbf{Y} = \mathcal{T}_2 \mathbf{X}\). Consequently, we can express \(\mathbf{Y}\) as \(\mathbf{Y} = \mathcal{T}_2 \mathcal{T}_1 \mathbf{Z}\). Since \(T\) is a transformation group, it follows that \(\mathcal{T}_2 \mathcal{T}_1 \in T\). Therefore, the inverse \((\mathcal{T}_2 \mathcal{T}_1)^{-1} \in T\), which implies that \(\mathbf{Z} = (\mathcal{T}_2 \mathcal{T}_1)^{-1} \mathbf{Y} \in F\). This leads to the conclusion that \(G \subset F\). By a similar argument, one can establish that \(F \subset G\), thereby concluding that \(G = F\). This indicates that the family is invariant with respect to the choice of representative element. ◻
Definition 13.4. （Location-scale families）
Let \(\mathbf{X} = (\mathbf{X}_1, \mathbf{X}_2, \dots, \mathbf{X}_{n})\) be an \(n\)-dimensional random vector with a fixed joint distribution function \(F\). We define the following families of distributions based on affine transformations of \(\mathbf{X}\):
Location family: For any constant vector \(a = (a_1, a_2, \dots, a_n) \in \mathbb{R}^n\), define a shifted random vector \(\mathbf{Y} = \mathbf{X} + a\). The distribution function of \(\mathbf{Y}\) is given by:
\[ P(\mathbf{Y}_1 \leqslant y_1, \mathbf{Y}_2 \leqslant y_2, \dots, \mathbf{Y}_n \leqslant y_n) = F(y_1 - a_1, y_2 - y_2, \dots, y_n - a_n) \]
The collection of all such distributions for a fixed \(F\) and varying \(a\) is called a location family.
Scale family: For any constant vector \(b = (b_1, b_2, \dots, b_n) \in \mathbb{R}^n\) with all \(b_i > 0\), define the scaled random vector \(\mathbf{Y} = b \otimes \mathbf{X}\) (element-wise multiplication). Then the distribution function of \(\mathbf{Y}\) is:
\[ P(\mathbf{Y}_1 \leqslant y_1, \mathbf{Y}_2 \leqslant y_2, \dots, \mathbf{Y}_n \leqslant y_n) = F\left(\frac{y_1}{b_1}, \frac{y_2}{b_2}, \dots, \frac{y_n}{b_n}\right) \]
The collection of such distributions for fixed \(F\) and varying \(b\) is called a scale family.
Location-scale family: When both a shift and scaling are applied simultaneously, we define the transformed vector as:
\[ \mathbf{Y} = a + b \otimes \mathbf{X} \]
where \(a \in \mathbb{R}^n\) is a location vector and \(b \in \mathbb{R}^n\) is a scale vector with strictly positive entries. The resulting collection of distributions for all such transformations of a fixed \(F\) forms a location-scale family.
Definition 13.5. When neither the \(T_i\)’s nor the \(\theta_i\)’s satisfy a linear constraint, the exponential family is said to be minimal. If the \(\theta\)’s are related in a nonlinear way, it forms a curved exponential family.
Property 13.1.2. The exponential families has the following properties:
Exponential families have natural conjugate priors;
\(\operatorname{E}(T_i)=\dfrac{\partial f(\theta)}{\partial\theta_i},\;\operatorname{Cov}(T_i,T_j)=\dfrac{\partial^2f(\theta)}{\partial\theta_i\partial\theta_j}\);
证明. (1)The likelihood corresponding to a sequence \(y = (y_1, y_2, \dots, y_{m})\) of i.i.d observations is:
\[\begin{equation*} p(y|\theta)\propto\prod_{i=1}^{m}\exp\left[\sum_{j=1}^{n}\theta_jT_j(y_i)-f(\theta)\right]=\exp\left[\sum_{i=1}^{n}\theta_i\sum_{j=1}^{m}T_i(y_j)-mf(\theta)\right] \end{equation*}\]
If \(\theta\)’s pdf is specified as:
\[\begin{equation*} p(\theta)\propto\exp\left[\sum_{i=1}^{n}\theta_i\alpha_i-g(\theta)\right] \end{equation*}\]
We have:
\[\begin{equation*} p(\theta|y)\propto\exp\left\{\sum_{i=1}^{n}\theta_i\left[\sum_{j=1}^{m}T_i(y_j)+\alpha_i\right]-[mf(\theta)+g(\theta)]\right\} \end{equation*}\]
which completes the proof. ◻
13.2 Bayesian
The process of Bayesian data analysis can be idealized by dividing it into the following three stes:
Setting up a full probability model which means a joint probability distribution for all observable quantities in a problem.
Conditioning on observed data: calculating and interpreting the appropriate posterior distribution——the conditional probability distribution of the unobserved
Theorem 13.2. 链接全概率公式与条件概率，同时写完测度后重新定义
Definition 13.6. The parameters associated with the prior distribution are termed hyperparameters.
Definition 13.7. The highest posterior density region is defined as the set of values that encompasses \(100(1-\alpha)\%\) of the posterior probability, characterized by the property that the density within this region is never less than that outside of it.
A central interval of posterior probability, within the context of a \(100(1 - \alpha)\%\) interval, denotes the range of values that includes exactly \(100(\alpha/2)\%\) of the posterior probability on each side.
13.2.1 Binomial Distribution
\[\begin{equation*} p(y|\theta)=\operatorname{Binom}(y|n,\theta)=\binom{n}{y}\theta^y(1-\theta)^{n-y} \end{equation*}\]
Theorem 13.3. When the prior distribution for the parameter \(\theta\) is specified as uniform over the interval \([0,1]\), the resulting posterior distribution for the binomial model \(y\sim\operatorname{B}(n,\theta)\) is characterized by a Beta distribution, specifically \(\operatorname{Beta}(y+1, n-y+1)\). Letting \(\tilde{y}\) represent the result of a new trial which is exchangeable with the first \(n\), then:
\[\begin{equation*} P(\tilde{y}=1| y)=\frac{y+1}{n+2} \end{equation*}\]
证明. As 定理 13.2 shows:
\[\begin{equation*} p(\theta | y) = \frac{p(y | \theta) p(\theta)}{p(y)} \propto p(\theta) p(y | \theta) = p(y | \theta) = \binom{n}{y} \theta^y (1 - \theta)^{n - y} \propto \theta^y (1 - \theta)^{n - y} \end{equation*}\]
Since the prior distribution \(p(\theta)\) is uniform over \([0,1]\), we have \(p(\theta) = 1\) for \(\theta \in [0,1]\). Therefore, the posterior distribution is proportional to the likelihood:
\[ p(\theta | y) \propto \theta^y(1 - \theta)^{n - y} \]
This is the kernel of a Beta distribution. The probability density function of the Beta distribution with parameters \(\alpha\) and \(\beta\) is given by:
\[ p(\theta ; \alpha, \beta) = \frac{1}{B(\alpha, \beta)} \theta^{\alpha - 1} (1 - \theta)^{\beta - 1} \]
for \(\theta \in [0,1]\), where \(B(\alpha, \beta)\) is the Beta function, which acts as the normalization constant.
Comparing this with the posterior kernel \(\theta^y(1 - \theta)^{n - y}\), we identify \(\alpha = y + 1\) and \(\beta = n - y + 1\). Therefore, the posterior distribution of \(\theta\) is:
\[ \theta | y \sim \operatorname{Beta}(y + 1, n - y + 1) \]
Now consider a new trial result \(\tilde{y}\) that is exchangeable with the original \(n\) trials. The predictive probability of success in the new trial is given by the expectation of \(\theta\) under the posterior distribution:
\[\begin{align*} P(\tilde{y} = 1 | y) &= \int_0^1 P(\tilde{y} = 1 | \theta, y)p(\theta | y)\mathop{}\!\mathrm{d}\theta \\ &= \int_0^1 \theta p(\theta | y)\mathop{}\!\mathrm{d}\theta = \operatorname{E}(\theta | y) = \frac{y + 1}{n + 2} \end{align*}\]
which completes the proof. ◻
Theorem 13.4. The Beta distribution is the conjugate prior for the binomial likelihood. Specifically, if the prior distribution of the success probability \(\theta\) is \(\operatorname{Beta}(\alpha, \beta)\), and the data consist of \(y\) successes in \(n\) Bernoulli trials, then the posterior distribution of \(\theta\) is \(\operatorname{Beta}(\alpha + y, \beta + n - y)\).
证明. Assume that the prior distribution of \(\theta\) follows a Beta distribution with hyperparameters \(\alpha\) and \(\beta\), so that:
\[ p(\theta) \propto \theta^{\alpha - 1} (1 - \theta)^{\beta - 1} \]
The likelihood function for observing \(y\) successes out of \(n\) trials in a binomial experiment is:
\[ p(y | \theta) \propto \theta^y (1 - \theta)^{n - y} \]
Applying 定理 13.2, the posterior distribution is given by:
\[\begin{align*} p(\theta | y) &\propto p(\theta)p(y | \theta) \\ &\propto \theta^{\alpha - 1} (1 - \theta)^{\beta - 1}\theta^y (1 - \theta)^{n - y} \\ &\propto \theta^{\alpha + y - 1} (1 - \theta)^{\beta + n - y - 1} \end{align*}\]
This expression matches the kernel of a Beta distribution with updated parameters. Therefore, the posterior distribution is:
\[ \theta | y \sim \operatorname{Beta}(\alpha + y, \beta + n - y) \]
This confirms that the Beta distribution is conjugate to the binomial likelihood, completing the proof. ◻
13.2.2 Normal Distribution
13.2.2.1 Known Variance
Theorem 13.5. When the variance is known, the normal distribution is the conjugate prior for the mean \(\mu\) of a normal likelihood. Specifically, suppose there are \(n\) i.i.d observations \(y=(y_1, y_2, \dots, y_{n})\) and the likelihood is \(y_i|\mu\sim\operatorname{N}(\mu,\sigma^2)\), where \(\sigma^2\) is known. If the prior for \(\mu\) is \(\mu\sim\operatorname{N}(\nu,\tau^2)\), then the posterior distribution of \(\mu\) is also normal:
\[\begin{equation*} \mu|y\sim\operatorname{N}\left(\frac{n\tau^2\bar{y} + \sigma^2\nu}{n\tau^2 + \sigma^2}, \frac{\sigma^2\tau^2}{n\tau^2 + \sigma^2}\right) \end{equation*}\]
where \(\bar{y} = \dfrac{1}{n}\sum\limits_{i=1}^n y_i\) is the sample mean.
证明.
\[\begin{align*} p(\mu|y)&\propto p(\mu)p(y|\mu)\propto \exp\left\{-\frac{1}{2}\left[\frac{1}{\sigma^2}\sum_{i=1}^{n}(y_i-\mu)^2+\frac{(\mu-\nu)^2}{\tau^2}\right]\right\} \\ &\propto\exp\left\{-\frac{1}{2}\left[\frac{1}{\sigma^2}\sum_{i=1}^{n}(y_i^2-2y_i\mu+\mu^2)+\frac{(\mu-\nu)^2}{\tau^2}\right]\right\} \\ &\propto\exp\left\{-\frac{1}{2}\left[\frac{n\mu^2-2n\bar{y}\mu}{\sigma^2}+\frac{(\mu-\nu)^2}{\tau^2}\right]\right\} \\ &=\exp\left[-\frac{n\tau^2\mu^2-2n\tau^2\bar{y}\mu+\sigma^2(\mu^2-2\nu\mu+\nu^2)}{2\sigma^2\tau^2}\right] \\ &\propto\exp\left[-\frac{(n\tau^2+\sigma^2)\mu^2-2(n\tau^2\bar{y}+\sigma^2\nu)\mu}{2\sigma^2\tau^2}\right] \\ &=\exp\left[-\frac{\mu^2-2\dfrac{n\tau^2\bar{y}+\sigma^2\nu}{\sigma^2+\tau^2}\mu}{\dfrac{2\sigma^2\tau^2}{n\tau^2+\sigma^2}}\right] \\ &\propto\exp\left[-\frac{1}{2}\frac{\left(\mu-\dfrac{n\tau^2\bar{y}+\sigma^2\nu}{\sigma^2+\tau^2}\right)^2}{\dfrac{\sigma^2\tau^2}{n\tau^2+\sigma^2}}\right] \end{align*}\]
◻
Theorem 13.6. The posterior distribution of a future observation \(\tilde{y}\) is:
\[\begin{equation*} \tilde{y}|y\sim\operatorname{N}\left(\frac{n\tau^2\bar{y}+\sigma^2\nu}{n\tau^2+\sigma^2},\frac{\sigma^2\tau^2}{n\tau^2+\sigma^2}+\sigma^2\right) \end{equation*}\]
证明. Let:
\[\begin{equation*} \mu_1=\frac{n\tau^2\bar{y}+\sigma^2\nu}{n\tau^2+\sigma^2},\;\sigma^2_1=\frac{\sigma^2\tau^2}{n\tau^2+\sigma^2} \end{equation*}\]
Given that the value of \(\tilde{y}\) is independent of \(y\) when conditioned on \(\mu\), we can express the conditional probability as follows:
\[\begin{align*} p(\tilde{y}|y) &=\int_{}p(\tilde{y}|\mu,y)p(\mu|y)\mathop{}\!\mathrm{d}\mu=\int_{}p(\tilde{y}|\mu)p(\mu|y)\mathop{}\!\mathrm{d}\mu \\ &\propto\int_{}\exp\left[-\frac{(\tilde{y}-\mu)^2}{2\sigma^2}\right]\exp\left[-\frac{(\mu-\mu_1)^2}{2\sigma_1^2}\right]\mathop{}\!\mathrm{d}\mu \end{align*}\]
The above expression can be interpreted as the marginal density of the bivariate normal distribution \((\tilde{y}, \mu)\) with respect to \(\tilde{y}\). According to 性质 6.4.6(3), it follows that \(\tilde{y}\) is normally distributed. Utilizing 性质 6.3.2 and 性质 6.3.5, we derive the following results:
\[\begin{gather*} \operatorname{E}(\tilde{y}|y)=\operatorname{E}[\operatorname{E}(\tilde{y}|\mu,y)|y]=\operatorname{E}[\operatorname{E}(\tilde{y}|\mu)|y]=\operatorname{E}(\mu|y)=\mu_1 \\ \begin{aligned} \operatorname{Var}(\tilde{y}|y)&=\operatorname{E}[\operatorname{Var}(\tilde{y}|\theta,y)|y]+\operatorname{Var}[\operatorname{E}(\tilde{y}|\theta,y)|y]=\operatorname{E}[\operatorname{Var}(\tilde{y}|\theta)|y]+\operatorname{Var}[\operatorname{E}(\tilde{y}|\theta)|y] \\ &=\operatorname{E}(\sigma^2|y)+\operatorname{Var}(\mu|y)=\sigma_1^2+\sigma^2 \end{aligned} \end{gather*}\]
◻
13.2.2.2 Known Mean
Theorem 13.7. When the mean is known, the inverse-gamma distribution is the conjugate prior for the variance \(\sigma^2\) of a normal likelihood. Specifically, suppose there are \(n\) i.i.d observations \(y=(y_1, y_2, \dots, y_{n})\) and the likelihood is \(y_i|\mu\sim\operatorname{N}(\mu,\sigma^2)\), where \(\mu\) is known. If the prior for \(\sigma^2\) is \(\sigma^2\sim\operatorname{Inv-Gamma}(\alpha,\lambda)\), then the posterior distribution of \(\sigma^2\) is:
\[\begin{equation*} \sigma^2|y\sim\operatorname{Inv-Gamma}\left[\frac{n}{2}+\alpha,\lambda+\frac{1}{2}\sum_{i=1}^{n}(y_i-\mu)^2\right] \end{equation*}\]
证明.
\[\begin{align*} p(\sigma^2|y)&\propto p(\sigma^2)p(y|\sigma^2) \propto(\sigma^2)^{-\alpha-1}\exp\left(-\frac{\lambda}{\sigma^2}\right)\sigma^{-n}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(y_i-\mu)^2\right] \\ &=(\sigma^2)^{-\frac{n}{2}-\alpha-1}\exp\left[-\frac{2\lambda+\sum\limits_{i=1}^{n}(y_i-\mu)^2}{2\sigma^2}\right] \end{align*}\]
◻
13.2.3 Possion Model
Theorem 13.8. The gamma distribution is the conjugate prior for the possion likelihood. Specifically, suppose there are \(n\) i.i.d observations \(y=(y_1, y_2, \dots, y_{n})\) and the likelihood is \(y_i|\lambda\sim\operatorname{Possion}(\lambda)\). If the prior for \(\lambda\) is \(\lambda\sim\operatorname{Gamma}(\alpha,\beta)\), then the posterior distribution of \(\lambda\) is:
\[\begin{equation*} \lambda|y\sim\operatorname{Gamma}\left(\alpha+n\bar{y},\beta+n\right) \end{equation*}\]
证明. \(p(\lambda|y)\propto p(\lambda)p(y|\lambda) \propto\lambda^{\alpha-1}e^{-\beta\lambda}\lambda^{n\bar{y}}e^{-n\lambda}=\lambda^{\alpha+n\bar{y}-1}e^{-(\beta+n)\lambda}\). ◻
13.3 Prior Distribution
13.3.1 Conjugate Prior Distribution
Definition 13.8. If \(\mathcal{F}\) is a class of sampling distributions \(p(y|\theta)\), and \(\mathcal{P}\) is a class of prior distributions for \(\theta\), then the class \(\mathcal{P}\) is conjugate for \(\mathcal{F}\) if:
\[\begin{equation*} \forall\;p(\cdot|\theta)\in\mathcal{F}\;and\;p(\cdot)\in\mathcal{P},\;p(\theta|y)\in\mathcal{P} \end{equation*}\]
The set of all probability functions that share the same functional form as the likelihood is referred to as the natural conjugate prior families.
13.3.2 Noninformative Prior Distribution
13.3.2.1 Jeffreys’ Invariance Principle
The Jeffreys’ invariance principle posits that the prior distribution should remain invariant when subjected to a transformation of the coordinate system for the parameter vector.
Definition 13.9. In accordance with Jeffreys’ invariance principle, the prior distribution for the parameter vector \(\theta\) is proportional to \(|\det[I(\theta)]|^{\frac{1}{2}}\).
13.3.2.2 Prior Distribution for the Location-Scale Family
Theorem 13.9. The location parameter can be conceptualized as the particular instantiation of the same parameter across various origins. Consequently, it is posited that the prior distribution of the location parameter ought not to be influenced by the selection of origin. In accordance with this principle, the prior distribution for the location parameter should be formulated as follows:
\[\begin{equation*} p(\theta)\propto1 \end{equation*}\]
Similarly, the scale parameter can be interpreted as the specific realization of the same parameter across different scales, so we believe that the prior distribution of the scale parameter should not depend on the choice of scale. In alignment with this principle, the prior distribution for the scale parameter should be articulated as follows:
\[\begin{equation*} p(\theta)\propto\frac{1}{\theta} \end{equation*}\]
证明. Initially, we will assume that \(\theta\) serves as a location parameter. In accordance with the aforementioned principle, we can express the following relationship:
\[\begin{equation*} p(\theta) = p(\theta + \alpha), \;\forall\; \alpha \in \mathbb{R}^{} \end{equation*}\]
This implies that \(p(\theta)\) must be a constant across the real line.
Next, we consider the case of a scale parameter, where we define \(\tau = \dfrac{\theta}{\alpha}\), with \(\alpha > 0\). Referring to 随机变量函数的分布, we need to establish the following equation:
\[\begin{equation*} p(\tau) = p(\alpha \tau) \left| \frac{\mathop{}\!\mathrm{d}\theta}{\mathop{}\!\mathrm{d}\tau} \right| = p(\alpha \tau) \alpha = p(\theta) \end{equation*}\]
From this, we derive:
\[\begin{equation*} \frac{p(\alpha \tau)}{p(\theta)} = \frac{1}{\alpha} \end{equation*}\]
This leads us to conclude that:
\[\begin{equation*} p(\theta) \propto \frac{1}{\theta} \end{equation*}\]
◻
13.4 贝叶斯统计推断
Definition 13.10. 基于后验分布的统计推断就只考虑已出现的数据，认为未出现的数据与统计推断无关，这种观点被称为条件观点。
note 13.1. 由于条件观点，在贝叶斯角度下的点估计不使用无偏性衡量统计量的好坏（无偏性要求统计量在所有可能出现的样本意义下以待估量真值为期望）。
13.4.1 贝叶斯点估计
Definition 13.11. 用后验密度\(p(\theta|x)\)达到最大值时\(\theta\)的值作为\(\theta\)的估计量，称为\(\theta\)的后验众数估计或广义最大似然估计，记为\(\hat{\theta}_{MD}\)。用后验分布的中位数作为\(\theta\)的估计量，称为后验中位数估计，记为\(\hat{\theta}_{ME}\)。用后验分布的期望值作为\(\theta\)的估计量，称为后验期望估计，记为\(\hat{\theta}_E\)。