第 13 章 统计计算方法

CHAPTER STATUS编写进度10%查看全书进度

13.1 非线性方程的求解

13.1.0.1 二分法

Method 13.1. (Bisection Method)

设函数\(f\)在\([a,b]\)上连续,且满足\(f(a)f(b)<0\)。为求解非线性方程\(f(x)=0\),由引理 3.3可知,存在至少一个根\(x^{\star}\in(a,b)\)。二分法通过反复对区间进行二等分来逐步缩小根所在的区间:令:

\[\begin{equation*} c=\frac{a+b}{2} \end{equation*}\]

若\(f(c)=0\),则\(c\)即为方程的解;否则根据符号判定准则,取满足:

\[\begin{equation*} f(a)f(c)<0\quad\text{或}\quad f(c)f(b)<0 \end{equation*}\]

的子区间作为新的区间\([a,b]\),并重复上述过程。

经过\(n\)次迭代后,所得区间长度不超过\((b-a)/2^n\),任意迭代点\(x_n\)与真根\(x^{\star}\)的误差满足:

\[\begin{equation*} |x_n-x^{\star}|\leqslant\frac{b-a}{2^n} \end{equation*}\]

因此,可通过给定误差容许值\(\varepsilon\),选取满足:

\[\begin{equation*} \frac{b-a}{2^n}\leqslant\varepsilon \end{equation*}\]

的最小整数\(n\)来控制计算精度。

13.1.0.2 不动点迭代法

Method 13.2. (Fixed-point Iteration Method)

为求解非线性方程\(f(x)=0\),可将其改写为等价的形式\(x=T(x)\),若\(T\)是一个压缩映射,由定理 3.20可求得非线性方程的根。对于迭代次数问题,可用定理 3.20中的第二种误差估计公式来控制精度。

Definition 13.1. 设映射\(T\)有不动点\(x^{\star}\)。若存在\(x^{\star}\)的邻域\(\bar{U}(x^{\star},\delta)\)满足对任意的\(x\in\bar{U}(x^{\star},\delta)\),序列\(\{x_n=T^nx\}\to x^{\star}\),则称此时的迭代法局部收敛(local convergence)。

Theorem 13.1. 设映射\(T\)有不动点\(x^{\star}\),\(T'\)在\(x^{\star}\)的某个邻域内连续,且\(|T'x^{\star}|<1\),则此时的迭代法局部收敛。

证明. 由条件和性质 3.2.9(5)可知存在\(x^{\star}\)的邻域\(U\)和\(L\),使得对任意的\(x\in U\)有\(|T'x|\leqslant L<1\)。根据定理 4.3可知对任意的\(x'\in U\)有:

\[\begin{equation*} |Tx-Tx^{\star}|\leqslant L|x-x^{\star}| \end{equation*}\]

即\(T\)在\(x^{\star}\)的某个闭邻域内是压缩映射,由定理 3.8、性质 3.2.6(1)和定理 3.20可知此时的迭代法局部收敛。 ◻

Definition 13.2. 设迭代过程\(x_{n+1}=Tx_n\)收敛于\(T\)的不动点\(x^{\star}\)。令\(\varepsilon_n=x_n-x^{\star}\),若有:

\[\begin{equation*} \lim_{n\to+\infty}\frac{\varepsilon_{n+1}}{\varepsilon_n^p}=C\in\mathbb{R}\setminus\{0\} \end{equation*}\]

则称该迭代过程是\(p\)阶收敛的。特别的,\(p=1\)且\(|C|<1\)时称为线性收敛(linear convergence),\(p>1\)时称为超线性收敛(superlinear convergence)。

Theorem 13.2. 设迭代过程\(x_{n+1}=Tx_n\)。若对于\(p\in\mathbb{N}^+\),\(T^{(p)}\)在\(T\)的不动点\(x^{\star}\)的某个邻域内连续,并且有:

\[\begin{equation*} T'x^{\star}=T''x^{\star}=\cdots=T^{(p-1)}x^{\star}=0,\;T^{(p)}x^{\star}\ne0 \end{equation*}\]

则该迭代过程在\(x^{\star}\)邻近是\(p\)阶收敛的。

证明. 由\(T'x^{\star}=0\)和定理 13.1可知该迭代过程是局部收敛的。根据泰勒展开Lagrange余项可得:

\[\begin{equation*} Tx_n=Tx^{\star}+\frac{T^{(p)}\xi_n}{p!}(x_n-x^{\star})^p,\quad\xi_n\text{在$x_n$与$x^{\star}$之间} \end{equation*}\]

所以:

\[\begin{equation*} \frac{Tx_n-x^{\star}}{(x_n-x^{\star})^p}=\frac{T^{(p)}\xi_n}{p!} \end{equation*}\]

由\(T^{(p)}\)在\(x^{\star}\)某个邻域内的连续性和性质 3.2.9(8.c)(4.a)可得:

\[\begin{equation*} \lim_{n\to+\infty}\frac{Tx_n-x^{\star}}{(x_n-x^{\star})^p}=\lim_{n\to+\infty}\frac{T^{(p)}\xi_n}{p!}=\frac{1}{p!}T^{(p)}\left(\lim_{n\to+\infty}\xi_n\right)=\frac{T^{(p)}x^{\star}}{p!} \end{equation*}\]

◻

13.1.0.3 牛顿法

Method 13.3. (Newton Method)

设\(f\)在根\(x^{\star}\)的某个邻域内是\(C^2\)类函数,且满足\(f(x^{\star})=0,\;f'(x^{\star})\ne0\)。为求解非线性方程\(f(x)=0\),牛顿法对\(f\)在当前迭代点\(x_n\)处作一阶Taylor展开,得到:

\[\begin{equation*} f(x)\approx f(x_n)+f'(x_n)(x-x_n) \end{equation*}\]

令上式为\(0\)得到迭代格式为:

\[\begin{equation*} x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)} \end{equation*}\]

牛顿法的迭代函数为:

\[\begin{equation*} Tx=x-\frac{f(x)}{f'(x)} \end{equation*}\]

于是有:

\[\begin{equation*} T'x=1-\frac{[f'(x)]^2-f(x)f''(x)}{[f'(x)]^2}=\frac{f(x)f''(x)}{[f'(x)]^2} \end{equation*}\]

根据定理 13.2可知牛顿法至少是二阶收敛的。

13.2 插值法

Definition 13.3. 设函数\(f\)在\([a,b]\)上有定义,且已知在点\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)上的值\(y_0,y_1,\dots,y_n\)。若存在一个简单函数\(P(x)\)使得:

\[\begin{equation*} P(x_i)=y_i,\quad i=0,1,\dots,n \end{equation*}\]

则称\(P\)为\(f\)的插值函数(interpolating function),点\(x_0,x_1,\dots,x_n\)被称为插值节点(interpolation nodes),\([a,b]\)为插值区间(interpolation interval),求插值函数\(P\)的方法称为插值法(interpolation method)。若\(P\)是不超过\(n\)次的多项式,即:

\[\begin{equation*} P(x)=\sum_{i=0}^{n}a_ix^i,\quad a_i\in\mathbb{R},\;i=0,1,\dots,n \end{equation*}\]

则称\(P\)为插值多项式(interpolation polynomial),对应的插值法被称为多项式插值(polynomial interpolation)。若\(P\)是分段的多项式,则称对应的插值法为分段插值(piecewise interpolation)。

Theorem 13.3. 设函数\(f\)在\([a,b]\)上有定义,且已知在点\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)上的值\(y_0,y_1,\dots,y_n\)。\(f\)在\([a,b]\)上次数不超过\(n\)的插值多项式存在且唯一。

证明. 设\(P(x)=\sum\limits_{i=0}^{n}a_ix^i\),则由条件可得到:

\[\begin{equation*} \begin{pmatrix} 1 & x_0 & x_0^2 & \cdots & x_0^n \\ 1 & x_1 & x_1^2 & \cdots & x_1^n \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^n \end{pmatrix} \begin{pmatrix} a_0 \\ a_1 \\ \vdots \\ a_n \end{pmatrix} = \begin{pmatrix} y_0 \\ y_1 \\ \vdots \\ y_n \end{pmatrix} \end{equation*}\]

根据性质 2.1.7(13)、性质 1.1.3(1)和定理 2.4(2)可知上述线性方程组的解存在且唯一,即\(f\)在\([a,b]\)上次数不超过\(n\)的插值多项式存在且唯一。 ◻

13.2.0.1 Lagrange插值多项式

Definition 13.4. 若\(n\)次多项式\(l_i(x),\;i=0,1,\dots,n\)在\(n+1\)个节点\(x_0<x_1<\cdots<x_n\)上满足:

\[\begin{equation*} l_i(x_j)= \begin{cases} 1,& i=j \\ 0,& i\ne j \end{cases},\quad i,j=0,1,\dots,n \end{equation*}\]

则称\(l_0(x),l_1(x),\dots,l_n(x)\)为节点\(x_0,x_1,\dots,x_n\)上的\(n\)次插值基函数(interpolation basis functions)。

Definition 13.5. 设函数\(f\)在\([a,b]\)上有定义,且已知在点\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)上的值\(y_0,y_1,\dots,y_n\)。定义:

\[\begin{equation*} l_i(x)=\frac{(x-x_0)(x-x_1)\dots(x-x_{i-1})(x-x_{i+1})\dots(x-x_n)}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)},\quad i=0,1,\dots,n \end{equation*}\]

称:

\[\begin{equation*} L_n(x)\coloneq\sum_{i=0}^{n}y_il_i(x) \end{equation*}\]

为\(f\)在节点\(x_0,x_1,\dots,x_n\)上的拉格朗日插值多项式(Lagrange interpolation polynomial)。

Property 13.2.1. 设函数\(f\)在\([a,b]\)上有定义,且已知在点\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)上的值\(y_0,y_1,\dots,y_n\),\(f\in C^n([a,b])\),\(f^{(n+1)}\)在\((a,b)\)上存在,\(L_n\)为\(f\)在节点\(x_0,x_1,\dots,x_n\)上的Lagrange插值多项式。\(L_n\)具有如下性质:

  1. 记\(w_{n+1}(x)=(x-x_0)(x-x_1)\cdots(x-x_n)\),则:

    \[\begin{equation*} L_n(x)=\sum_{i=0}^{n}y_i\frac{w_{n+1}(x)}{(x-x_i)w_{n+1}'(x_i)} \end{equation*}\]

  2. 对于任意的\(x\in[a,b]\),有误差估计:

    \[\begin{equation*} f(x)-L_n(x)=\frac{f^{(n+1)}(\xi)}{(n+1)!}w_{n+1}(x) \end{equation*}\]

    其中\(\xi\in(a,b)\);

  3. \(\sum\limits_{i=0}^{n}x_i^ml_i(x)=x^m(m\leqslant n)\);

  4. \(L_n\)有如下形式:

    \[\begin{equation*} L_n(x)=\frac{\sum\limits_{i=0}^{n}y_i\dfrac{1}{(x-x_i)w_{n+1}'(x_i)}}{\sum\limits_{i=0}^{n}\dfrac{1}{(x-x_i)w_{n+1}'(x_i)}} \end{equation*}\]

证明. (1)由乘积的微分可知:

\[\begin{equation*} w_{n+1}'(x_i)=(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n) \end{equation*}\]

于是:

\[\begin{equation*} l_i(x)=\frac{w_{n+1}(x)}{(x-x_i)w_{n+1}'(x_i)} \end{equation*}\]

(2)由条件可设\(f(x)-L_n(x)=K(x)w_{n+1}(x)\),其中\(K(x)\)是一个待定函数。作辅助函数:

\[\begin{equation*} \varphi(t)=f(t)-L_n(t)-K(x)w_{n+1}(t) \end{equation*}\]

由性质 3.3.3(5.b)和性质 4.1.1(4)可知\(\varphi^{(n)}\)在\([a,b]\)上连续,\(\varphi^{(n+1)}\)在\((a,b)\)上存在。因为\(\varphi\)在\(x_0,x_1,\dots,x_n\)和\(x\)处均为\(0\),所以\(\varphi\)在\([a,b]\)上有\(n+2\)个零点,根据性质 3.3.3(5.b)、性质 4.1.1(4)和定理 4.2可知\(\varphi'\)在\(\varphi\)的两个零点间至少存在一个零点,所以\(\varphi'\)在\([a,b]\)上至少存在\(n+1\)个零点。对\(\varphi'\)再使用定理 4.2可知\(\varphi''\)在\([a,b]\)上至少存在\(n\)个零点。依此类推可知\(\varphi^{(n+1)}\)在\((a,b)\)上至少存在一个零点\(\xi\),即:

\[\begin{equation*} \varphi^{(n+1)}(\xi)=f^{(n+1)}(\xi)-K(x)(n+1)!=0 \end{equation*}\]

于是:

\[\begin{equation*} K(x)=\frac{f^{(n+1)}(\xi)}{(n+1)!} \end{equation*}\]

即:

\[\begin{equation*} f(x)-L_n(x)=\frac{f^{(n+1)}(\xi)}{(n+1)!}w_{n+1}(x) \end{equation*}\]

(3)设\(f=x^m\),由(2)即可得出结论。

(4)在(3)中取\(m=0\),由(1)(3)可得:

\[\begin{equation*} 1=\sum_{i=0}^{n}l_i(x)=\sum_{i=0}^{n}\frac{w_{n+1}(x)}{(x-x_i)w_{n+1}'(x_i)}=w_{n+1}(x)\sum_{i=0}^{n}\frac{1}{(x-x_i)w_{n+1}'(x_i)} \end{equation*}\]

所以:

\[\begin{equation*} w_{n+1}(x)=\frac{1}{\sum\limits_{i=0}^{n}\dfrac{1}{(x-x_i)w_{n+1}'(x_i)}} \end{equation*}\]

于是有:

\[\begin{align*} L_n(x)&=\sum_{i=0}^{n}y_i\frac{w_{n+1}(x)}{(x-x_i)w_{n+1}'(x_i)}=w_{n+1}(x)\sum_{i=0}^{n}y_i\frac{1}{(x-x_i)w_{n+1}'(x_i)} \\ &=\frac{\sum\limits_{i=0}^{n}y_i\dfrac{1}{(x-x_i)w_{n+1}'(x_i)}}{\sum\limits_{i=0}^{n}\dfrac{1}{(x-x_i)w_{n+1}'(x_i)}} \end{align*}\]

◻

note 13.1. 性质 13.2.1(4)中\(L_n\)的形式又被称为Barycentric形式。该表示形式在数学上与拉格朗日插值多项式完全等价,但在数值计算中避免了高次多项式乘积的显式计算,即计算:

\[\begin{equation*} w_{n+1}(x)=(x-x_0)(x-x_1)\cdots(x-x_n) \end{equation*}\]

从而减少了由大数乘积和相消所引起的数值误差,具有显著更优的稳定性。基于上述原因,在实际数值计算中通常优先采用Barycentric形式来实现多项式插值。

    import numpy as np
    from scipy.interpolate import BarycentricInterpolator
    x_nodes = np.array([0.0,1.0,2.0,3.0])
    y_nodes = np.array([1.0,2.0,0.0,5.0])
    interp = BarycentricInterpolator(x_nodes,y_nodes)
    print(interp(1.5))

13.2.0.2 Newton插值多项式

Definition 13.6. 设函数\(f\)在\([a,b]\)上有定义,\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)。记\(f[x_0]=f(x_0)\),称:

\[\begin{equation*} f[x_0,x_n]=\frac{f(x_n)-f(x_0)}{x_n-x_0} \end{equation*}\]

为\(f\)关于点\(x_0,x_n\)的一阶均差(first-order divided difference)。称:

\[\begin{equation*} f[x_0,x_1,\dots,x_n]=\frac{f[x_1,x_2,\dots,x_n]-f[x_0,x_1,\dots,x_{n-1}]}{x_n-x_0} \end{equation*}\]

为\(f\)的n阶均差(n-th order divided difference)。

Property 13.2.2. 设函数\(f\)在\([a,b]\)上有定义,\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)。均差具有如下性质:

  1. \(f\)的\(n\)阶均差可以表示为\(f(x_0),f(x_1),\dots,f(x_n)\)的线性组合:

    \[\begin{equation*} f[x_0,x_1,\dots,x_n]=\sum_{i=0}^{n}\frac{f(x_i)}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)} \end{equation*}\]

  2. \(f[x_0,x_1,\dots,x_n]\)对\(\{x_0,x_1,\dots,x_n\}\)的任意顺序置换保持不变;

  3. 对任意的\(\alpha,\beta\in\mathbb{R}\)和\([a,b]\)上的函数\(g\),有:

    \[\begin{equation*} (\alpha f+\beta g)[x_0,x_1,\dots,x_n]=\alpha f[x_0,x_1,\dots,x_n]+\beta g[x_0,x_1,\dots,x_n] \end{equation*}\]

  4. 若\(f\in C^n([a,b])\),则存在\(\xi\in[a,b]\)使得:

    \[\begin{equation*} f[x_0,x_1,\dots,x_n]=\frac{f^{(n)}(\xi)}{n!} \end{equation*}\]

证明. (1)使用数学归纳法。当\(n=1\)时有:

\[\begin{equation*} f[x_0,x_1]=\frac{f(x_1)-f(x_0)}{x_1-x_0}=\frac{f(x_0)}{x_0-x_1}+\frac{f(x_1)}{x_1-x_0} \end{equation*}\]

结论成立。设结论对\(n-1\)成立,下面证明结论对\(n\)也成立。

根据归纳假设:

\[\begin{align*} &f[x_0,x_1,\dots,x_n]=\frac{f[x_1,x_2,\dots,x_n]-f[x_0,x_1,\dots,x_{n-1}]}{x_n-x_0} \\ =&\frac{1}{x_n-x_0}\left[\sum_{i=1}^{n}\frac{f(x_i)}{(x_i-x_1)(x_i-x_2)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)}\right. \\ &\left.-\sum_{i=0}^{n-1}\frac{f(x_i)}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_{n-1})}\right] \\ =&\frac{f(x_n)}{(x_n-x_0)(x_n-x_1)\dots(x_n-x_{n-1})}+\frac{f(x_0)}{(x_0-x_1)(x_0-x_2)\dots(x_0-x_n)} \\ &+\sum_{i=1}^{n-1}\left[\frac{f(x_i)}{(x_n-x_0)(x_i-x_1)(x_i-x_2)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)}\right. \\ &\left.-\frac{f(x_i)}{(x_n-x_0)(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_{n-1})}\right] \end{align*}\]

对于最后一项:

\[\begin{align*} &\sum_{i=1}^{n-1}\left[\frac{(x_i-x_0)f(x_i)}{(x_n-x_0)(x_i-x_0)(x_i-x_1)(x_i-x_2)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)}\right. \\ &\left.-\frac{(x_i-x_n)f(x_i)}{(x_n-x_0)(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_{n-1})(x_i-x_n)}\right] \\ =&\sum_{i=1}^{n-1}\frac{(x_i-x_0)f(x_i)-(x_i-x_n)f(x_i)}{(x_n-x_0)(x_i-x_0)(x_i-x_1)(x_i-x_2)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)} \\ =&\sum_{i=1}^{n-1}\frac{(x_n-x_0)f(x_i)}{(x_n-x_0)(x_i-x_0)(x_i-x_1)(x_i-x_2)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)} \\ =&\sum_{i=1}^{n-1}\frac{f(x_i)}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)} \end{align*}\]

所以结论成立。

(2)由(1)立即可得。

(3)由(1)立即可得。

(4)令\(L_{n-1}\)为\(f\)在节点\(x_0,x_1,\dots,x_{n-1}\)上的Lagrange插值多项式,作辅助函数:

\[\begin{equation*} \varphi(x)=f(x)-L_{n-1}(x)-f[x_0,x_1,\dots,x_n]w_n(x) \end{equation*}\]

由\(L_{n-1}\)的定义可知\(f-L_{n-1}\)在节点\(x_0,x_1,\dots,x_{n-1}\)上为\(0\),而\(w_n\)在节点\(x_0,x_1,\dots,x_{n-1}\)上为\(0\),所以\(x_0,x_1,\dots,x_{n-1}\)是\(\varphi\)的零点。注意到:

\[\begin{align*} &f(x_n)-L_{n-1}(x_n)-f[x_0,x_1,\dots,x_n]w_n(x_n) \\ =&f(x_n)-\sum_{i=0}^{n-1}\frac{f(x_i)(x_n-x_0)(x_n-x_1)\dots(x_n-x_{i-1})(x_n-x_{i+1})\dots(x_n-x_{n-1})}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_{n-1})} \\ &-\sum_{i=0}^{n}\frac{f(x_i)(x_n-x_0)(x_n-x_1)\cdots(x_n-x_{n-1})}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)} \\ =&f(x_n)-\sum_{i=0}^{n-1}\frac{f(x_i)(x_n-x_0)(x_n-x_1)\dots(x_n-x_{i-1})(x_n-x_{i+1})\dots(x_n-x_{n-1})}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_{n-1})} \\ &-f(x_n)-\sum_{i=0}^{n-1}\frac{f(x_i)(x_n-x_0)(x_n-x_1)\cdots(x_n-x_{n-1})}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)} \\ =&-\sum_{i=0}^{n-1}\frac{f(x_i)(x_n-x_0)(x_n-x_1)\dots(x_n-x_{n-1})}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_{n-1})(x_n-x_i)} \\ &-\sum_{i=0}^{n-1}\frac{f(x_i)(x_n-x_0)(x_n-x_1)\cdots(x_n-x_{n-1})}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)}=0 \end{align*}\]

所以\(x_n\)也是\(\varphi\)的零点。根据性质 3.3.3(5.b)、性质 4.1.1(4)和定理 4.2可知\(\varphi'\)在\([a,b]\)上存在\(n\)个零点。对\(\varphi'\)再使用定理 4.2可知\(\varphi''\)在\([a,b]\)上存在\(n-1\)个零点。依此类推可知\(\varphi^{(n)}\)在\([a,b]\)上存在一个零点\(\xi\),即:

\[\begin{equation*} \varphi^{(n)}(\xi)=f^{(n)}(\xi)-f[x_0,x_1,\dots,x_n]n!=0 \end{equation*}\]

所以:

\[\begin{equation*} f[x_0,x_1,\dots,x_n]=\frac{f^{(n)}(\xi)}{n!} \end{equation*}\]

◻

Definition 13.7. 设函数\(f\)在\([a,b]\)上有定义,且已知在点\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)上的值\(y_0,y_1,\dots,y_n\)。称:

\[\begin{equation*} P_n(x)=f[x_0]+f[x_0,x_1](x-x_0)+f[x_0,x_1,x_2](x-x_0)(x-x_1)+\cdots+f[x_0,x_1,\dots,x_n]\prod_{i=0}^{n-1}(x-x_i) \end{equation*}\]

为\(f\)在节点\(x_0,x_1,\dots,x_n\)上的牛顿插值多项式(Newton interpolation polynomial)。

Property 13.2.3. Newton插值多项式与Lagrange插值多项式等价。

证明. 设函数\(f\)在\([a,b]\)上有定义,且已知在点\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)上的值\(y_0,y_1,\dots,y_n\)。因为\(f\)在节点\(x_0,x_1,\dots,x_n\)上的Newton插值多项式\(P_n\)的次数不超过\(n\),根据定理 13.3只需证明\(P_n(x_i)=f(x_i),\;i=0,1,\dots,n\)。

使用数学归纳法。当\(i=0\)时由\(P_n\)的定义可知\(P_n(x_0)=f(x_0)\),设结论对\(n-1\)成立,下面证明结论对\(n\)也成立。

对于:

\[\begin{equation*} P_n(x)=P_{n-1}(x)+f[x_0,x_1,\dots,x_n]w_n(x) \end{equation*}\]

因为对任意的\(i=0,1,\dots,n-1\)有\(w_n(x_i)=0\),所以由归纳假设可知:

\[\begin{equation*} P_n(x_i)=P_{n-1}(x_i)=f(x_i),\quad i=0,1,\dots,n-1 \end{equation*}\]

下面证明\(P_n(x_n)=f(x_n)\)。

作辅助函数\(R(x)=f(x)-P_{n-1}(x)\),于是有:

\[\begin{equation*} \frac{R(x_n)}{w_n(x_n)}=\frac{f(x_n)-P_{n-1}(x_n)}{w_n(x_n)} \end{equation*}\]

由性质 13.2.2(1)可得:

\[\begin{equation*} R[x_0,x_1,\dots,x_n]=\sum_{i=0}^{n}\frac{R(x_i)}{(x_i-x_0)(x_i-x_1)\dots(x_i-x_{i-1})(x_i-x_{i+1})\dots(x_i-x_n)}=\frac{R(x_n)}{w_n(x_n)} \end{equation*}\]

所以:

\[\begin{equation*} \frac{f(x_n)-P_{n-1}(x_n)}{w_n(x_n)}=R[x_0,x_1,\dots,x_n] \end{equation*}\]

根据性质 13.2.2(3)可知:

\[\begin{equation*} R[x_0,x_1,\dots,x_n]=f[x_0,x_1,\dots,x_n]-P_{n-1}[x_0,x_1,\dots,x_n] \end{equation*}\]

因为\(P_{n-1}\)的次数不超过\(n-1\),由性质 13.2.2(4)可知\(P_{n-1}[x_0,x_1,\dots,x_n]=0\),于是有:

\[\begin{equation*} R[x_0,x_1,\dots,x_n]=f[x_0,x_1,\dots,x_n] \end{equation*}\]

所以:

\[\begin{equation*} \frac{f(x_n)-P_{n-1}(x_n)}{w_n(x_n)}=f[x_0,x_1,\dots,x_n] \end{equation*}\]

即:

\[\begin{equation*} P_{n-1}(x_n)=f(x_n)-f[x_0,x_1,\dots,x_n]w_n(x_n) \end{equation*}\]

于是:

\[\begin{equation*} P_n(x_n)=P_{n-1}(x_n)+f[x_0,x_1,\dots,x_n]w_n(x_n)=f(x_n) \end{equation*}\]

◻

note 13.2. 根据性质 13.2.3,Newton插值多项式和Lagrange插值多项式是等价的,于是它具有了Lagrange插值多项式的全部性质。既然等价,那为什么要研究它呢?

从构造方式看,Lagrange插值是一次性使用全部节点进行整体构造,而Newton插值具有天然的递推性。当新增插值节点时,Lagrange插值多项式通常需要重新计算,而Newton插值多项式只需增加一个新的均差项即可完成更新,这使其在逐点插值与在线计算中具有明显优势。

13.2.0.3 Hermite插值

Definition 13.8. 设函数\(f\)在\([a,b]\)上有定义,且已知\(f^{(j)}(x_i)\),\(i=0,1,\dots,n,\;j=0,1,\dots,m_i\),其中\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\)。令\(N=\sum\limits_{i=0}^{n}m_i-1\),称满足:

\[\begin{equation*} H^{(j)}_N(x_i)=f^{(j)}(x_i),\quad i=0,1,\dots,n,\;j=0,1,\dots,m_i \end{equation*}\]

的不超过\(N\)次的多项式\(H_N(x)\)为埃尔米特插值多项式(Hermite interpolation polynomial)。

Definition 13.9. 定义:

\[\begin{equation*} f[\underbrace{x,x,\dots,x}_{n+1\text{个}x}]\coloneq\frac{f^{(n)}(x)}{n!} \end{equation*}\]

Property 13.2.4. 设函数\(f\)在\([a,b]\)上有定义,且已知\(f^{(j)}(x_i)\),\(i=0,1,\dots,n,\;j=0,1,\dots,m_i\),其中\(a\leqslant x_0<x_1<\cdots<x_n\leqslant b\),\(N=\sum\limits_{i=0}^{n}m_i-1\)。Hermite插值多项式具有如下性质:

  1. Hermite插值多项式存在且唯一;

  2. 若将每个节点\(x_i\)重复\(m_i+1\)次,按重复顺序排列为\(y_0,y_1,\dots,y_M\),则:

    \[\begin{equation*} H_N(x)=f[y_0]+f[y_0,y_1](x-y_0)+\cdots+f[y_0,y_1,\dots,y_M]\prod_{i=0}^{M-1}(x-y_i) \end{equation*}\]

    import numpy as np
    from scipy.interpolate import KroghInterpolator
    # Hermite interpolation data
    # f(0) = 1, f'(0) = 0
    # f(1) = 2, f'(1) = -1
    x = np.array([0.0, 0.0, 1.0, 1.0])
    y = np.array([1.0, 0.0, 2.0, -1.0])
    # Hermite interpolation
    hermite_poly = KroghInterpolator(x, y)
    # evaluate
    x_eval = np.linspace(0.0, 1.0, 100)
    y_eval = hermite_poly(x_eval)
    print(hermite_poly(0.5))

13.3 变分法

13.3.1 贝叶斯统计中的一般变分法原理

Definition 13.10. 设\(\varphi,\psi\)是可测空间\((X,\mathscr{F})\)上的概率测度,且\(\varphi\ll\psi\)。由定理 5.33,取\(\varphi\)关于\(\psi\)的非负Radon-Nikodym导数:

\[\begin{equation*} r(x)\coloneq\dfrac{\mathop{}\!\mathrm{d}\varphi}{\mathop{}\!\mathrm{d}\psi}(x) \end{equation*}\]

在集合\(\{r=0\}\)上将\(\ln r\)定义为\(0\),并约定\(0\ln0=0\)。根据引理 5.10,称:

\[\begin{equation*} \operatorname{KL}(\varphi\|\psi)\coloneq\int_{X}\ln r(x)\mathop{}\!\mathrm{d}\varphi=\int_{X}r(x)\ln r(x)\mathop{}\!\mathrm{d}\psi \end{equation*}\]

为\(\varphi\)关于\(\psi\)的KL散度(Kullback–Leibler divergence)。

note 13.3. 设\(\mu\)是可测空间\((X,\mathscr{F})\)上的概率测度,我们希望求解\(\mu\)但是它很难求解,变分法(variational method)的思想就是取一族形式简单、便于计算的概率函数,并在这一族分布中找到一个最接近\(\mu\)的。由不等式 11可以看出KL散度可以被用来衡量分布之间的差异性。

在贝叶斯统计中,设\(f,g\)分别是从概率空间\((X,\mathscr{F},P)\)到可测空间\((Y,\mathscr{A})\)和\((Z,\mathscr{B})\)的可测映射,其中\(f\)表示需要推断的对象,\(g\)表示观测数据。设\(\mu,\nu\)分别是\((Y,\mathscr{A})\)和\((Z,\mathscr{B})\)上的\(\sigma\)有限测度,并假设:

\[\begin{equation*} P(f,g)^{-1}\ll\mu\times\nu \end{equation*}\]

根据定理 5.33和定理 5.38,取:

\[\begin{equation*} p_{f,g}=\dfrac{\mathop{}\!\mathrm{d}P(f,g)^{-1}}{\mathop{}\!\mathrm{d}(\mu\times\nu)},\quad p_g(z)=\dfrac{\mathop{}\!\mathrm{d}Pg^{-1}}{\mathop{}\!\mathrm{d}\nu}=\int_{Y}p_{f,g}(y,z)\mathop{}\!\mathrm{d}\mu \end{equation*}\]

由性质 6.3.6(1)和性质 6.3.5(2)可知\(f\)关于\(g\)相对于\(\mu\)的条件概率密度函数\(p_{f\mid g}\)存在并且:

\[\begin{equation*} \forall\;z\in Z,\;A\in\mathscr{A},\;K_{f\mid g}(z,A)=\int_Ap_{f\mid g}(y\mid z)\mathop{}\!\mathrm{d}\mu \end{equation*}\]

给出了\(f\)关于\(g\)的条件概率核。当实际观测到\(g=z_0\)时,我们希望确定\(f\)在该观测下的条件概率分布\(K_{f\mid g}(z_0,\cdot)\)。

固定满足\(0<p_g(z_0)<+\infty\)的观测值\(z_0\in Z\)(由性质 5.4.2(8)和性质 5.2.1(3)中的次有限可加性可知这一假设并不构成实质上的限制,只是排除了一些异常情况),记后验概率测度为:

\[\begin{equation*} \forall\;A\in\mathscr{A},\;\varphi_{z_0}(A)=K_{f\mid g}(z_0,A) =\int_{A}p_{f\mid g}(y\mid z_0)\mathop{}\!\mathrm{d}\mu \end{equation*}\]

由性质 5.4.3(1)可知\(\varphi_{z_0}\ll\mu\),此时有:

\[\begin{equation*} p_{f\mid g}(y\mid z_0)=\frac{p_{f,g}(y,z_0)}{p_g(z_0)} \end{equation*}\]

设\(\psi_q\)是\((Y,\mathscr{A})\)上的概率测度,满足\(\psi_q\ll\mu\),根据定理 5.33,记:

\[\begin{equation*} q(y)=\dfrac{\mathop{}\!\mathrm{d}\psi_q}{\mathop{}\!\mathrm{d}\mu}(y) \end{equation*}\]

若\(\psi_q\ll\varphi_{z_0}\)且\(\operatorname{KL}(\psi_q\|\varphi_{z_0})<+\infty\),则由性质 5.6.3(5)、引理 5.10和性质 5.4.3(6)可得:

\[\begin{align*} &\operatorname{KL}(\psi_q\|\varphi_{z_0})=\int_{Y}\ln\frac{\mathop{}\!\mathrm{d}\psi_q}{\mathop{}\!\mathrm{d}\varphi_{z_0}}(y)\mathop{}\!\mathrm{d}\psi_q=\int_{Y}\ln\frac{q(y)}{p_{f\mid g}(y\mid z_0)}\mathop{}\!\mathrm{d}\psi_q=\int_{Y}q(y)\ln\frac{q(y)}{p_{f\mid g}(y\mid z_0)}\mathop{}\!\mathrm{d}\mu \\ =&\int_{Y}q(y)\ln\frac{q(y)p_g(z_0)}{p_{f,g}(y,z_0)}\mathop{}\!\mathrm{d}\mu=\ln p_g(z_0)-\int_{Y}q(y)\ln\frac{p_{f,g}(y,z_0)}{q(y)}\mathop{}\!\mathrm{d}\mu \end{align*}\]

Definition 13.11. 在上述条件下,称:

\[\begin{equation*} \mathcal{L}(q;z_0)\coloneq\int_{Y}q(y)\ln\frac{p_{f,g}(y,z_0)}{q(y)}\mathop{}\!\mathrm{d}\mu \end{equation*}\]

为给定观测\(z_0\)时关于\(q\)的证据下界(evidence lower bound, ELBO)。

由上述推导可知:

\[\begin{equation*} \ln p_g(z_0)=\mathcal{L}(q;z_0)+\operatorname{KL}(\psi_q\|\varphi_{z_0}) \end{equation*}\]

实际计算中,变分法(variational method)预先选定一族便于计算的概率密度函数\(\mathscr{Q}\),并求解:

\[\begin{equation*} q^{*}\in\underset{q\in\mathscr{Q}}{\arg\max}\;\mathcal{L}(q;z_0) \end{equation*}\]

因为\(\ln p_g(z_0)\)与\(q\)无关,所以最大化\(\mathcal{L}(q;z_0)\)等价于在\(\{\psi_q:q\in\mathscr{Q}\}\)中最小化\(\operatorname{KL}(\psi_q\|\varphi_{z_0})\)。因此\(q^{*}\)所对应的概率测度是后验概率测度\(\varphi_{z_0}\)在\(\{\psi_q:q\in\mathscr{Q}\}\)中的最优近似。

13.3.2 EM算法

设\((X,\mathscr{F},\mathscr{P})\)是参数统计结构,其中\(\mathscr{P}=\{P_{\theta}:\theta\in\Theta\}\)。设\(f\)是从\((X,\mathscr{F})\)到可测空间\((Y,\mathscr{A})\)的可测映射,表示能够观测到的数据,\(\mu\)是\((Y,\mathscr{A})\)上的\(\sigma\)有限测度,对任意的\(\theta\in\Theta\)有\(P_{\theta}f^{-1}\ll\mu\)。根据定理 5.33,取非负的:

\[\begin{equation*} p_{\theta;f}=\dfrac{\mathop{}\!\mathrm{d}P_{\theta}f^{-1}}{\mathop{}\!\mathrm{d}\mu} \end{equation*}\]

当观测到\(f=y\)时,极大似然估计需要求解:

\[\begin{equation*} \hat{\theta}\in\underset{\theta\in\Theta}{\arg\max}\;\ln p_{\theta;f}(y) \end{equation*}\]

如果\(p_{\theta;f}(y)\)能够直接写出,并且容易关于\(\theta\)进行优化,那么可以直接采用通常的极大似然估计方法。然而在许多模型中,实际观测到的\(f\)并不足以直接给出完整模型,因而\(p_{\theta;f}(y)\)往往难以直接写出1。

为了补全产生观测数据的模型结构,我们寻找一个从\((X,\mathscr{F})\)到另一可测空间\((Z,\mathscr{B})\)的可测映射\(g\),使得\((f,g)\)能够给出完整模型,而实际观测中仅能获得其中的\(f\)。这样的\(g\)称为隐变量(latent variable),\((f,g)\)称为完整数据。

设\(\nu\)是\((Z,\mathscr{B})\)上的\(\sigma\)有限测度,并假设对任意的\(\theta\in\Theta\)有\(P_{\theta}(f,g)^{-1}\ll\mu\times\nu\)。由定理 5.33,取非负的:

\[\begin{equation*} p_{\theta;f,g}(y,z)=\frac{\mathop{}\!\mathrm{d}P_{\theta}(f,g)^{-1}}{\mathop{}\!\mathrm{d}(\mu\times\nu)}(y,z) \end{equation*}\]

根据定理 5.38可知:

\[\begin{equation*} p_{\theta;f}(y)=\frac{\mathop{}\!\mathrm{d}P_{\theta}f^{-1}}{\mathop{}\!\mathrm{d}\mu}(y)=\int_{Z}p_{\theta;f,g}(y,z)\mathop{}\!\mathrm{d}\nu \end{equation*}\]

于是极大似然估计所面对的困难具体表现为对数内含有一个关于隐变量的积分:

\[\begin{equation*} \ln p_{\theta;f}(y)=\ln\int_{Z}p_{\theta;f,g}(y,z)\mathop{}\!\mathrm{d}\nu \end{equation*}\]

固定满足\(0<p_{\theta;f}(y)<+\infty\)的参数\(\theta\),这一条件保证对数似然\(\ln p_{\theta;f}(y)\)是良定义的有限实数。由性质 6.3.6(1)和性质 6.3.5(2)可知\(g\)关于\(f\)相对于\(\nu\)的条件概率密度函数\(p_{\theta;g\mid f}\)和\(g\)关于\(f\)的条件概率核\(K_{\theta;g\mid f}\)都存在。取\((Z,\mathscr{B})\)上关于\(\nu\)绝对连续的辅助概率测度\(\psi_q\),根据变分法部分的讨论可知:

\[\begin{equation*} \ln p_{\theta;f}(y)=\mathcal{L}(q,\theta;y)+\operatorname{KL}[\psi_q\mid K_{\theta;g\mid f}(y,\cdot)] \end{equation*}\]

其中:

\[\begin{equation*} q(z)=\frac{\mathop{}\!\mathrm{d}\psi_q}{\mathop{}\!\mathrm{d}\nu}(z) \end{equation*}\]

由不等式 11可知\(\operatorname{KL}[\psi_q\mid K_{\theta;g\mid f}(y,\cdot)]\geqslant0\),所以:

\[\begin{equation*} \ln p_{\theta;f}(y)\geqslant\mathcal{L}(q,\theta;y) \end{equation*}\]

等号成立当且仅当\(\psi_q=K_{\theta;g\mid f}(y,\cdot)\)。

由于证据下界\(\mathcal{L}(q,\theta;y)\)关于参数\(\theta\)通常比原始对数似然\(\ln p_{\theta;f}(y)\)更容易优化,因此可以利用辅助概率测度\(\psi_q\),将原本直接作用于边缘对数似然的参数优化问题转化为关于辅助分布与模型参数的交替优化问题。在给定当前参数时,将\(\psi_q\)取为\(K_{\theta;g\mid f}(y,\cdot)\),使证据下界与当前对数似然相等;随后固定\(\psi_q\),仅关于\(\theta\)优化证据下界。通过交替执行这两个步骤,可以构造一个使\(\ln p_{\theta;f}(y)\)单调不减的参数迭代过程。

Method 13.4. 给定观测值\(y\)和初始参数\(\theta^{(0)}\),并假设在迭代过程中涉及的参数均有:

\[\begin{equation*} 0<p_{\theta;f}(y)<+\infty \end{equation*}\]

令\(t=0\),最大期望算法(expectation-maximization algorithm)重复以下步骤:

E步:固定当前参数\(\theta^{(t)}\),将辅助概率测度取为当前参数下隐变量关于观测变量的条件分布:

\[\begin{equation*} \psi_{q^{(t)}}=K_{\theta^{(t)};g\mid f}(y,\cdot) \end{equation*}\]

M步:固定E步得到的辅助概率测度\(\psi_{q^{(t)}}\),关于参数\(\theta\)最大化证据下界。由性质 5.4.3(6)可得:

\[\begin{align*} \theta^{(t+1)}&\in\underset{\theta\in\Theta}{\arg\max}\;\mathcal{L}(q^{(t)},\theta;y)=\underset{\theta\in\Theta}{\arg\max}\;\int_{Z}q^{(t)}(z)\ln\frac{p_{\theta;f,g}(y,z)}{q^{(t)}(z)}\mathop{}\!\mathrm{d}\nu \\ &=\underset{\theta\in\Theta}{\arg\max}\;\left[\int_{Z}q^{(t)}(z)\ln p_{\theta;f,g}(y,z)\mathop{}\!\mathrm{d}\nu-\int_{Z}q^{(t)}(z)\ln q^{(t)}(z)\mathop{}\!\mathrm{d}\nu\right] \\ &=\underset{\theta\in\Theta}{\arg\max}\;\int_{Z}q^{(t)}(z)\ln p_{\theta;f,g}(y,z)\mathop{}\!\mathrm{d}\nu \end{align*}\]

称:

\[\begin{equation*} Q(\theta\mid\theta^{(t)})=\int_{Z}q^{(t)}(z)\ln p_{\theta;f,g}(y,z)\mathop{}\!\mathrm{d}\nu \end{equation*}\]

为\(Q\)函数。令\(t\leftarrow t+1\),继续执行E步和M步,直至参数序列或对数似然满足给定的收敛条件。

13.4 Monte Carlo算法

设\(f\)是概率空间\((X,\mathscr{F},P)\)上可积的随机变量,在许多统计计算问题中,我们关心的目标都可以归结为计算\(\operatorname{E}(f)\)。然而在实际问题中,这一积分往往难以直接求得,因此需要借助数值方法对其进行近似。

Monte Carlo算法的基本思想是利用随机抽样将积分计算转化为样本平均。设\(\{f_n\}\)是\(f\)的一列独立同分布副本,由定理 7.5可知:

\[\begin{equation*} \frac{1}{n}\sum_{i=1}^{n}f_i\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f) \end{equation*}\]

于是只要能够获得\(f_1, f_2, \dots, f_{n}\)的一组实现值\(f(x_1),f(x_2),\dots,f(x_n)\),就可以利用\(\dfrac{1}{n}\sum\limits_{i=1}^{n}f(x_i)\)逼近\(\operatorname{E}(f)\)。

13.4.1 重要性抽样

在上述Monte Carlo框架中,一个直接的思路是利用逆变换法或接受–拒绝法生成服从\(Pf^{-1}\)的样本。这两个方法的目标都是精确地从\(Pf^{-1}\)中抽样,但当我们的最终目标仅仅是计算\(\operatorname{E}(f)\)时,并不一定需要真正生成服从\(Pf^{-1}\)的样本。从一个更容易抽样的分布中抽样并对样本赋予相应的权重,可以在保持期望不变的同时绕开直接从\(Pf^{-1}\)抽样所带来的困难(分布函数左连续逆难以计算或接受拒绝法接受概率低),这正是下述抽样方法的基本思想。

Method 13.5. (Importance Sampling)

设\((Y,\mathscr{A},\mu)\)是\(\sigma\)有限测度空间,\(\tilde{\pi}:Y\to[0,+\infty)\)是\((Y,\mathscr{A})\)上目标概率分布\(\pi\)的未归一化概率密度,即:

\[\begin{equation*} Z=\int_Y\tilde{\pi}(y)\mathop{}\!\mathrm{d}\mu\in(0,+\infty),\qquad\forall\;A\in\mathscr{A},\;\pi(A)=\frac{1}{Z}\int_{A}\tilde{\pi}(y)\mathop{}\!\mathrm{d}\mu \end{equation*}\]

设\(Q\)是\((Y,\mathscr{A})\)上的概率测度,满足\(\pi\ll Q\ll\mu\)。根据定理 5.33,取\(q=\dfrac{\mathop{}\!\mathrm{d}Q}{\mathop{}\!\mathrm{d}\mu}\)的一个非负版本。定义重要性权重:

\[\begin{equation*} w(y)= \begin{cases} \dfrac{\tilde{\pi}(y)}{q(y)}, &q(y)>0 \\ 0, &q(y)=0 \end{cases} \end{equation*}\]

则对于任意关于\(\pi\)可积的\((Y,\mathscr{A})\)上的可测函数\(f\),有:

\[\begin{equation*} Z=\int_{Y}w(y)\mathop{}\!\mathrm{d}Q,\quad\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi=\frac{1}{Z}\int_{Y}f(y)w(y)\mathop{}\!\mathrm{d}Q \end{equation*}\]

即:

\[\begin{equation*} \int_{Y}f(y)\mathop{}\!\mathrm{d}\pi=\frac{\int_{Y}f(y)w(y)\mathop{}\!\mathrm{d}Q}{\int_{Y}w(y)\mathop{}\!\mathrm{d}Q} \end{equation*}\]

设\(g\)为从概率空间\((X,\mathscr{F},P)\)到\((Y,\mathscr{A})\)上的可测映射,且\(Pg^{-1}=Q\)。取\(g\)的一列独立同分布副本\(g_1, g_2, \dots, g_{n}\),并生成相应的观测值\(y_1, y_2, \dots, y_{n}\)。重要性采样按如下步骤估计\(\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\):

  1. 生成\(g_i\)的观测值\(y_i\),\(i=1,2,\dots,n\);

  2. 对每个样本计算\(w(y_i)\);

  3. 计算(由下面的证明可知当\(n\to+\infty\)时分母几乎必然不为\(0\)):

    \[\begin{equation*} \frac{\sum\limits_{i=1}^{n}w(y_i)f(y_i)}{\sum\limits_{i=1}^{n}w(y_i)} \end{equation*}\]

对于该方法有:

\[\begin{equation*} \hat{I}_n=\frac{\sum\limits_{i=1}^{n}[(wf)\circ g_i](x)}{\sum\limits_{i=1}^{n}(w\circ g_i)(x)}\overset{\text{a.s.}}{\longrightarrow}\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi \end{equation*}\]

证明. 由性质 5.3.3(2)可知\(A=\{q(y)=0\}\in\mathscr{A}\),根据性质 5.4.3(9)和\(\pi\ll Q\)可知:

\[\begin{equation*} Q(A)=\int_{\{q(y)=0\}}q(y)\mathop{}\!\mathrm{d}\mu=0=\pi(A) \end{equation*}\]

根据性质 5.4.3(9)可知\(\tilde{\pi}(y)=0\;\)a.e.于\((A,A\cap\mathscr{A},\mu)\),由性质 5.1.6(2)可知\(w(y)q(y)=\tilde{\pi}(y)\;\)a.e.于\((Y,\mathscr{A},\mu)\)。可以验证\(w(y)\)是\((Y,\mathscr{A})\)上的可测函数,根据引理 5.10和性质 5.4.3(8)可得:

\[\begin{equation*} \int_{Y}w(y)\mathop{}\!\mathrm{d}Q=\int_{Y}w(y)q(y)\mathop{}\!\mathrm{d}\mu=\int_{Y}\tilde{\pi}(y)\mathop{}\!\mathrm{d}\mu=Z \end{equation*}\]

由性质 5.3.3(5.b)、引理 5.10、性质 5.4.3(8)和性质 5.4.3(6)可知对于任意关于\(\pi\)可积的\((Y,\mathscr{A})\)上的可测函数\(f\)有:

\[\begin{equation*} \int_{Y}w(y)f(y)\mathop{}\!\mathrm{d}Q=\int_{Y}w(y)q(y)f(y)\mathop{}\!\mathrm{d}\mu=\int_{Y}\tilde{\pi}(y)f(y)\mathop{}\!\mathrm{d}\mu=Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi \end{equation*}\]

所以:

\[\begin{equation*} \int_{Y}f(y)\mathop{}\!\mathrm{d}\pi=\frac{\int_{Y}w(y)f(y)\mathop{}\!\mathrm{d}Q}{\int_{Y}w(y)\mathop{}\!\mathrm{d}Q} \end{equation*}\]

因为\(f\)关于\(\pi\)可积,类似于上上式的推导,由性质 5.3.3(5.d)和定理 5.28可知:

\[\begin{align*} \int_{Y}|w(y)f(y)|\mathop{}\!\mathrm{d}Q=\int_{X}(|wf|)\circ g(x)\mathop{}\!\mathrm{d}P<+\infty \end{align*}\]

同时:

\[\begin{equation*} \int_{Y}w(y)\mathop{}\!\mathrm{d}Q=\int_{X}w\circ g(x)\mathop{}\!\mathrm{d}P=Z<+\infty \end{equation*}\]

所以\((wf)\circ g\)和\(w\circ g\)均关于\(P\)可积。记\((wf)_i=(wf)\circ g_i,\;w_i=w\circ g_i\),则\((wf)_i\)和\(w_i\)均关于\(P\)可积,同时可以验证\((wf)_i\)之间同分布且\(w_i\)之间同分布。由性质 5.3.3(5.b)和性质 6.2.1(4)可知\((wf)_i\)之间独立且\(w_i\)之间独立,于是\((wf)_i\)之间独立同分布且\(w_i\)之间独立同分布。根据定理 7.5可得:

\[\begin{equation*} \frac{1}{n}\sum_{i=1}^{n}(wf)_i\overset{\text{a.s.}}{\longrightarrow}Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi,\quad\frac{1}{n}\sum_{i=1}^{n}w_i\overset{\text{a.s.}}{\longrightarrow}Z \end{equation*}\]

根据性质 7.0.9和定理 7.3(1)即可得出结论。 ◻

上述重要性采样方法通常被称为自加权重要性采样。与接受拒绝法类似,即使 \(Z\)未知或无法计算,该方法仍然有效。当\(Z\)已知时,则可以采用非自加权的重要性采样估计量,并获得更为简洁的理论性质。

Property 13.4.1. 对于重要性采样,若\(Z\)已知且\(wf\in L_2(Y,\mathscr{A},Q)\),则:

\[\begin{gather*} \hat{I}_n=\frac{1}{nZ}\sum_{i=1}^{n}[(wf)\circ g_i](x)\overset{\text{a.s.}}{\longrightarrow}\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi \\ \forall\;n\in\mathbb{N}^+,\;\operatorname{E}(\hat{I}_n)=\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi,\;\operatorname{Var}(\hat{I}_n)=\frac{1}{n}\left\{\frac{1}{Z^2}\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu-\left[\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2\right\} \end{gather*}\]

当\(q(y)\propto\tilde{\pi}(y)|f(y)|\)时\(\operatorname{Var}(\hat{I}_n)\)取得最小值。

证明. 此时\(\hat{I}_n\)的渐进性由自加权重要性采样的结论可以立即得出,下证明其期望与方差。

对任意的\(n\in\mathbb{N}^+\),由性质 5.4.3(6)、定理 5.28和自加权重要性采样中已证明的结论可得:

\[\begin{align*} &\operatorname{E}(\hat{I}_n)=\int_{X}\frac{1}{nZ}\sum_{i=1}^{n}[(wf)\circ g_i](x)\mathop{}\!\mathrm{d}P=\frac{1}{nZ}\sum_{i=1}^{n}\int_{X}(wf)\circ g_i(x)\mathop{}\!\mathrm{d}P \\ =&\frac{1}{nZ}\sum_{i=1}^{n}\int_{Y}w(y)f(y)\mathop{}\!\mathrm{d}Q=\frac{1}{nZ}\sum_{i=1}^{n}Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi=\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi \end{align*}\]

根据性质 6.4.2(3)、性质 6.4.3(7)和性质 5.4.3(6)可知:

\[\begin{equation*} \operatorname{Var}(\hat{I}_n)=\sum_{i=1}^{n}\operatorname{Var}\left[\frac{1}{nZ}(wf)\circ g_i\right]=n\operatorname{Var}\left[\frac{1}{nZ}(wf)\circ g\right]=\frac{1}{nZ^2}\operatorname{Var}[(wf)\circ g] \end{equation*}\]

由性质 6.4.2(1)、性质 5.3.3(5.b)、定理 5.28、自加权重要性采样中已证明的结论、性质 5.4.3(8)和引理 5.10可得:

\[\begin{align*} &\operatorname{Var}[(wf)\circ g]=\operatorname{E}\{[(wf)\circ g]^2\}-\{\operatorname{E}[(wf)\circ g]\}^2 \\ =&\int_{X}(wf)^2\circ g(x)\mathop{}\!\mathrm{d}P-\left[Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2=\int_{Y}[w(y)f(y)]^2\mathop{}\!\mathrm{d}Q-\left[Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2 \\ =&\int_{Y}\left[\frac{\tilde{\pi}(y)}{q(y)}f(y)\right]^2\mathop{}\!\mathrm{d}Q-\left[Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2=\int_{Y}\left[\frac{\tilde{\pi}(y)}{q(y)}f(y)\right]^2q(y)\mathop{}\!\mathrm{d}\mu-\left[Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2 \\ =&\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu-\left[Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2 \end{align*}\]

所以:

\[\begin{align*} \operatorname{Var}(\hat{I}_n)&=\frac{1}{nZ^2}\left\{\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu-\left[Z\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2\right\} \\ &=\frac{1}{n}\left\{\frac{1}{Z^2}\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu-\left[\int_{Y}f(y)\mathop{}\!\mathrm{d}\pi\right]^2\right\} \end{align*}\]

最小化\(\operatorname{Var}(\hat{I}_n)\)等价于:

\[\begin{equation*} \min_{q}\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu \end{equation*}\]

由不等式 1可知:

\[\begin{align*} &\left[\int_{Y}\tilde{\pi}(y)|f(y)|\mathop{}\!\mathrm{d}\mu\right]^2=\left[\int_{Y}\frac{\tilde{\pi}(y)|f(y)|}{\sqrt{q(y)}}\sqrt{q(y)}\mathop{}\!\mathrm{d}\mu\right]^2 \\ \leqslant&\left[\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu\right]\cdot\left\{\int_{Y}\left[\sqrt{q(y)}\right]^2\mathop{}\!\mathrm{d}\mu\right\}=\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu \end{align*}\]

上式取等当且仅当\(q(y)\propto\tilde{\pi}(y)|f(y)|\)。 ◻

note 13.4. 对于一般随机抽样的Monte Carlo算法而言,根据性质 5.4.3(6)、性质 6.4.2(1)和性质 6.3.1(2)可知:

\[\begin{align*} &\operatorname{Var}\left(\dfrac{1}{n}\sum_{i=1}^{n}f_i\right)=\frac{1}{n^2}\operatorname{Var}\left(\sum_{i=1}^{n}f_i\right)=\frac{1}{n^2}\left\{\operatorname{E}\left[\left(\sum_{i=1}^{n}f_i\right)^2\right]-\left[\operatorname{E}\left(\sum_{i=1}^{n}f_i\right)\right]^2\right\} \\ =&\frac{1}{n^2}\{n\operatorname{E}(f^2)+n(n-1)[\operatorname{E}(f)]^2-[n\operatorname{E}(f)]^2\}=\frac{1}{n^2}\{n\operatorname{E}(f^2)-n[\operatorname{E}(f)]^2\} \\ =&\frac{1}{n}\{\operatorname{E}(f^2)-[\operatorname{E}(f)]^2\} \end{align*}\]

将该方差与性质 13.4.1中的方差进行比较可知,只要:

\[\begin{equation*} \frac{1}{Z^2}\int_{Y}\frac{[\tilde{\pi}(y)f(y)]^2}{q(y)}\mathop{}\!\mathrm{d}\mu<\operatorname{E}(f^2) \end{equation*}\]

重要性采样的方差便小于一般的Monte Carlo估计量。进一步可以发现,一般的Monte Carlo实际上是重要性采样的一个特例,只需取\(q\)为\(\pi\)关于\(\mu\)的概率密度函数即可。

综上,重要性采样不仅可以降低抽样难度,在合理设计下还可以在保持抽样次数不变的情况下降低估计量的方差。

13.4.2 方差缩减技术

本节介绍一些Monte Carlo算法中在保持抽样次数不变的情况下降低估计量方差的方法。

13.4.2.1 对偶变量法

Method 13.6. 设\(f,g\in L_2(X,\mathscr{F},P)\)且二者同分布,\(\{(f_i,g_i)\}_{i=1}^n\)是\((f,g)\)的一列独立同分布副本。称\(g\)为\(f\)的对偶变量,定义对偶变量估计量:

\[\begin{equation*} \hat{I}_{n,\mathrm{AV}}\coloneq\frac{1}{n}\sum_{i=1}^{n}\frac{f_i+g_i}{2} \end{equation*}\]

则:

\[\begin{gather*} \hat{I}_{n,\mathrm{AV}}\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f) \\ \operatorname{E}(\hat{I}_{n,\mathrm{AV}})=\operatorname{E}(f),\quad\operatorname{Var}(\hat{I}_{n,\mathrm{AV}})=\frac{\operatorname{Var}(f)+\operatorname{Cov}(f,g)}{2n} \end{gather*}\]

当\(\operatorname{Cov}(f,g)<0\)且抽样次数为\(2n\)时,\(\operatorname{Var}(\hat{I}_{n,\mathrm{AV}})\)小于一般Monte Carlo估计量的方差。

证明. 由性质 5.4.3(6)可得\(\operatorname{E}(\hat{I}_{n,\mathrm{AV}})=\operatorname{E}(f)\)。根据定理 5.35和性质 6.2.1(4)可知\(\left\{\dfrac{f_i+g_i}{2}\right\}_{i=1}^n\)独立同分布,由性质 5.4.3(6)、性质 6.4.2(3)和性质 6.4.3(7)可得:

\[\begin{align*} \operatorname{Var}(\hat{I}_{n,\mathrm{AV}}) &=\frac{1}{n}\operatorname{Var}\left(\frac{f+g}{2}\right) =\frac{\operatorname{Var}(f)+2\operatorname{Cov}(f,g)+\operatorname{Var}(g)}{4n}\\ &=\frac{\operatorname{Var}(f)+\operatorname{Cov}(f,g)}{2n} \end{align*}\]

根据性质 5.4.3(7)(6)可知:

\[\begin{equation*} \operatorname{E}\left(\left|\frac{f_i+g_i}{2}\right|\right)\leqslant\operatorname{E}\left(\frac{|f_i|+|g_i|}{2}\right)=\frac{1}{2}[\operatorname{E}(|f_i|)+\operatorname{E}(|g_i|)]<+\infty \end{equation*}\]

由定理 7.5可得\(\hat{I}_{n,\mathrm{AV}}\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f)\)。 ◻

13.4.2.2 控制变量法

Method 13.7. 设\(f,g\in L_2(X,\mathscr{F},P)\),\(\eta=\operatorname{E}(g)\)已知且\(\operatorname{Var}(g)>0\),\(\{(f_i,g_i)\}_{i=1}^n\)是\((f,g)\)的一列独立同分布副本。称\(g\)为\(f\)的控制变量,对固定的\(b\in\mathbb{R}^{}\)定义控制变量估计量:

\[\begin{equation*} \hat{I}_{n,\mathrm{CV}}(b)=\frac{1}{n}\sum_{i=1}^{n}[f_i-b(g_i-\eta)] \end{equation*}\]

则:

\[\begin{gather*} \hat{I}_{n,\mathrm{CV}}(b)\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f) \\ \operatorname{E}[\hat{I}_{n,\mathrm{CV}}(b)]=\operatorname{E}(f),\quad\operatorname{Var}[\hat{I}_{n,\mathrm{CV}}(b)]=\frac{\operatorname{Var}(f)-2b\operatorname{Cov}(f,g)+b^2\operatorname{Var}(g)}{n} \end{gather*}\]

\(\operatorname{Var}[\hat{I}_{n,\mathrm{CV}}(b)]\)在\(b^*=\operatorname{Cov}(f,g)/\operatorname{Var}(g)\)处取到最小值:

\[\begin{equation*} \operatorname{Var}[\hat{I}_{n,\mathrm{CV}}(b^*)]=\frac{1}{n}\left[\operatorname{Var}(f)-\frac{\operatorname{Cov}(f,g)^2}{\operatorname{Var}(g)}\right]=\frac{1-\rho^2}{n}\operatorname{Var}(f) \end{equation*}\]

其中\(\rho=\operatorname{Corr}(f,g)\)。

证明. 由性质 5.4.3(6)可得\(\operatorname{E}[\hat{I}_{n,\mathrm{CV}}(b)]=\operatorname{E}(f)\)。根据定理 5.35和性质 6.2.1(4)可知\(\{f_i-b(g_i-\eta)\}_{i=1}^n\)独立同分布,由性质 5.4.3(6)、性质 6.4.2(3)和性质 6.4.3(7)(5)(4)(3)可得:

\[\begin{align*} &\operatorname{Var}[\hat{I}_{n,\mathrm{CV}}(b)]=\frac{1}{n^2}\operatorname{Var}\left\{\sum_{i=1}^{n}[f_i-b(g_i-\eta)]\right\}=\frac{1}{n}\sum_{i=1}^{n}\operatorname{Var}[f_i-b(g_i-\eta)] \\ =&\frac{\operatorname{Var}[f_i-b(g_i-\eta)]}{n}=\frac{\operatorname{Var}[f-b(g-\eta)]}{n}=\frac{\operatorname{Var}(f)+\operatorname{Var}[b(g-\eta)]-2\operatorname{Cov}[f,b(g-\eta)]}{n} \\ =&\frac{\operatorname{Var}(f)+b^2\operatorname{Var}(g)-2b\operatorname{Cov}(f,g)}{n} \end{align*}\]

这是一个关于\(b\)的二次函数,最小值点为\(b^*\operatorname{Cov}(f,g)/\operatorname{Var}(g)\),对应的最小值为:

\[\begin{align*} &\frac{1}{n}\left\{\operatorname{Var}(f)+\frac{[\operatorname{Cov}(f,g)]^2}{[\operatorname{Var}(g)]^2}\operatorname{Var}(g)-2\frac{\operatorname{Cov}(f,g)}{\operatorname{Var}(g)}\operatorname{Cov}(f,g)\right\} \\ =&\frac{1}{n}\left\{\operatorname{Var}(f)-\frac{[\operatorname{Cov}(f,g)]^2}{\operatorname{Var}(g)}\right\}=\frac{\operatorname{Var}(f)-\rho^2\operatorname{Var}(f)}{n}=\frac{1-\rho^2}{n}\operatorname{Var}(f) \end{align*}\]

根据性质 5.4.3(7)(6)可知:

\[\begin{align*} &\operatorname{E}[|f_i-b(g_i-\eta)|]\leqslant\operatorname{E}[|f_i|+|b(g_i-\eta)|]=\operatorname{E}(|f_i|)+|b|\operatorname{E}(|g_i-\eta|) \\ \leqslant&\operatorname{E}(|f_i|)+|b|\operatorname{E}(|g_i|+|\eta|)=\operatorname{E}(|f_i|)+|b|\operatorname{E}(|g_i|)+|b\eta|<+\infty \end{align*}\]

由定理 7.5可得\(\hat{I}_{n,\mathrm{CV}}(b)\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f)\)。 ◻

\(\rho^2\)越接近\(1\),控制变量法的效果越好。实际中\(b^*\)通常未知,可先用独立的预模拟样本估计\(b^*\),再固定该系数进行正式的Monte Carlo。

13.4.2.3 条件期望法

Method 13.8. 设\(f\in L_2(X,\mathscr{F},P)\),\(\mathscr{A}\)是\(\mathscr{F}\)的子\(\sigma\)域,\(g=\operatorname{E}(f\mid\mathscr{A})\),\(\{g_i\}_{i=1}^n\)是\(g\)的一列独立同分布副本,定义条件期望估计量:

\[\begin{equation*} \hat{I}_{n,\mathrm{CE}}\coloneq\frac{1}{n}\sum_{i=1}^{n}g_i \end{equation*}\]

则:

\[\begin{gather*} \hat{I}_{n,\mathrm{CE}}\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f) \\ \operatorname{E}(\hat{I}_{n,\mathrm{CE}})=\operatorname{E}(f),\quad\operatorname{Var}(\hat{I}_{n,\mathrm{CE}})=\frac{1}{n}\operatorname{Var}[\operatorname{E}(f\mid\mathscr{A})] \end{gather*}\]

且\(\operatorname{Var}(\hat{I}_{n,\mathrm{CE}})\)不大于一般Monte Carlo估计量的方差。

证明. 由性质 5.4.3(6)和性质 6.3.2(3)可得\(\operatorname{E}(\hat{I}_{n,\mathrm{CE}})=\operatorname{E}(g)=\operatorname{E}(f)\)。因为\(\{g_i\}_{i=1}^n\)独立同分布,根据性质 5.4.3(6)、性质 6.4.2(3)和性质 6.4.3(7)可知:

\[\begin{equation*} \operatorname{Var}(\hat{I}_{n,\mathrm{CE}})=\frac{1}{n^2}\operatorname{Var}\left(\sum_{i=1}^{n}h_i\right)=\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(h_i)=\frac{1}{n}\operatorname{Var}(h)=\frac{1}{n}\operatorname{Var}[\operatorname{E}(f\mid\mathscr{A})] \end{equation*}\]

由性质 6.4.2(2)可得:

\[\begin{equation*} \operatorname{Var}(f)=\operatorname{E}[\operatorname{Var}(f\mid\mathscr{A})]+\operatorname{Var}[\operatorname{E}(f\mid\mathscr{A})] \end{equation*}\]

根据性质 5.4.2(2)可知:

\[\begin{equation*} \operatorname{Var}(\hat{I}_{n,\mathrm{CE}})=\frac{1}{n}\{\operatorname{Var}(f)-\operatorname{E}[\operatorname{Var}(f\mid\mathscr{A})]\}\leqslant\frac{\operatorname{Var}(f)}{n} \end{equation*}\]

由不等式 10可得:

\[\begin{equation*} g^2=[\operatorname{E}(f\mid\mathscr{A})]^2\leqslant\operatorname{E}(f^2\mid\mathscr{A}) \end{equation*}\]

根据性质 5.4.3(7)和性质 6.3.2(3)可知:

\[\begin{equation*} \operatorname{E}(g^2)\leqslant\operatorname{E}(f^2)<+\infty \end{equation*}\]

由性质 5.5.1(2)可得\(g\in L_2(X,\mathscr{A},P)\subseteq L_1(X,\mathscr{A},P)\)。根据定理 7.5可知\(\hat{I}_{n,\mathrm{CE}}(b)\overset{\text{a.s.}}{\longrightarrow}\operatorname{E}(f)\)。 ◻


  1. 具体例子可以参考机器学习部分的高斯混合模型。↩︎

COMPUTATIONAL NOTES

计算实验

实验01

非线性方程求解:迭代误差

Python · Jupyter

非线性方程:误差随迭代次数的变化

求解方程

cos x − x = 0.

使用 SciPy 的 Brent 法计算高精度参考根,再比较二分法、不动点迭代与 Newton 法的逐步误差。

import numpy as np
import matplotlib.pyplot as plt
from figure_settings import configure_matplotlib
configure_matplotlib()
from scipy.optimize import root_scalar

def f(x):
    return np.cos(x) - x

def df(x):
    return -np.sin(x) - 1.0

reference = root_scalar(
    f,
    bracket=(0.0, 1.0),
    method="brentq",
    xtol=1e-14,
).root
print(f"Reference root x* = {reference:.16f}")
Reference root x* = 0.7390851332151612
def bisection_history(left, right, iterations):
    values = []
    for _ in range(iterations):
        midpoint = (left + right) / 2.0
        values.append(midpoint)
        if f(left) * f(midpoint) <= 0:
            right = midpoint
        else:
            left = midpoint
    return np.asarray(values)

def fixed_point_history(x0, iterations):
    values = [x0]
    for _ in range(iterations):
        values.append(np.cos(values[-1]))
    return np.asarray(values)

def newton_history(x0, iterations):
    values = [x0]
    for _ in range(iterations):
        current = values[-1]
        values.append(current - f(current) / df(current))
    return np.asarray(values)

histories = {
    "Bisection": bisection_history(0.0, 1.0, 45),
    "Fixed-point iteration": fixed_point_history(0.5, 45),
    "Newton's method": newton_history(0.5, 8),
}

tiny = np.finfo(float).tiny
errors = {
    name: np.maximum(np.abs(values - reference), tiny)
    for name, values in histories.items()
}
colors = {
    "Bisection": "#52649a",
    "Fixed-point iteration": "#c26b4a",
    "Newton's method": "#397d78",
}

fig, ax = plt.subplots(figsize=(9.2, 5.4), constrained_layout=True)
for name, values in errors.items():
    ax.semilogy(
        np.arange(len(values)),
        values,
        marker="o",
        markersize=3.5,
        linewidth=1.8,
        color=colors[name],
        label=name,
    )

ax.set(
    title="Error in Nonlinear Equation Solvers",
    xlabel="Iteration $k$",
    ylabel=r"Absolute error $|x_k-x^*|$",
)
ax.legend()
plt.show()

print("Method                    Iterates    Final absolute error")
for name, values in errors.items():
    print(f"{name:<12} {len(values):>6d}    {values[-1]:.3e}")

实验 01「非线性方程求解:迭代误差」结果 1(图 1)

Method                    Iterates    Final absolute error
Bisection        45    2.276e-14
Fixed-point iteration     46    4.021e-09
Newton's method      9    5.551e-16

观察

  • 二分法误差稳定地按固定比例下降。
  • 不动点迭代为线性收敛,速度取决于根附近的压缩系数。
  • Newton 法进入根的邻域后误差近似平方下降,很快达到浮点精度。
实验02

多项式逼近:节点选择与逼近误差

Python · Jupyter

多项式插值:节点选择与逼近误差

在区间 [−1, 1] 上逼近 Runge 函数

$$f(x)=\frac{1}{1+25x^2}.$$

使用 SciPy 的 BarycentricInterpolator,比较等距节点与 Chebyshev 节点。

import numpy as np
import matplotlib.pyplot as plt
from figure_settings import configure_matplotlib
configure_matplotlib()
from scipy.interpolate import BarycentricInterpolator

def runge(x):
    return 1.0 / (1.0 + 25.0 * x**2)

def equidistant_nodes(degree):
    return np.linspace(-1.0, 1.0, degree + 1)

def chebyshev_nodes(degree):
    k = np.arange(degree + 1)
    return np.cos((2 * k + 1) * np.pi / (2 * (degree + 1)))

grid = np.linspace(-1.0, 1.0, 3000)
truth = runge(grid)
degrees = [4, 8, 12, 16]
records = []

for degree in degrees:
    for node_name, node_function in [
        ("Equidistant nodes", equidistant_nodes),
        ("Chebyshev nodes", chebyshev_nodes),
    ]:
        nodes = node_function(degree)
        interpolator = BarycentricInterpolator(nodes, runge(nodes))
        approximation = interpolator(grid)
        maximum_error = np.max(np.abs(approximation - truth))
        records.append((degree, node_name, maximum_error))

print("Degree    Node type                 Maximum absolute error")
for degree, node_name, maximum_error in records:
    print(f"{degree:>4d}    {node_name:<18} {maximum_error:.4e}")
Degree    Node type                 Maximum absolute error
   4    Equidistant nodes  4.3836e-01
   4    Chebyshev nodes    4.0202e-01
   8    Equidistant nodes  1.0452e+00
   8    Chebyshev nodes    1.7084e-01
  12    Equidistant nodes  3.6634e+00
  12    Chebyshev nodes    6.9215e-02
  16    Equidistant nodes  1.4393e+01
  16    Chebyshev nodes    3.2613e-02
degree = 12
nodes_equal = equidistant_nodes(degree)
nodes_chebyshev = chebyshev_nodes(degree)

polynomial_equal = BarycentricInterpolator(
    nodes_equal,
    runge(nodes_equal),
)(grid)
polynomial_chebyshev = BarycentricInterpolator(
    nodes_chebyshev,
    runge(nodes_chebyshev),
)(grid)

fig, axes = plt.subplots(1, 2, figsize=(12, 4.8), constrained_layout=True)

axes[0].plot(grid, truth, color="#1f2430", linewidth=2.2, label="Original function")
axes[0].plot(grid, polynomial_equal, color="#c26b4a", linewidth=1.7, label="Equidistant nodes")
axes[0].plot(grid, polynomial_chebyshev, color="#397d78", linewidth=1.7, label="Chebyshev nodes")
axes[0].scatter(nodes_equal, runge(nodes_equal), color="#c26b4a", s=18, zorder=3)
axes[0].set(
    title=f"Degree-{degree} Interpolation",
    xlabel="x",
    ylabel="Function value",
    ylim=(-0.5, 1.5),
)
axes[0].legend()

axes[1].semilogy(
    grid,
    np.maximum(np.abs(polynomial_equal - truth), np.finfo(float).tiny),
    color="#c26b4a",
    linewidth=1.7,
    label="Equidistant nodes",
)
axes[1].semilogy(
    grid,
    np.maximum(np.abs(polynomial_chebyshev - truth), np.finfo(float).tiny),
    color="#397d78",
    linewidth=1.7,
    label="Chebyshev nodes",
)
axes[1].set(
    title="Pointwise Absolute Error",
    xlabel="x",
    ylabel="Absolute error",
)
axes[1].legend()
plt.show()

实验 02「多项式逼近:节点选择与逼近误差」结果 1(图 1)

观察

提高多项式次数并不必然改善全区间逼近。等距节点在区间端点附近可能出现明显振荡;Chebyshev 节点在端点附近更密集,通常能显著降低最大误差。