Definition 16.9. 设\(\mathbf{X},\mathbf{Y}\)分别是\(m,n\)维随机向量，\(m+n\)维随机向量\((\mathbf{X},\mathbf{Y})^{\top}\)的均值向量\(\boldsymbol{\mu}=\mathbf{0}\)，协方差矩阵\(\Sigma\)是正定阵。对\(\mathbf{X},\mathbf{Y}\)分别进行线性变换\(\mathcal{T}_1,\mathcal{T}_2\)得到两个\(p\)维随机向量\(\mathbf{U}=\mathbf{U}_1, \mathbf{U}_2, \dots, \mathbf{U}_{p},\;\mathbf{V}=\mathbf{V}_1, \mathbf{V}_2, \dots, \mathbf{V}_{p}\)，\(\mathcal{T}_1\)和\(\mathcal{T}_2\)的矩阵分别为：
\[\begin{equation*}
A=
\begin{pmatrix}
\alpha_1^{\top} \\
\alpha_2^{\top} \\
\vdots \\
\alpha_p^{\top}
\end{pmatrix},\quad
B=
\begin{pmatrix}
\beta_1^{\top} \\
\beta_2^{\top} \\
\vdots \\
\beta_p^{\top}
\end{pmatrix}
\end{equation*}\]
若：
\(\mathbf{U}_i,\mathbf{V}_i\)和之前的\(i-1\)对变量都不相关，即：
\[\begin{equation*}
\operatorname{Corr}(\mathbf{U}_i,\mathbf{U}_j)=\operatorname{Corr}(\mathbf{U}_i,\mathbf{V}_j)=\operatorname{Corr}(\mathbf{V}_i,\mathbf{V}_j)=\operatorname{Corr}(\mathbf{V}_i,\mathbf{U}_j)=0,\quad j=1,2,\dots,i-1
\end{equation*}\]
\(\operatorname{Corr}(\mathbf{U}_i,\mathbf{V}_i)=\max\limits_{\alpha,\beta}\operatorname{Corr}(\alpha^{\top}\mathbf{X},\beta^{\top}\mathbf{Y})\)。
则称\(\mathbf{U}_i,\mathbf{V}_i\)是第\(i\)对典型变量（canonical variable），\(i=1,2,\dots,p\)。
Definition 16.10. 设\(\mathbf{X},\mathbf{Y}\)分别是\(m,n\)维随机向量，\(m+n\)维随机向量\((\mathbf{X},\mathbf{Y})^{\top}\)的均值向量\(\boldsymbol{\mu}=\mathbf{0}\)，协方差矩阵\(\Sigma\)是正定阵。对\(\mathbf{X},\mathbf{Y}\)分别进行线性变换\(\mathcal{T}_1,\mathcal{T}_2\)得到两个\(p\)维随机向量\(\mathbf{U}=\mathbf{U}_1, \mathbf{U}_2, \dots, \mathbf{U}_{p},\;\mathbf{V}=\mathbf{V}_1, \mathbf{V}_2, \dots, \mathbf{V}_{p}\)，\(\mathcal{T}_1\)和\(\mathcal{T}_2\)的矩阵分别为：
\[\begin{equation*}
A=
\begin{pmatrix}
\alpha_1^{\top} \\
\alpha_2^{\top} \\
\vdots \\
\alpha_p^{\top}
\end{pmatrix},\quad
B=
\begin{pmatrix}
\beta_1^{\top} \\
\beta_2^{\top} \\
\vdots \\
\beta_p^{\top}
\end{pmatrix}
\end{equation*}\]
若：
\(\mathbf{U}_i,\mathbf{V}_i\)和之前的\(i-1\)对变量都不相关，即：
\[\begin{equation*}
\operatorname{Corr}(\mathbf{U}_i,\mathbf{U}_j)=\operatorname{Corr}(\mathbf{U}_i,\mathbf{V}_j)=\operatorname{Corr}(\mathbf{V}_i,\mathbf{V}_j)=\operatorname{Corr}(\mathbf{V}_i,\mathbf{U}_j)=0,\quad j=1,2,\dots,i-1
\end{equation*}\]
\(\operatorname{Corr}(\mathbf{U}_i,\mathbf{V}_i)=\max\limits_{\alpha,\beta}\operatorname{Corr}(\alpha^{\top}\mathbf{X},\beta^{\top}\mathbf{Y})\)；
\(\operatorname{Var}(\mathbf{U}_i)=\operatorname{Var}(\mathbf{V}_i)=1,\;i=1,2,\dots,p\)。
则称\(\mathbf{U}_i,\mathbf{V}_i\)是第\(i\)对典型变量，\(\operatorname{Corr}(\mathbf{U}_i,\mathbf{V}_i)\)为第\(i\)典型相关系数，\(i=1,2,\dots,p\)。
Theorem 16.4. 设\(\mathbf{X},\mathbf{Y}\)分别是\(m,n\)维随机向量，\(m+n\)维随机向量\(\mathbf{Z}=(\mathbf{X},\mathbf{Y})^{\top}\)的均值向量\(\boldsymbol{\mu}=\mathbf{0}\)，协方差矩阵\(\Sigma\)是正定阵，其中：
\[\begin{equation*}
\Sigma=
\begin{pmatrix}
\Sigma_{11} & \Sigma_{12} \\
\Sigma_{21} & \Sigma_{22}
\end{pmatrix}
\end{equation*}\]
令\(M=\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\)，\(M\)的特征值从大到小为\(\lambda_1, \lambda_2, \dots, \lambda_{m}\)，对应的正交化特征向量为\(a_1, a_2, \dots, a_{m}\)，设：
\[\begin{equation*}
b_i=\Sigma_{22}^{-1}\Sigma_{21}a_i,\quad\alpha_i=\frac{a_i}{\sqrt{a_i^{\top}\Sigma_{11}a_i}},\quad\beta_i=\frac{b_i}{\sqrt{b_i^{\top}\Sigma_{22}b_i}}
\end{equation*}\]
则\(\sqrt{\lambda_i^2}\)为第\(i\)对典型相关系数，\(\mathbf{U}_i=\alpha_i^{\top}\mathbf{X},\mathbf{V}_i=\beta_i^{\top}\mathbf{Y}\)是第\(i\)对典型变量。
证明. 求第一对典型相关系数的过程等价于求解下述约束优化问题：
\[\begin{gather*}
\max\operatorname{Corr}(\mathbf{U}_1,\mathbf{V_1})=\alpha_1^{\top}\Sigma_{12}\beta_1 \\
\operatorname{s.t.}
\begin{cases}
\operatorname{Var}(\mathbf{U}_1)=\alpha_1^{\top}\Sigma_{11}\alpha_1=1 \\
\operatorname{Var}(\mathbf{V}_1)=\beta_1^{\top}\Sigma_{22}\beta_1=1
\end{cases}
\end{gather*}\]
使用Lagrange乘子法进行求解，构造Lagrange函数：
\[\begin{equation*}
L(\alpha_1,\beta_1,s,t)=\alpha_1^{\top}\Sigma_{12}\beta_1-\frac{s}{2}(\alpha_1^{\top}\Sigma_{11}\alpha_1-1)-\frac{t}{2}(\beta_1^{\top}\Sigma_{22}\beta_1-1)
\end{equation*}\]
利用矩阵求导可得：
\[\begin{gather*}
\frac{\partial L(\alpha_1,\beta_1,s,t)}{\partial\alpha_1}=\Sigma_{12}\beta_1-s\Sigma_{11}\alpha_1 \\
\frac{\partial L(\alpha_1,\beta_1,s,t)}{\partial\beta_1}=\Sigma_{21}\alpha_1-t\Sigma_{22}\beta_1 \\
\frac{\partial L(\alpha_1,\beta_1,s,t)}{\partial s}=\alpha_1^{\top}\Sigma_{11}\alpha_1-1 \\
\frac{\partial L(\alpha_1,\beta_1,s,t)}{\partial t}=\beta_1^{\top}\Sigma_{22}\beta_1-1
\end{gather*}\]
令上四式全部为\(0\)，即：
\[\begin{equation*}
\Sigma_{12}\beta_1-s\Sigma_{11}\alpha_1=0,\quad\Sigma_{21}\alpha_1-t\Sigma_{22}\beta_1=0,\quad\alpha_1^{\top}\Sigma_{11}\alpha_1-1=0,\quad\beta_1^{\top}\Sigma_{22}\beta_1-1=0
\end{equation*}\]
在上第一式和第二式两边分别同乘\(\alpha_1^{\top}\)和\(\beta_1^{\top}\)可得\(s=t\)。因为\(\Sigma\)正定，由定理 2.17(6)可知\(\Sigma\)可逆，于是\(\Sigma_{11}^{-1},\Sigma_{22}^{-1}\)存在。注意到：
\[\begin{equation*}
s\alpha_1=\Sigma_{11}^{-1}\Sigma_{12}\beta_1,\quad s\beta_1=\Sigma_{22}^{-1}\Sigma_{21}\alpha
\end{equation*}\]
于是有：
\[\begin{equation*}
s^2\alpha_1=\Sigma_{11}^{-1}\Sigma_{12}s\beta_1=\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\alpha_1,\quad
s^2\beta_1=\Sigma_{22}^{-1}\Sigma_{21}s\alpha_1=\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}\beta_1
\end{equation*}\]
即\(s^2\)是\(\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\)和\(\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}\)共同的特征值。由性质 2.6.1(3)可知\(\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\)和\(\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}\)具有相同的非零特征值。因为：
\[\begin{equation*}
\operatorname{Corr}(\mathbf{U}_1,\mathbf{V}_1)=\alpha_1^{\top}\Sigma_{12}\beta_1=\alpha_1^{\top}s\Sigma_{11}\alpha_1=s\alpha_1^{\top}\Sigma_{11}\alpha_1=s
\end{equation*}\]
所以优化问题的解为\(\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\)和\(\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}\)的最大特征值\(s^2\)、\(\Sigma_{11}^{-1}\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\)的特征值\(s^2\)对应的满足\(\alpha_1^{\top}\Sigma_{11}\alpha_1=1\)的特征向量\(\alpha_1\)和\(\Sigma_{22}^{-1}\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}\)的特征值\(s^2\)对应的满足\(\beta_1^{\top}\Sigma_{22}\beta_1=1\)的特征向量\(\beta_1\)。显然结论对第\(1\)对典型变量成立。
考虑第\(i\)对典型变量，此时的优化问题为：
\[\begin{gather*}
\max\operatorname{Corr}(\mathbf{U}_i,\mathbf{V}_i)=\alpha_i^{\top}\Sigma_{12}\beta_i \\
\operatorname{s.t.}
\begin{cases}
\operatorname{Var}(\mathbf{U}_i)=\alpha_i^{\top}\Sigma_{11}\alpha_i=1 \\
\operatorname{Var}(\mathbf{V}_i)=\beta_i^{\top}\Sigma_{22}\beta_i=1 \\
\operatorname{Cov}(\mathbf{U}_i,\mathbf{U}_j)=\alpha_i^{\top}\Sigma_{11}\alpha_j=0,&j=1,2,\dots,i-1 \\
\operatorname{Cov}(\mathbf{V}_i,\mathbf{V}_j)=\beta_i^{\top}\Sigma_{12}\beta_j=0,&j=1,2,\dots,i-1 \\
\operatorname{Cov}(\mathbf{U}_i,\mathbf{V_j})=\alpha_i^{\top}\Sigma_{12}\beta_j=0,&j=1,2,\dots,i-1 \\
\operatorname{Cov}(\mathbf{V}_i,\mathbf{U}_j)=\beta_i^{\top}\Sigma_{21}\alpha_j=0,&j=1,2,\dots,i-1
\end{cases}
\end{gather*}\]
因为\(\Sigma\)正定，所以\(\Sigma_{11}\)和\(\Sigma_{22}\)正定，即存在\(\Sigma_{11}^{-\frac{1}{2}}\)和\(\Sigma_{22}^{-\frac{1}{2}}\)。令：
\[\begin{equation*}
\gamma=\Sigma_{11}^{\frac{1}{2}}\alpha_i,\quad\delta=\Sigma_{22}^{\frac{1}{2}}\beta_i,\quad\gamma_j=\Sigma_{11}^{\frac{1}{2}}\alpha_j,\quad\delta_j=\Sigma_{22}^{\frac{1}{2}}\beta_j,\quad j=1,2,\dots,i-1
\end{equation*}\]
同时\(\gamma\)和\(\alpha_i\)、\(\delta\)和\(\beta_i\)、\(\gamma_j\)和\(\alpha_j\)、\(\delta_j\)和\(\beta_j\)都是一一对应的关系。由性质 19.4.4(3)，此时约束条件可改写为：
\[\begin{equation*}
\begin{cases}
\operatorname{Var}(\mathbf{U}_i)=\alpha_i^{\top}\Sigma_{11}\alpha_i=||\gamma||=1 \\
\operatorname{Var}(\mathbf{V}_i)=\beta_i^{\top}\Sigma_{22}\beta_i=||\delta||=1 \\
\operatorname{Cov}(\mathbf{U}_i,\mathbf{U}_j)=\alpha_i^{\top}\Sigma_{11}\alpha_j=\gamma^{\top}\gamma_j=0 \\
\operatorname{Cov}(\mathbf{V}_i,\mathbf{V}_j)=\beta_i^{\top}\Sigma_{12}\beta_j=\delta^{\top}\delta_j=0 \\
\operatorname{Cov}(\mathbf{U}_i,\mathbf{V_j})=\alpha_i^{\top}\Sigma_{12}\beta_j=(\Sigma_{11}^{-\frac{1}{2}}\gamma)^{\top}\Sigma_{12}\Sigma_{22}^{-\frac{1}{2}}\delta_j=\gamma^{\top}\Sigma_{11}^{-\frac{1}{2}}\Sigma_{12}\Sigma_{22}^{-\frac{1}{2}}\delta_j=0 \\
\operatorname{Cov}(\mathbf{V}_i,\mathbf{U}_j)=\beta_i^{\top}\Sigma_{21}\alpha_j=(\Sigma_{22}^{-\frac{1}{2}}\delta)^{\top}\Sigma_{21}(\Sigma_{11}^{-\frac{1}{2}}\gamma_j)=\delta^{\top}\Sigma_{22}^{-\frac{1}{2}}\Sigma_{21}\Sigma_{11}^{-\frac{1}{2}}\gamma_j=0
\end{cases}
\end{equation*}\]
令\(K=\Sigma_{11}^{-\frac{1}{2}}\Sigma_{12}\Sigma_{22}^{-\frac{1}{2}}\)，则最后两个约束条件可改写为\(\gamma^{\top}K\delta_j=0,\;\delta^{\top}K^{\top}\gamma_j=0,\;j=1,2,\dots,i-1\)，目标函数可改写为\(\alpha_i^{\top}\Sigma_{12}\beta_i=(\Sigma_{11}^{-\frac{1}{2}}\gamma)^{\top}\Sigma_{12}(\Sigma_{22}^{-\frac{1}{2}}\delta)=\gamma^{\top}\Sigma_{11}^{-\frac{1}{2}}\Sigma_{12}\Sigma_{22}^{-\frac{1}{2}}\delta=\gamma^{\top}K\delta\)。由不等式 3可得：
\[\begin{equation*}
|\gamma^{\top}K\delta|\leqslant||K^{\top}\gamma||\;||\delta||
\end{equation*}\]
由取等条件和第二个约束条件可知固定\(\gamma\)情况下\(\delta\)的最优取值为\(\dfrac{K^{\top}\gamma}{||K^{\top}\gamma||}\)，此时目标函数值为：
\[\begin{equation*}
\gamma^{\top}K\delta=\frac{\gamma^{\top}KK^{\top}\gamma}{||K^{\top}\gamma||}=(\gamma^{\top}KK^{\top}\gamma)^\frac{1}{2}
\end{equation*}\]
接下来在第一个约束条件和第三个约束条件下求上式的最大值。
设\(\operatorname{rank}(K)=r\)，由定理 2.25可知\(KK^{\top}\)是半正定矩阵，根据定理 2.18(3.5)和性质 2.6.3(3)可知\(\gamma^{\top}KK^{\top}\gamma\)有如下分解：
\[\begin{equation*}
\gamma^{\top}KK^{\top}\gamma=\gamma^{\top}\left(\sum_{i=1}^{r}\varphi_i^2l_i^{\top}l_i\right)\gamma=
\end{equation*}\]
◻
（Initialization step）