Linear Algebra

Transpose

$$\begin{array}{rl}(AB{)}^T& =B^T{A}^T\\ (ABC{)}^T& =C^T{B}^T{A}^T\end{array}$$

---

Norm

L2 norm $\Vert x{\Vert }_2$ has to be $x^{T}x$ - scalar , dot product, can not be $x^{T}x$ - matrix

The matrix is invertible(singular matrix) if its $\det \ne 0$ , invertible means if it has inverse or not; so if the matrix has inverse, then its det must not be 0. ($A^{-1}$ exists ⇔ $\det (A)\ne 0$).

The matrix is non-singular if its $\det \ne 0$ (Determinant Test for Non-Singularity), otherwise it’s singular

Determinant

See Linear Algebra - Determinant.

Calculate Determinant

for 2x2 and 3x3, its easy to calculate

what about $n\ge 3$ matrix, what will be the faster way or at least temp fast way to calculate

Laplace expansion

---

Rank, Linearly Independent

See Linear Algebra - Rank.

Systems of Linear Equations

How to Solve a Linear System $Ax=b$​ (general rules):

$b=0$ (homogeneous case)
If $\det (A)\ne 0$ then the system has a unique trivial (zero) solution.
If $\det (A)=0$​ then the system has an infinite number of solutions.

$b\ne 0$​​ (non-homogeneous case)
If $\det (A)\ne 0$​​ then the system has a unique solution.
If $\det (A)=0$​​ then:
a). $\mathrm{rank}(A)=\mathrm{rank}(\tilde{A})\Rightarrow$​​ the system has an infinite number of solutions.
b). $\mathrm{rank}(A)\ne \mathrm{rank}(\tilde{A})\Rightarrow$​​ the system is inconsistent.

Augmented Matrix

See Linear Algebra - Augmented Matrix.

LU Decomposition

See Linear Algebra - LU Decomposition.

See Linear Algebra - LU Decomposition Example.

Eigenvalue and Eigenvector

Dot Product

$\left[\begin{array}{c}4\\ 1\end{array}\right]\cdot \left[\begin{array}{c}2\\ 3\end{array}\right]=4\cdot 2+1\cdot 3=11$
is same as matrix-vector multiplication:
$\left[\begin{array}{cc}4& 1\end{array}\right]\left[\begin{array}{c}2\\ 3\end{array}\right]=4\times 2+1\times 3=11$
dot product:

$$\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]\cdot \left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]=1\cdot 1+2\cdot 2+3\cdot 3=1+4+9=14$$

outer product:

$$\left[\begin{array}{ccc}1& 2& 3\\ 1& 2& 3\\ 1& 2& 3\end{array}\right]\cdot \left[\begin{array}{ccc}1& 2& 3\\ 1& 2& 3\\ 1& 2& 3\end{array}\right]=\left[\begin{array}{ccc}6& 12& 18\\ 6& 12& 18\\ 6& 12& 18\end{array}\right]$$

inner/dot product	outer product
result is scalar	result is Matrix

PSD, PD, SPD

a matrix A is positive semi-definite if $x^{T}Ax\ge 0,\forall x\in {\mathbb{R}}^n$
eigenvalue non-negative, including 0

a matrix A is positive definite if $x^{T}Ax>0,\forall x\in {\mathbb{R}}^n\setminus \{0\}$
eigenvalues are strictly positive, can not be 0

PD $\in$ PSD

Hessian and Jacobian

Practice question
Understanding Jacobian and Hessian matrices with example

Symmetric Matrix

$A^T=A$

Orthogonal Matrix

$A^T=A^{-1}$

Reference

• A COOK-BOOK of MATH(Zotero)
• QuickMath