Matrix

The matrix of a linear map $\mathcal{M}(T)$ is defined with respect to a set of basis for $V$ and $W$. The entries of $\mathcal{M}(T)$ are defined as $$\begin{array}{r}T{v}_k=A_{1,k}{w}_1+A_{2,k}{w}_2+\cdots +A_{m,k}{w}_{m}k=1,\dots ,n\end{array}$$ where $v_1,\dots ,v_n$ and $w_1,\dots ,w_m$ are the basis vectors of $V$ and $W$ respectively. $\mathcal{M}(T)$ is a $m$-by-$n$ matrix and each element $\in \mathcal{F}$. Unless stated otherwise, we will use the standard basis. It is important to note that the matrix of a linear map is always defined with respect to a set of basis vectors.

Addition and Multiplication

Addition of matrices of the same size is defined as matrix whose each element is the sum of the corresponding elements of the two matrices. $$\begin{array}{r}(A+B{)}_{i,j}=A_{i,j}+B_{i,j}\end{array}$$

Thus, the matrix of sum of linear maps is the sum of matrices of those linear maps $$\begin{array}{r}\mathcal{M}(S+T)=\mathcal{M}(S)+\mathcal{M}(T)S,T\in \mathcal{L}(V,W)\end{array}$$

The scalar multiplication with $\lambda \in \mathcal{F}$ is same as multiplying each element of the original matrix with $\lambda$ $$\begin{array}{r}(\lambda A{)}_{i,j}=\lambda (A_{i,j})\end{array}$$

Thus, the matrix of a scalar times a linear map is same as the scalar times the matrix of the linear map $$\begin{array}{r}\mathcal{M}(\lambda T)=\lambda \mathcal{M}(T)T\in \mathcal{L}(V,W)\end{array}$$

We will denote the set of all $m-$by$-n$ matrices with elements in $\mathcal{F}$ by ${\mathcal{F}}^{m,n}$. This set of all matrices is also a vector space on the addition and scalar multiplication rules for matrices defined above. The basis for such space is the collection of all matrices with all but one element zeros, and one element 1. There are $mn$ such matrices meaning dim ${\mathcal{F}}^{m,n}=mn$.

Matrix Multiplication

Matrix multiplication is important to define in order to work with product of linear maps. We wish to have the following hold $$\begin{array}{r}\mathcal{M}(ST)=\mathcal{M}(S)\mathcal{M}(T)T\in \mathcal{L}(U,V),S\in \mathcal{L}(V,W)\end{array}$$ For two matrices $A$ and $B$ of sizes $m$-by-$n$ and $n$-by-$p$ respectively, their matrix multiplication is defined as $$\begin{array}{r}(AB{)}_{i,j}=\sum _{k=1}^p{A}_{i,k}{B}_{k,j}\end{array}$$ Which is the sum product of the $i^{th}$ row of $A$ and $j^{th}$ column of $B$. Notice that the matrices have one dimension same. It is a necessary condition for the multiplication to be valid and also implies that the range of $T$ is same as domain of $S$. Furthermore, $AB$ is of size $m$-by-$p$.

Matrix multiplication is not commutative (since the multiplication may not be defined or the products may not be the same otherwise), but is distributive and associative.

Inverse Linear Map

A linear map $T\in \mathcal{L}(V,W)$ is invertible if there exists $T\in \mathcal{L}(W,V)$ such that $ST$ is the identity map on $V$ and $TS$ is the identity map on $W$. $S$ is said to be the inverse of $T$ with $ST=I$ and $TS=I$, and there exists a unique inverse for a linear map (can be proven by contradiction).

We denote the unique inverse of a linear map $T\in \mathcal{L}(V,W)$ with $T^{-1}\in \mathcal{L}(W,V)$ and they satisfy $T{T}^{-1}=I$ and $T^{-1}T=I$.

A linear map is invertible if and only if it is both injective and surjective. To prove this, we first assume the linear map to be invertible and show that the latter part of statement is true and vice versa.

Two vector spaces are isomorphic if there is an isomorphism (an invertible linear map) from one vector space onto the other.

Operator

A linear map from a vector space onto itself is called an operator. $\mathcal{L}(V)$ denotes the set of all operators from $V$ onto itself. It is the same notation as $\mathcal{L}(V)=\mathcal{L}(V,V)$.

On a finite dimensional vector space $V$, any linear map $T\in \mathcal{L}(V)$ is \textbf{invertible}, \textbf{injective} and \textbf{surjective}.

Rank of a Matrix

Suppose $A\in {\mathcal{F}}^{m,n}$. Row and column ranks are two non negative integers defined as

• Row Rank: Dimension of the span of the rows of $A$ in ${\mathcal{F}}^{1,n}$
• Column Rank: Dimension of the spaan of the columns of $A$ in ${\mathcal{F}}^{m,1}$

But for any $A\in {\mathcal{F}}^{m,n}$, row rank $=$ column rank (can be proved using duality). Hence, we simply use the word rank to denote the column rank of a matrix.

Rank is defined for a matrix, whereas dimension is defined for a vector space. This is the important distinction between the two. The rank can be found by counting the number of non-zero rows in the row echelon form of a matrix (see here).

Rank-Nullity Theorem

We restate the theorem from fundamental theorem of linear maps using the concepts of dimension and ranks. For any vector spaces $V$ and $W$, and linear map $T\in \mathcal{L}(V,W)$ $$\begin{array}{r}\text{rank}(T)+\text{Nullity}(T)=\text{dim}(V)\end{array}$$ where dim$(V)$ is the number of basis vectors of the vector space $V$. Usually, the dim$(V)$ is the number of columns in the matrix $T$. For a matrix, the dimension of the null space will be determined by the number of basis vectors of the solution set to $Ax=0$.

Block Matrix

Sometimes, block matrices can help simplify cumbersome operations. For the purpose of demonstration, consider a matrix $A$ of dimensions $n\times n$, which we break into four blocks: $B$ of size $m\times m$, $C$ of size $m\times n-m$, $D$ of size $n-m\times m$, and $E$ of size $n-m\times n-m$. Then, $$\begin{array}{r}A=\left[\begin{array}{cc}B& C\\ D& E\end{array}\right]\end{array}$$ Then, the calculation of $A^2$ is $$\begin{array}{rl}{A}^2& =\left[\begin{array}{cc}B& C\\ D& E\end{array}\right]\left[\begin{array}{cc}B& C\\ D& E\end{array}\right]=\left[\begin{array}{cc}{B}^2+CD& BC+CE\\ DB+ED& DC+E^2\end{array}\right]\end{array}$$

We obtained $B^2$ using $2\times 2$ multiplication on the four elements of the block matrix. All the multiplications agree in terms of the dimensions. This technique is especially useful when the matrix can be broken in such a way that individual matrix are identity matrix, diagonal matrix etc.

Orthogonal Matrix

Orthogonal or \textbf{orthonormal} matrix is a matrix whose rows and columns are orthonormal vectors. An orthogonal matrix $Q$ will satisfy $$\begin{array}{rl}Q{Q}^T& =Q^{T}Q=I\\ Q^{-1}& =Q^T\\ det(Q)& =\pm 1\end{array}$$

The last one follows from the fact that $$\begin{array}{r}1=det(I)=det(Q{Q}^T)=det(Q{)}^2\end{array}$$

Orthogonal matrix play an important role in QR decomposition and SVD.

Determinant

Determinant is defined for square matrices and represents a transformation from ${\mathbb{R}}^{n\times n}\to \mathbb{R}$. Its defined as follows for $2\times 2$ and $3\times 3$ matrices $$\begin{array}{rl}A& =\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]\\ det(A)& =ac-bd\\ A& =\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right]\\ det(A)& =a(ei-hf)-b(di-gf)+c(dh-ge)\end{array}$$

The term outside the bracket is the entry from the first row, and the entry inside the brackets is the determinant of the $n-1\times n-1$ matrix formed by not considering the first row and the column corresponding to the entry outside the bracket. We put alternating $+$ and $-$ signs before each term. This way the definition extends to any $n\times n$ matrix.

If the determinant of a matrix is non zero, then the matrix has full rank, or all its rows and columns are linearly independent, and the matrix is invertible.