Matrix calculus pops up connected societal media from clip to time, chiefly owed to the accrued fame of instrumentality learning. It lets you differentiate functions of vector aliases matrix variables moreover if you are not a instrumentality of automatic differentiation. It involves each kinds of analyzable rules, depending connected whether your vectors are statement aliases file vectors, and it gets moreover worse for matrices. Let’s not speak of tensors.
You do not request matrix calculus. When physists started moving pinch tensors almost a period ago, they figured retired the easy measurement to do this: alternatively of trying to travel up pinch a strategy for tensor differentiation \(\partial A / \partial B\), conscionable constitute retired the formulas for \(A_{ijk}\) and \(B_{ijk}\) and usage mean differentiation.
The advantage is that this strategy ever works, you only request to cognize really to do mean differentiation, and it’s usually quicker, too. Matrix calculus only useful for formulas successful very circumstantial form, and you request to memorize a bunch of caller rules to use it. Even then, it doesn’t really prevention overmuch time, if any. Just constitute retired the indices!
Consider the neuron activation function:
\[\mathsf{activation}(x) = \max(0, Ax + b)\]The max present is to beryllium interpreted elementwise:
\[\mathsf{activation}(x)_i = \max(0, (Ax)_i + b_i) = \begin{cases} 0 & \text{ if } (Ax)_i + b_i < 0\\ (Ax)_i + b_i & \text{ otherwise} \end{cases}\]Let’s conscionable constitute retired the matrix multiplication:
\[\mathsf{activation}(x)_i = \begin{cases} 0 & \text{ if } (Ax)_i + b_i < 0\\ \sum_j A_{ij}x_j + b_i & \text{ otherwise} \end{cases}\]Let’s differentiate:
\[\mathsf{activation}(x)_i' = \begin{cases} 0 & \text{ if } (Ax)_i + b_i < 0\\ \sum_j (A_{ij}'x_j + A_{ij}x_j' + b_i') & \text{ otherwise} \end{cases}\]Note that we haven’t said pinch respect to what we’re differentiating, we conscionable utilized the \(y'\) notation. We person 3 possibilities:
- Differentiate pinch respect to \(x_k\)
- Differentiate pinch respect to \(A_{kl}\)
- Differentiate pinch respect to \(b_k\)
To do so, we simply group the corresponding primed adaptable to \(1\) and the different primed variables to \(0\). For lawsuit to differentiate pinch respect to \(A_{kl}\) we group \(A_{kl}' = 1\) and each the different \(A_{ij}' = 0\) and \(x_i'= 0\) and \(b_i' = 0\):
\[\frac{\partial \mathsf{activation}(x)_i}{\partial A_{kl}} = \begin{cases} 0 & \text{ if } (Ax)_i + b_i < 0\\ x_l & \text{ otherwise} \end{cases}\]That’s it. For the others:
\[\frac{\partial \mathsf{activation}(x)_i}{\partial x_{k}} = \begin{cases} 0 & \text{ if } (Ax)_i + b_i < 0\\ A_{ik} & \text{ otherwise} \end{cases}\] \[\frac{\partial \mathsf{activation}(x)_i}{\partial b_{k}} = \begin{cases} 0 & \text{ if } (Ax)_i + b_i < 0 \\ 1 & \text{ if } one = k\\ 0 & \text{ otherwise} \end{cases}\]We sewage 3 for the value of one. Compare pinch the matrix calculus derivation. Can you decipher their last consequence for the derivative pinch respect to \(w\)? It’s trickier than you think.
Just constitute retired the indices!
English (US) · 
Indonesian (ID) · 
