Johnny's Knowledge

Pytorch

Properties1
tagsdeep-learning, pytorch, tools

Learning Plan

The goal is to connect Deep Learning theory with step-by-step implementation. For CNN, RNN, and LSTM, the useful path is to start from simple neural networks, type the code by hand, then map each line back to the mathematical concept.

Knowing neural-network theory is not the same as being able to implement it. If I truly understand a model, I should be able to code it from scratch and explain it to a non-expert.

Practice sequence:

  • Binary classification with a simple neural network
  • Binary classification with a CNN
  • Backpropagation and manual parameter updates
  • Tensor reshaping, batching, and projection operations

Theoretical reference:

Parameters

named_parameters() lists a model’s trainable parameters by name. It is useful for inspecting layers, debugging gradients, and deciding which parameters should be optimized.

Backpropagation

In PyTorch, loss.backward() computes gradients for all tensors in the computation graph that have requires_grad=True. This is the practical bridge between the loss function and optimization.

model = nn.Sequential(
    nn.Linear(16, 8),
    nn.Linear(8, 4),
    nn.Linear(4, 2),
)
 
prediction = model(sample)
criterion = CrossEntropyLoss()
loss = criterion(prediction, target)
loss.backward()
 
lr = 0.001
 
weight = model[0].weight
weight_grad = model[0].weight.grad
weight = weight - lr * weight_grad
 
bias = model[0].bias
bias_grad = model[0].bias.grad
bias = bias - lr * bias_grad

For a three-layer Sequential model, gradients are available at:

model[0].weight.grad
model[0].bias.grad
model[1].weight.grad
model[1].bias.grad
model[2].weight.grad
model[2].bias.grad

Common Tensor Operations

F.one_hot()

F.one_hot() converts class indices into one-hot vectors, which is useful when a label needs to be represented as a categorical vector.

import torch
import torch.nn.functional as F
 
one_hot_pytorch = F.one_hot(torch.tensor(1), num_classes=3)
# tensor([0, 1, 0])

torch.stack

torch.stack combines a sequence of tensors along a new dimension. It is commonly used to merge multiple batch outputs into one tensor.

view()

view() reshapes a tensor without copying memory when the tensor layout allows it.

import torch
 
a = torch.arange(1, 17)
a_view = a.view(4, 4)
a_view_neg_1 = a.view(4, -1)
a_view_neg_2 = a.view(-1, 4)

The -1 dimension asks PyTorch to infer that dimension automatically. If the tensor has 16 elements, both view(4, -1) and view(-1, 4) produce a 4 x 4 tensor.

torch.einsum()

torch.einsum("bct,bcd->bdt", x, weights) performs a Batched Matrix Multiplication:

  • x has shape (batch_size, channels, time_steps), written as (b, c, t).
  • weights has shape (batch_size, channels, depth), written as (b, c, d).
  • The output has shape (batch_size, depth, time_steps), written as (b, d, t).

The shared c dimension is summed over. Each time step in x is projected from the channel dimension into the depth dimension.

Equivalent code without einsum:

conditioning = torch.matmul(weights.transpose(1, 2), x)
# (b, d, c) @ (b, c, t) -> (b, d, t)

Example:

x = torch.rand(1, 128, 512)
weights = torch.rand(1, 128, 128)
output = torch.einsum("bct,bcd->bdt", x, weights)
 
output.shape == (1, 128, 512)

Training Utilities

tqdm

tqdm shows progress for an iterable, which is useful during model training loops.

from tqdm import tqdm
 
for i in tqdm(range(10000)):
    ...

References

Backlinks