Loading
0x10Lesson 2 of 17

Tensors, shapes and broadcasting

Store data as multi-dimensional arrays, reason about their shapes, and let broadcasting do the loops.

26 min 7-question quiz 2 code exercises
By the end of this lesson you can
  • Describe tensors by rank and shape, including the batch dimension
  • Predict the result shape of matrix multiplication, broadcasting and reductions
  • Reshape, transpose and flatten tensors without losing track of the data

Neural networks speak one language: tensors - arrays of numbers with any number of dimensions. In numpy (and PyTorch, which copies numpy’s style) a tensor has a shape, a tuple giving the size of each dimension:

RankExampleShape
0 (scalar)a loss value()
1 (vector)one doodle’s 25 pixels(25,)
2 (matrix)a batch of 32 doodles(32, 25)
3a batch of 32 grayscale 5×5 images(32, 5, 5)
432 color images, 3 channels, 64×64(32, 3, 64, 64) in PyTorch

The first dimension is almost always the batch: networks process many examples at once, because one big matrix multiplication is much faster than many small ones.

shapes.py
1import numpy as np
2
3doodles = np.arange(2 * 3 * 4).reshape(2, 3, 4)   # 2 doodles, 3 rows, 4 columns
4print(doodles.shape, doodles.ndim, doodles.size)
5flat = doodles.reshape(2, -1)                      # -1: "work this one out"
6print(flat.shape)
7print(flat[1, :5])
8print(doodles.transpose(0, 2, 1).shape)
9print(doodles.sum(axis=0).shape, doodles.sum(axis=(1, 2)))
Output
(2, 3, 4) 3 24
(2, 12)
[12 13 14 15 16]
(2, 4, 3)
(3, 4) [ 66 210]

Reductions like sum, mean and max take an axis: that dimension disappears from the shape. Pass keepdims=True to keep it as size 1 - handy when you want to divide by the result.

reshape never moves data; it reinterprets the same numbers in a new shape, filling the last dimension first. transpose swaps dimensions around.

Matrix multiplication

The workhorse of every layer is @, matrix multiplication. Shapes must line up on the inside, and the outside dimensions survive:

(N×K)  @  (K×M)  →  (N×M)(N \times K) \; @ \; (K \times M) \;\rightarrow\; (N \times M)

For a layer: inputs (batch, in_features) @ (in_features, out_features) gives (batch, out_features). Each output is a weighted sum - one dot product per row and column.

Broadcasting

Element-wise operations between tensors of different shapes broadcast: numpy lines the shapes up from the right, and a dimension of size 1 (or a missing one) stretches to match. No data is copied.

ShapesResult
(32, 10) + (10,)(32, 10) - one bias per column, added to every row
(32, 10) − (32, 1)(32, 10) - one value per row
(3, 1) × (1, 4)(3, 4) - an outer product
(32, 10) + (32,)error: 10 and 32 don’t match
broadcasting.py
1import numpy as np
2
3X = np.array([[1.0, 2.0, 3.0],
4              [4.0, 5.0, 6.0]])           # 2 examples, 3 features
5bias = np.array([10.0, 20.0, 30.0])
6print(X + bias)
7row_means = X.mean(axis=1, keepdims=True)  # shape (2, 1)
8print(X - row_means)
9W = np.ones((3, 4))
10print((X @ W).shape)
11try:
12    X + np.array([1.0, 2.0])
13except ValueError as error:
14    print("Error:", str(error).split(" with ")[0])
Output
[[11. 22. 33.]
 [14. 25. 36.]]
[[-1.  0.  1.]
 [-1.  0.  1.]]
(2, 4)
Error: operands could not be broadcast together

Try it

Up to 15 XP

Will it broadcast?

Line the shapes up from the right: each pair of dimensions must be equal, or one of them must be 1 (or missing). For @, the inner dimensions must match.

0 of 7 sortedScore 0/0
  • “(64, 128) + (128,)”

  • “(64, 128) + (64,)”

  • “(64, 1) * (1, 10)”

  • “(8, 3) @ (3, 5)”

  • “(8, 3) @ (8, 3)”

  • “(16, 3, 32, 32) + (3, 1, 1)”

  • “(5, 4) - (5, 4).mean(axis=1)”

Key takeaways

  • Tensors are n-dimensional arrays; their shape lists each dimension’s size, batch first.

  • @ multiplies (N, K) by (K, M) into (N, M) - the core of every layer.

  • Broadcasting aligns shapes from the right and stretches size-1 dimensions.

  • Reductions remove the axis you name; keepdims=True keeps it as size 1.

Lesson quiz

7 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Standardize the doodle features

+25 XP

Each input line is one doodle’s features (same count on every line). Standardize each column: subtract the column mean and divide by the column standard deviation (np.std, the default). Print each row with values to 2 decimals separated by spaces, then the shape of the result.

  • Three doodles
  • Four doodles
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

A layer for a whole batch

+25 XP

Each input line is one example with 3 features. Stack them into a batch X, then compute a layer’s outputs X @ W + b with the given W (3 inputs → 2 outputs) and b, for all examples at once (no loop over rows). Print the shapes of X and of the output, then each output row to 1 decimal.

  • Two examples
  • Three examples
main.py
Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…