Types and pattern matching in Python

Lecture 2: dataclasses, unions, match and a type checker.

Programming Languages Lab · University of Pisa
Vincenzo Ciancia, CNR-ISTI
vincenzo.ciancia@isti.cnr.it · github.com/vincenzoml/PLDW

The exercise you took home


def calc(text: str) -> int: ...

calc("2 + 5 - 3")      # 4
calc("10 - 4 - 3")     # 3: left to right
calc("7")              # 7
calc("2 + + 3")        # error: not well formed
calc("2 + x")          # error

Lecture 1B, exercise 4

Input a string, output a number. Only + and -, no parentheses, evaluated left to right.

Malformed input raises ValueError.

One solution: a loop


def calc(text: str) -> int:
    total = 0
    op = "+"
    expect_number = True
    for token in text.split():
        if expect_number:
            n = int(token)
            if op == "+":
                total = total + n
            else:
                total = total - n
            expect_number = False
        elif token == "+" or token == "-":
            op = token
            expect_number = True
        else:
            raise ValueError(f"expected + or -, got '{token}'")
    if expect_number:
        raise ValueError("expected a number at the end")
    return total

One possible solution

A loop keeps a running total and updates it at each token.

expect_number says what the next token must be: a number, or an operator.

calc("10 - 4 - 3") gives 3.

On bad input, int("x") already raises ValueError; the other errors are raised explicitly.

Grow the language

Now add *, which binds tighter. And parentheses.

This is hard to do with the loop.

It only keeps a number, the total so far, and not the expression.

From a running total to a tree

the loop: a number at each step 10 − 4 6 − 3 3 the tree of 10 - 4 - 3 − − 3 10 4

A tree can be evaluated, and also printed, simplified, checked or translated.

Precedence and parentheses determine which tree we build. A running total has no structure.

In lecture 1A we wrote one case per kind of node, so we need nodes.

The same loop, building a tree


def calc_tree(text: str) -> Expr:
    tree: Expr | None = None
    op = "+"
    expect_number = True
    for token in text.split():
        if expect_number:
            n = Number(float(token))
            if tree is None:
                tree = n
            elif op == "+":
                tree = BinaryOp(tree, "+", n)
            else:
                tree = BinaryOp(tree, "-", n)
            expect_number = False
        elif token == "+" or token == "-":
            op = token
            expect_number = True
        ...  # errors: as in calc
    return tree

The loop now builds nodes instead of numbers; evaluate does the arithmetic.

evaluate(calc_tree("10 - 4 - 3"), {}) gives 3.0.

Each step puts the tree built so far on the left. This is how left-to-right evaluation appears in the tree.

Still no *. Next lecture we parse strings properly, with a parser generator; today we focus on the tree.

To extend the language we need a tree whose structure can be checked by a tool.

Python does not check types when it runs. We can still write the types down, and a separate program can check them.

Type hints

Type hints are not checked at run time

Who reads the type hints

calc.py with hints Python python3 calc.py a type checker Pylance · mypy · pyright runs it, ignores the hints reads it, does not run it, reports type errors

Recall that add("2", "3") returns "23". Hints are ignored so that annotated and unannotated code can be mixed.

Three ways to run a type checker

IN VS CODE

The Python extension includes Pylance, which is based on pyright. Errors are underlined in red as you edit.

MYPY

pip install mypy, then
python3 -m mypy file.py

PYRIGHT

The checker used by VS Code, from the command line: pyright file.py

mypy output, calling add with two strings


checker_catches.py:19: error: Argument 1 to "add" has incompatible type "str"; expected "int"

Why an interpreter needs a type checker

parse BinaryOp(…, "^", …) simplify evaluate crash far from the bug without a checker: an error at run time, only for some inputs with a checker: an error at the line that builds the tree

All functions of an interpreter take a tree as input. A malformed tree causes an error later, not where it is built.

Dataclasses and unions

Defining the nodes of the tree

How to represent a node

A DICTIONARY

{"kind": "num", "value": 2.0}

A typo in a key gives a KeyError at run time. The checker only knows that keys are strings.

A TUPLE

("num", 2.0)

Compact, but the meaning of each position is not written anywhere.

A CLASS WITH __init__

Works, but takes about ten lines per node, and an interpreter has many nodes.

@dataclass

Generates those lines from the list of fields. Fields have names and types that the checker uses.

The expression tree as dataclasses


from __future__ import annotations
from dataclasses import dataclass
from typing import Literal, assert_never

type Op = Literal["+", "-", "*", "/"]

@dataclass(frozen=True)
class Number:
    value: float

@dataclass(frozen=True)
class Variable:
    name: str

@dataclass(frozen=True)
class BinaryOp:
    left: Expr
    operator: Op
    right: Expr

type Expr = Number | Variable | BinaryOp

The tree of lecture 1A, except that operator has type Op instead of str.

left: Expr uses Expr before it is defined. The first line tells Python not to evaluate annotations; Python 3.14 does this by default.

The next slides go through this code.

What @dataclass writes for you


# 2 + (x * 3)
e = BinaryOp(Number(2.0), "+",
             BinaryOp(Variable("x"), "*", Number(3.0)))

print(Number(2.0))
# Number(value=2.0)

# Equality is structural
print(Number(2.0) == Number(2.0))      # True
print(e == BinaryOp(Number(2.0), "+",
            BinaryOp(Variable("x"), "*", Number(3.0))))
# True

__init__

fields in order, or by name.

__repr__

a readable printout, useful to debug a parser.

__eq__

compares nodes field by field, so parse("2 + x * 3") == e is a complete test.

Some calls are split over two lines here to fit; the file has them on one.

Why frozen=True


e = BinaryOp(Number(2.0), "+", Variable("x"))
try:
    e.operator = "-"
except AttributeError as err:
    print("AttributeError:", err)
# AttributeError: cannot assign to field 'operator'

# Frozen nodes are hashable: set members, dict keys
seen: set[Expr] = {Number(2.0), Number(2.0),
                   Variable("x")}
print(len(seen))  # 2

With mutable nodes, evaluate could modify the tree it visits, and two parts of a program sharing a node could change it under each other.

An interpreter does not need to change trees in place: each transformation builds a new tree.

Frozen nodes are also hashable, which is useful for caching.

A class hierarchy or a union

a base class, three subclasses Expr Number Variable BinaryOp ??? can be added from any file a union type Expr = Number | Variable | BinaryOp and no other types.

A union is a closed list, so the checker can report a match that misses a case. In this course every tree is a union of dataclasses.

Product and sum

PRODUCT · AND

\( \mathit{BinaryOp} = \mathit{Expr} \times \mathit{Op} \times \mathit{Expr} \)

A dataclass: a BinaryOp holds a left and an operator and a right.

SUM · OR

\( \mathit{Expr} = \mathit{Number} + \mathit{Variable} + \mathit{BinaryOp} \)

A union: an Expr is a Number or a Variable or a BinaryOp.

OCaml, Haskell and Rust have a single construct for both. Python uses dataclasses for products and the type statement for unions.

The type statement


type Expr = Number | Variable | BinaryOp
type Env = dict[str, float]

In lecture 1B we wrote

Expr = Num | Add, a plain assignment, evaluated immediately. For this reason Add had left: "Expr" in quotes.

The right-hand side of a type statement is evaluated lazily, only when needed.

So Expr and BinaryOp can refer to each other, in any order.

With = or the older Union[…] the right-hand side is evaluated immediately, which causes problems with recursive types. type is also handy for names like Env, used in many signatures.

Literal: a union of constants


type Op = Literal["+", "-", "*", "/"]

bad = BinaryOp(Number(2.0), "^", Number(3.0))
print(bad)       # Python builds it

mypy


checker_catches.py:33: error: Argument 2 to "BinaryOp"
  has incompatible type "Literal['^']";
  expected "Literal['+', '-', '*', '/']"

With operator: str, a tree containing "^" gives an error only when evaluate reaches it.

With operator: Op, the checker reports it at the line that builds the node.

Op is a union of four constants, as Expr is a union of three classes.

Pattern matching

Inspecting nodes with match

Four kinds of pattern


def evaluate(expr: Expr, env: Env) -> float:
    match expr:
        case Number(value):
            return value
        ...
        case BinaryOp(left, operator, right):
            lv = evaluate(left, env)
            rv = evaluate(right, env)
            match operator:
                case "+":
                    return lv + rv
                ...
                case _:
                    assert_never(operator)
        case _:
            assert_never(expr)

CLASS · Number(value)

checks that it is a Number and extracts its field.

LITERAL · "+"

compares with ==.

CAPTURE · value, left

matches anything, binds the name.

WILDCARD · _

matches anything, binds nothing.

By position, or by keyword


print(BinaryOp.__match_args__)
# ('left', 'operator', 'right')

def is_sum(expr: Expr) -> bool:
    match expr:
        case BinaryOp(operator="+"):
            return True
        case _:
            return False

print(is_sum(e), is_sum(Number(1.0)))  # True False

BinaryOp(left, operator, right) works because @dataclass stores the order of the fields in __match_args__.

With keywords, only the named fields are checked. This is clearer for nodes with many fields. The two forms can be combined.

A common mistake: what does this print?


ZERO = 0.0

def is_zero(expr: Expr) -> bool:
    match expr:
        case Number(ZERO):
            return True
        case _:
            return False

print(is_zero(Number(5.0)))

True.

A plain name in a pattern is always a capture: ZERO is bound to 5.0, not compared. Use a literal instead: case Number(0.0).

Only dotted names, such as math.pi, are compared by value. Also, case Number: without parentheses captures everything; the parentheses are required.

Nested patterns: a simplifier


def simplify(expr: Expr) -> Expr:
    match expr:
        case BinaryOp(Number(0), "+", other) | BinaryOp(other, "+", Number(0)):
            return simplify(other)
        case BinaryOp(Number(1), "*", other) | BinaryOp(other, "*", Number(1)):
            return simplify(other)
        case BinaryOp(Number(0), "*", _) | BinaryOp(_, "*", Number(0)):
            return Number(0.0)
        case BinaryOp(Number(a), operator, Number(b)) if operator != "/" or b != 0:
            return Number(evaluate(expr, {}))
        case BinaryOp(left, operator, right):
            return BinaryOp(simplify(left), operator, simplify(right))
        case Number() | Variable():
            return expr
        case _:
            assert_never(expr)

The first three cases implement the rules 0 + e = e, 1 · e = e, 0 · e = 0.

New constructs in this example

OR-PATTERN · |

A(other) | B(other)

tries the alternatives in order. All must bind the same names, so other is defined whichever one matched.

GUARD · if

… if b != 0

a condition checked after the pattern matches, using the names it bound.

ORDER

case BinaryOp(left, …)

cases are tried from top to bottom. If the general case came first, the specific ones would never run.

simplify, example 1: 0 + (x * 1)

+ 0 * x 1 simplify x

1. The root is 0 + …. The first case matches, with other = x * 1. It calls simplify(x * 1).

2. x * 1 matches the second case, with other = x. It calls simplify(x).

3. x is a Variable: the case Number() | Variable() returns it unchanged.

Two rules applied, one after the other, through the recursive calls.

simplify, example 2: (2 + 3) * y

* + y 2 3 simplify * 5 y

1. The root is (2 + 3) * y. No rule matches it: no 0 or 1, and the left child is not a number.

2. So the general BinaryOp(left, operator, right) case runs. It rebuilds the node with simplify(2 + 3) and simplify(y).

3. 2 + 3 has two numbers: the fourth case computes it and returns Number(5.0). y stays y.

One pass only: if a rule becomes applicable after the children are simplified, it is not tried again.

Without match: isinstance

One rule, without match


def drop_zero_plus(expr: Expr) -> Expr:
    if (
        isinstance(expr, BinaryOp)
        and expr.operator == "+"
        and isinstance(expr.left, Number)
        and expr.left.value == 0
    ):
        return expr.right
    return expr

The same rule, as a pattern


case BinaryOp(Number(0), "+", other):
    return simplify(other)

Both do the same test. With isinstance the shape is split over four conditions; the pattern writes it directly.

Sequence patterns: the shape of a list


def describe(items: list[int]) -> str:
    match items:
        case []:
            return "empty"
        case [x]:
            return f"just {x}"
        case [first, *rest]:
            return f"{first} then {len(rest)} more"
        case _:
            raise AssertionError("unreachable")

describe([1, 2, 3])        # '1 then 2 more'

match "10 - 4 - 3".split():
    case [*init, op, last]:
        print(init, op, last)  # ['10', '-', '4'] - 3

Lists are matched by length and position.

*rest collects the remaining elements in a new list, possibly empty. A starred name can appear anywhere in the pattern, but only once.

Why the last case _? The checker does not reason about list lengths, so without it, it reports a missing return.

Exhaustiveness

Finding missing cases before running

A common bug: a missing case


def evaluate_forgetful(expr: Expr, env: Env) -> float:
    match expr:
        case Number(value):
            return value
        case BinaryOp(left, _, right):
            return (evaluate_forgetful(left, env)
                    + evaluate_forgetful(right, env))
        # no case for Variable

evaluate_forgetful(
    BinaryOp(Number(1.0), "+", Variable("x")),
    {"x": 2.0})
# TypeError: unsupported operand type(s) for +:
#            'float' and 'NoneType'

When you add a kind of node, it is easy to forget to update a function. Python does not complain at the match.

No case matches, so the function returns None. The error appears later, in the addition, and does not mention the missing case.

Narrowing: how the checker follows the cases

the type of expr, line by line Number | Variable | BinaryOp case Number(value) did not match Variable | BinaryOp case BinaryOp(…) did not match Variable end of function checker_catches.py:49: error: Missing return statement With a class hierarchy the remaining type is just Expr: no error.

Exhaustiveness with assert_never


from typing import assert_never

def evaluate(expr: Expr, env: Env) -> float:
    match expr:
        ...
        case _:
            assert_never(expr)

mypy, with the Variable case deleted


error: Argument 1 to "assert_never" has
  incompatible type "Variable"; expected "Never"

Its parameter has type Never, the type with no values.

The call type-checks only if the cases above have removed every alternative. If it is reached at run time, it raises an error.

The raise ValueError of lecture 1A finds the same bug only at run time. With assert_never the checker reports it and names the missing case.

Choices that let the checker find mistakes

01a union of dataclasses
Instead of a class hierarchy. The cases are a closed list, so the checker can find a match that misses one.
02frozen=True
Instead of mutable nodes. The checker reports any assignment to a field of a node.
03Literal operators
Instead of str. A wrong operator is reported where the tree is built.
04assert_never
Instead of raise ValueError. The checker reports a missing case before running.

In practice: after adding a node to the union, run the checker; it lists the functions that need a new case.

In class

Exercises

Strings: what exercise 1 needs


a = "2.0"
b = "x"

print("(" + a + " + " + b + ")")   # (2.0 + x)
print(f"({a} + {b})")               # (2.0 + x)

v = 3.0
print("v is " + str(v))             # v is 3.0
print(f"v is {v}")                  # v is 3.0

+ joins two strings. Both sides must be strings: "v is " + v is a TypeError.

str(v) turns a number into its text.

An f-string, f"...", does both: each {expression} is evaluated and its text inserted.

A function can call itself on the children of a node, as evaluate does.

Exercise 1 · Print a tree


def show(expr: Expr) -> str: ...

show(e)
# '(2.0 + (x * 3.0))'
show(Number(7.0))
# '7.0'
show(left_first)    # 10 - 4 - 3
# '((10.0 - 4.0) - 3.0)'

Level 1 of 3

  1. Write it in code/lesson-02/exercises.py: running it checks the examples of these slides.
  2. One case per kind of node; an f-string for BinaryOp, with parentheses around every operation.
  3. End with case _: assert_never(expr), and run mypy: no errors.

Exercise 2 · Adding a node


@dataclass(frozen=True)
class Negate:
    operand: Expr

type Expr = Number | Variable | BinaryOp | Negate

evaluate(Negate(Variable("x")), {"x": 4.0})  # -4.0
evaluate(Negate(Negate(Number(3.0))), {})    # 3.0

Level 2 of 3

  1. Add Negate to the union. Run mypy before touching any function, and write down what it reports.
  2. Fix each function it names, starting with evaluate and your show.
  3. In Python, -v is the negation of v.

Exercise 3 · Derivatives: the rules


d/dx  c      =  0          a number
d/dx  x      =  1
d/dx  y      =  0          another variable
d/dx  f + g  =  f' + g'
d/dx  f - g  =  f' - g'
d/dx  f * g  =  f' * g + f * g'

The derivative of an expression is another expression.

Each rule says what to do for one kind of node. The rules for +, - and * use the derivatives of the children f and g, written f' and g': a recursive call, as in evaluate.

The result is a tree, so it can be printed with show or evaluated.

Exercise 3 · Derivatives: what to write

Write derive(expr: Expr, x: str) -> Expr. It computes no number: it builds a new tree by applying the rules. Printed with show:

expressionshow(derive(expression, "x"))rule usedin maths
x + 5(1.0 + 0.0)f' + g'1
3 * x((0.0 * x) + (3.0 * 1.0))f' * g + f * g'3
x * x((1.0 * x) + (x * 1.0))f' * g + f * g'2x

Level 3 of 3

One case per kind of node, one rule per case; for BinaryOp, a nested match on the operator. For /, raise a ValueError. End with assert_never and run mypy.