Lecture 3. The calculator by hand, then parsing with Lark.
Programming Languages Lab · University of Pisa
Vincenzo Ciancia, CNR-ISTI
vincenzo.ciancia@isti.cnr.it · github.com/vincenzoml/PLDW
Lecture 1B, exercise 4
def calc(text: str) -> int: ...
calc("2 + 5 - 3") # 4
calc("10 - 4 - 3") # 3: left to right
calc("7") # 7
calc("2 + + 3") # error: not well formed
calc("2 + x") # error
+ and - only; no parentheses. Compute from left to right.text.split() gives the pieces.ValueError that says what is wrong.Then make it work without spaces too: "2+5-3". The solution is in code/lesson-03/calculator.py.
def calc(text: str) -> int:
tokens = tokenize(text)
return compute(tokens)
calc("12+5 - 3")
# tokenize: ['12', '+', '5', '-', '3']
# compute: 4
calc("2 + x")
# tokenize: ValueError
# unexpected character 'x' at position 4
tokenize(text) → list[str]
Cuts the text into numbers and operators, skipping spaces.
compute(pieces) → int
Checks that numbers and operators alternate, and keeps a running total from left to right.
number(piece) → int
Called by compute on each number: checks that it is made of digits, then converts it.
The first version uses text.split() in place of tokenize. The next slides go through the functions one at a time.
def compute(pieces: list[str]) -> int:
if pieces == []:
raise ValueError("empty expression")
total = number(pieces[0])
for i in range(1, len(pieces), 2):
op = pieces[i]
if op not in ("+", "-"):
raise ValueError(f"expected + or -, found {op!r}")
if i + 1 == len(pieces):
raise ValueError(f"missing number after {op!r}")
n = number(pieces[i + 1])
if op == "+":
total = total + n
else:
total = total - n
return total
def calc_with_spaces(text: str) -> int:
return compute(text.split())
A well-formed list of pieces alternates: number, operator, number, …, number.
Numbers are at even positions, operators at odd positions. The loop visits the operators: range(1, len(pieces), 2).
total holds the value of everything read so far.
10 - 4 - 3, step by step
"10 - 4 - 3".split() # ['10', '-', '4', '-', '3']
| i | op | n | total |
|---|---|---|---|
| before the loop | 10 | ||
| 1 | - | 4 | 6 |
| 3 | - | 3 | 3 |
Left to right means (10 - 4) - 3, which is 3. Grouping on the right, 10 - (4 - 3), would give 9.
int("+3") # 3
int("1_000") # 1000
int(" 7 ") # 7
DIGITS = "0123456789"
def number(piece: str) -> int:
if piece == "" or any(c not in DIGITS for c in piece):
raise ValueError(f"expected a number, found {piece!r}")
return int(piece)
int reads Python's integers, which are not the same as ours.
With int alone, "2 + +3" would give 5: the piece +3 is accepted as a number.
So we check that the piece is made of digits, and only then call int.
str.isdigit() also accepts characters like ², on which int fails.
| input | pieces | message |
|---|---|---|
| 2 + + 3 | ['2', '+', '+', '3'] | expected a number, found '+' |
| 2 + x | ['2', '+', 'x'] | expected a number, found 'x' |
| 2 3 | ['2', '3'] | expected + or -, found '3' |
| 2 + | ['2', '+'] | missing number after '+' |
| (empty) | [] | empty expression |
eval(text) is not a solution: it evaluates any Python expression, including 2 * 3 and +3, and its errors are Python's, not ours.
def tokenize(text: str) -> list[str]:
tokens = []
i = 0
while i < len(text):
c = text[i]
if c == " ":
i += 1
elif c in "+-*()":
tokens.append(c)
i += 1
elif c in DIGITS:
j = i
while j < len(text) and text[j] in DIGITS:
j += 1
tokens.append(text[i:j])
i = j
else:
raise ValueError(f"unexpected character {c!r} at position {i}")
return tokens
def calc(text: str) -> int:
return compute(tokenize(text))
"2+5-3".split() gives ['2+5-3']: one piece.
We cut the text ourselves, one character at a time. An operator is one character; a number goes on while there are digits.
tokenize("12+5 - 3") gives ['12', '+', '5', '-', '3'], and compute does not change.
The pieces are called tokens. Cutting text into tokens is the first stage of every interpreter.
*: left to right is wrong
elif op == "*":
total = total * n
One more branch in the loop, and 2 + 5 * 3 gives 21.
In arithmetic it is 17: * binds tighter than + and -. This is called precedence.
A single running total cannot do this. When the loop reads 2 +, it does not yet know whether the 5 that follows is a number to add or the start of a product.
def compute_with_precedence(pieces: list[str]) -> int:
pieces = list(pieces)
i = 1
while i < len(pieces):
if pieces[i] == "*":
if i + 1 == len(pieces):
raise ValueError("missing number after '*'")
product = number(pieces[i - 1]) * number(pieces[i + 1])
pieces[i - 1 : i + 2] = [str(product)]
else:
i += 2
return compute(pieces)
First pass: replace each a * b by its value. Second pass: compute, as before.
['2', '+', '5', '*', '3']
['2', '+', '15']
17
After a replacement i stays where it is, so 2 * 3 * 4 becomes 6 * 4 and then 24.
One pass per level of precedence. With / on the same level as *, and a power operator above it that groups to the right, the passes and their special cases grow.
def balanced(text: str) -> bool:
stack = []
for c in text:
if c == "(":
stack.append(c)
elif c == ")":
if stack == []:
return False
stack.pop()
return stack == []
balanced("(1 + (2 - 3)) + 4") # True
balanced("(1 + 2))(") # False
balanced("(1 + 2") # False
Only checking that the parentheses are balanced already needs a stack: a list where we add and remove at the end.
A counter would also do here, but not with more kinds of bracket, such as ( and [.
Computing the value is harder: each open parenthesis needs its own running total, kept aside until the matching ).
| problem | by hand | today, with a grammar |
|---|---|---|
| cutting the text | tokenize, one character at a time | terminals, defined by regular expressions |
| well-formed or not | checks in compute and number | the parser reports syntax errors |
| precedence | one pass per level | one rule per level |
| parentheses | a stack | a rule that refers to itself |
Each new operator changed the code in several places. In the rest of the lecture the same changes are made in the grammar.
3 + 4 - 1. What about (1 + 2) * 3?A loop with one running total cannot handle nested parentheses. Last lecture we used a tree instead. Today we build the tree from the text.
From the calculator to Lark
Today we write a grammar, and a library builds the parser from it.
The library is Lark. The AST and the evaluator are still ours.
The code is in code/lesson-03/; the main file is mini_interpreter.py.
What the language looks like
from lark import Lark, Token, Tree
grammar = r"""
expr: bin | mono
mono: ground | paren
paren: "(" expr ")"
bin: expr OP mono
ground: NUMBER
NUMBER: /[0-9]+/
OP: "+" | "-" | "*" | "/" | "%"
%import common.WS
%ignore WS
"""
parser = Lark(grammar, start="expr")
Lark uses a variant of EBNF.
Terminals (upper case): the pieces of text, each defined by a regular expression or by strings.
Rules (lower case): how terminals and rules combine.
%ignore WS skips white space: 1+2 and 1 + 2 give the same result.
ground: NUMBER
paren: "(" expr ")"
expr, so expressions can nest.mono: ground | paren
bin: expr OP mono
mono. The two sides are different; the next slides show why.expr: bin | mono
bin or a mono. It is the start symbol: start="expr".10-4-3?In the left tree, the right operand of each - is a mono. Our grammar only allows this tree.
LEFT RECURSION
bin: expr OP monoOnly (10-4)-3 = 3. This is our grammar.
RIGHT RECURSION
bin: mono OP exprOnly 10-(4-3) = 9.
BOTH SIDES
bin: expr OP exprBoth trees are possible: the grammar is ambiguous. Lark accepts it anyway.
The evaluator only sees the tree, so the grammar decides the result.
By hand, one usually writes one function per rule (recursive descent). This works for most rules, but not for ours.
Works: the rule starts with a token
# paren: "(" expr ")"
def parse_paren(tokens, i):
expect(tokens, i, "(")
e, i = parse_expr(tokens, i + 1)
expect(tokens, i, ")")
return e, i + 1
It reads ( before calling parse_expr, so each call starts further on. The calls end when the tokens end.
Loops: the rule starts with itself
# bin: expr OP mono
def parse_bin(tokens, i):
left, i = parse_expr(tokens, i)
...
parse_expr tries parse_bin, which calls parse_expr at the same i, and so on: RecursionError. This is left recursion.
Lark accepts left recursion, so we keep the grammar as written.
Lark's output
The calculator: our tokenize
tokenize("(1 + 2) - 3")
# ['(', '1', '+', '2', ')', '-', '3']
Lark: the lexer built from the grammar
print(list(parser.lex("(1 + 2) - 3")))
[Token('LPAR', '('), Token('NUMBER', '1'), Token('OP', '+'), Token('NUMBER', '2'), Token('RPAR', ')'), Token('OP', '-'), Token('NUMBER', '3')]
The pieces are the same, and the spaces are dropped in both. We did not write Lark's lexer: Lark built it from the terminals of the grammar.
| in the grammar | token type | token value |
|---|---|---|
| NUMBER: /[0-9]+/ | NUMBER | '1', '2', '3' |
| OP: "+" | "-" | ... | OP | '+', '-' |
| "(" and ")" in paren | LPAR, RPAR | '(', ')' |
| %ignore WS | no token |
Our tokenize returned plain strings. A Lark token also records which terminal matched: its type. The matched text is its value, and it is still a string: '1', not the number 1.
LPAR and RPAR are names Lark makes up for strings written directly in a rule. The type is what the transform will look at later, with the pattern Token(type="NUMBER", value=digits).
# in Lark, simplified
class Token(str):
def __repr__(self): ...
t = list(parser.lex("(1 + 2) - 3"))[2]
print(t) # +
print([t]) # [Token('OP', '+')]
isinstance(t, str) # True
t == "+" # True
t.type # 'OP'
t.value # '+'
t.line, t.column # (1, 4)
class Token(str) means that Token is a subclass of str: every token is also a string.
From str it inherits comparison and the string methods, so t == "+" compares the text. Lark adds the attributes: the type, and the position in the input, which it uses in error messages.
print(t) uses str, inherited: it shows +. A list shows its elements with repr, which Token redefines: Token('OP', '+'). It is not a dataclass.
In a match, write the attributes by keyword: Token(type="OP", value=op). Token("OP", op) raises a TypeError.
parse_tree = parser.parse("(1 + 2) - 3")
print(parse_tree.pretty())
One node per rule applied, one leaf per token.
16 nodes, of which only 5 matter: two operations and three numbers.
The parenthesis tokens are not in the tree: the paren node and the nesting already record the grouping.
for bad in ["1 +", "1 ++ 2", "-1"]:
try:
parser.parse(bad)
except Exception as e:
print(f"{bad!r}: {type(e).__name__}: {str(e).splitlines()[0]}")
'1 +': UnexpectedEOF: Unexpected end-of-input. Expected one of:
'1 ++ 2': UnexpectedCharacters: No terminal matches '+' in the current parser context, at line 1 col 4
'-1': UnexpectedCharacters: No terminal matches '-' in the current parser context, at line 1 col 1
-1 is rejected: NUMBER is /[0-9]+/, digits only, so there are no negative literals.
Our part
01 · EXTRA NODES
expr around bin, mono around ground: they are there for the grammar. The evaluator has no use for them.
02 · A LIBRARY TYPE
Every later stage would match rule names as strings, and convert '1' to 1 each time. Renaming a rule, or changing library, would break all of them.
So we convert it to the AST of the last lecture.
type Op = Literal["+", "-", "*", "/", "%"]
@dataclass(frozen=True)
class Number:
value: int
@dataclass(frozen=True)
class BinaryExpression:
op: Op
left: Expression
right: Expression
type Expression = Number | BinaryExpression
Two node classes, where the grammar has five rules and two terminals.
Number holds an int. op is one of five strings, checked by mypy.
There is no Variable node, since the grammar has no names. Class names are longer than last lecture's (BinaryExpression for BinaryOp); we use these from now on.
def transform_parse_tree(tree: Tree) -> Expression:
match tree:
case Tree(data="expr", children=[subtree]):
return transform_parse_tree(subtree)
case Tree(data="mono", children=[subtree]):
return transform_parse_tree(subtree)
case Tree(data="paren", children=[subtree]):
return transform_parse_tree(subtree)
case Tree(data="ground",
children=[Token(type="NUMBER", value=digits)]):
return Number(value=int(digits))
... # the bin case: next slide
case _:
raise ValueError(f"Unexpected parse tree node: {tree}")
Lark's Tree and Token are ordinary classes, so we can use class patterns.
expr, mono and paren have one child: we transform the child. In ground the token is text, converted here to int.
The last case should never run. It catches a mismatch between the grammar and this function.
bin case
case Tree(
data="bin",
children=[
left,
Token(type="OP", value="+" | "-" | "*" | "/" | "%" as op),
right,
],
):
return BinaryExpression(
op=op,
left=transform_parse_tree(left),
right=transform_parse_tree(right),
)
This is the only case that builds a node with children. "+" | … as op checks that the token is an operator and binds it to op. The grammar already ensures this; the pattern also narrows op from str to Op for the type checker.
(1+2)-3
def parse_ast(expression: str) -> Expression:
parse_tree = parser.parse(expression)
return transform_parse_tree(parse_tree)
print(parse_ast("(1+2)-3"))
BinaryExpression(op='-',
left=BinaryExpression(op='+',
left=Number(value=1), right=Number(value=2)),
right=Number(value=3))
Output re-indented; Python prints it on one line.
Transformer
class ToAST(Transformer):
def ground(self, children):
return Number(int(children[0]))
def bin(self, children):
left, op, right = children
return BinaryExpression(str(op), left, right)
def mono(self, children):
return children[0]
def expr(self, children):
return children[0]
# no method for `paren`
print(ToAST().transform(parser.parse("(1+2)-3")))
Lark's own mechanism: one method per rule, found by name.
BinaryExpression(op='-',
left=Tree('paren', [BinaryExpression(...)]),
right=Number(value=3))
There is no method for paren, and Lark does not complain: it leaves that node as it was, a Lark Tree, in the middle of our AST.
The mistake shows up only later, when evaluate finds a node it does not know. With our match version, a missing case raises an error during the transform itself.
Code in alternatives.py.
def evaluate(ast: Expression) -> int:
match ast:
case Number(value):
return value
case BinaryExpression(op, left, right):
left_value = evaluate(left)
right_value = evaluate(right)
match op:
case "+":
return left_value + right_value
... # "-" and "*" alike; "/", "%": next slide
case _:
assert_never(op)
case _:
assert_never(ast)
\( ⟦n⟧ = n \)
\( ⟦e_1 \mathbin{op} e_2⟧ = ⟦e_1⟧ \mathbin{op} ⟦e_2⟧ \)
On the right, op is the operation on Python integers.
One assert_never for the node types, one for the operators. If Op gets a sixth operator and this match does not, mypy reports an error.
(0-7)/2?
case "/":
if right_value == 0:
raise ValueError("Division by zero")
return left_value // right_value
case "%":
if right_value == 0:
raise ValueError("Division by zero")
return left_value % right_value
Values are int, so the result cannot be -3.5. Which integer?
-3.5 lies between -4 and -3. Python's // always takes the smaller one: -4. C and Java drop the decimal part: -3.
The remainder % must agree with the quotient, so that quotient · 2 + remainder = -7. In Python, (0-7) % 2 is 1, since -4 · 2 + 1 = -7. In C it is -1, since -3 · 2 - 1 = -7.
1 · SYNTAX ERROR
Raised by Lark before evaluation: 1 + is not a program.
2 · INTERNAL ERROR
Raised by the last case of transform_parse_tree, Unexpected parse tree node. It happens if we add a rule to the grammar and forget its case in the transform. No input can cause it: the mistake is ours, not the user's.
3 · RUNTIME ERROR
1/(2-2) is a valid program with no value. We raise a ValueError.
Without the check, Python would raise ZeroDivisionError. We raise our own error so that the language defines it. Returning infinity or 0 would also be possible; the language has to pick one.
Another decision in the grammar
1+2*3?With a single OP and a single bin rule, all operators have the same priority. Like associativity, precedence is fixed by the grammar.
grammar_prec = r"""
?expr: sum
?sum: sum ADDOP prod -> bin
| prod
?prod: prod MULOP mono -> bin
| mono
?mono: ground | paren
paren: "(" expr ")"
ground: NUMBER
NUMBER: /[0-9]+/
ADDOP: "+" | "-"
MULOP: "*" | "/" | "%"
%import common.WS
%ignore WS
"""
No + inside a prod without parentheses: 1+2*3 is 7. ? removes one-child nodes; -> bin renames. precedence.py reuses the AST and evaluate.
earley = Lark(grammar, start="expr") # the default
lalr = Lark(grammar, start="expr", parser="lalr")
text = "+".join(["1"] * 3000)
earley: 0.115 s
lalr: 0.010 s
EARLEY · THE DEFAULT
Accepts any context-free grammar, including ambiguous ones. Slower.
LALR
A single pass; about ten times faster here. It rejects some grammars with a GrammarError, and resolves the ambiguity of expr OP expr without warning.
Our inputs are short and the grammar will change, so we keep the default. Timings from alternatives.py on one laptop.
Using and extending the interpreter
def REPL() -> None:
while True:
try:
expression = input("> ")
except EOFError:
print()
return
if expression == "exit":
return
try:
print(evaluate_string(expression))
except Exception as e:
print(f"Error: {e}")
> 1+2
3
> (10+20)*2
60
> 100/0
Error: Division by zero
> 1 +
Error: Unexpected end-of-input. Expected one of:
* LPAR
* NUMBER
> exit
Ctrl-D (end of input) exits cleanly. We catch every exception, so an error does not stop the loop.
01 · TOKENS
list(parser.lex(s))
Are the tokens the ones you expected? Look for a terminal that matches too much, or one that never matches.
02 · PARSE TREE
print(parser.parse(s).pretty())
Which tree was built? Precedence and associativity problems are visible here.
03 · AST
print(parse_ast(s))
Is the AST correct? If it is and the value is wrong, the bug is in evaluate.
The bug is in the stage after the last correct output.
s = "10 - 4 - 3"
print(list(parser.lex(s)))
[Token('NUMBER', '10'), Token('OP', '-'), Token('NUMBER', '4'), Token('OP', '-'), Token('NUMBER', '3')]
Five tokens, in the order of the text. 10 is one NUMBER, not two; the spaces are gone.
A problem here would be, for example, -4 read as one token, or a character that no terminal matches.
expr
bin
expr
bin
expr
mono
ground 10
-
mono
ground 4
-
mono
ground 3
print(parser.parse(s).pretty())
Indentation shows the children of each node. The outer bin has three children: an expr, the operator -, and the mono for 3.
The inner bin, 10 - 4, is the left operand of the outer one. So the tree groups as (10 - 4) - 3.
If the grouping is not the one you expected, the grammar is the problem, not the evaluator.
print(parse_ast(s))
print(evaluate(parse_ast(s)))
BinaryExpression(op='-',
left=BinaryExpression(op='-',
left=Number(value=10), right=Number(value=4)),
right=Number(value=3))
3
The same grouping as the parse tree, without the nodes that only the grammar needs (expr, mono, ground). The value is 3.
Output re-indented; Python prints the AST on one line. If this AST is right and the value is wrong, the bug is in evaluate.
grammar_cmp = r"""
?expr: cmp
?cmp: sum CMPOP sum -> compare
| sum
?sum: sum ADDOP prod -> bin
| prod
...
CMPOP: "<" | "<=" | "=="
"""
type CmpOp = Literal["<", "<=", "=="]
@dataclass(frozen=True)
class Comparison:
op: CmpOp
left: Expression
right: Expression
type Expression = Number | BinaryExpression | Comparison
A new level, looser than +: 1+2 <= 3 compares two sums.
The rule is sum CMPOP sum, so the level is non-associative: 1 < 2 < 3 is a syntax error.
Python reads it as 1 < 2 and 2 < 3, C as (1 < 2) < 3. We do not allow it.
1 < 2?
case Comparison(op, left, right):
a, b = evaluate(left), evaluate(right)
match op:
case "<":
return int(a < b)
case "<=":
return int(a <= b)
case "==":
return int(a == b)
case _:
assert_never(op)
evaluate_cmp("1+2*3 < 10") # 1
evaluate_cmp("(1<2) + (2<1)") # 1
1, as in C. All values are still int, and evaluate keeps its return type.
Returning True would mean values are int or bool, and we would have to define (1<2) + 1.
Which values a language has is a separate topic, for a later lecture.
GRAMMAR
If missing: Lark rejects 1 < 2.
AST
Comparison, added to Expression.
TRANSFORM
If missing: Unexpected parse tree node, at run time.
EVALUATE
If missing: mypy reports it before running, because of assert_never.
Only the missing evaluator case is caught before running. The transform needs tests.
In class
# mini_interpreter.py, flat grammar
parse_ast("2*3-1")
parse_ast("2-3*4")
parse_ast("8/2/2")
evaluate_string("2-3*4")
# precedence.py
evaluate(parse_ast_prec("2-3*4"))
Level 1 of 6
2-3*4 with both grammars and count the nodes.
evaluate_string("2^3") # 8
evaluate_string("2^3^2") # 64: left to right
evaluate_string("2^(0-1)") # ?
Level 2 of 6
^, power, to the flat grammar: OP, Op, the transform, the evaluator.mypy before the evaluator has its case. What does it say?2 ** -1 is 0.5, not an int. Decide what 2^(0-1) should be and implement it.
evaluate_string("-1") # -1
evaluate_string("-(2+3)") # -5
evaluate_string("2*-3") # ?
evaluate_string("--1") # ?
evaluate_string("1--2") # ?
Level 3 of 6
? should be, before you start.Hint: a minus followed by a mono is itself a mono.
def calc(text: str) -> int: ...
calc("1+2*3") # 7
calc("2^3^2") # 512: right to left
calc("2*3^2") # 18
calc("(1+2") # ValueError
calc("7/(1-1)") # ValueError
Level 4 of 6 · take home
precedence.py.^ as a new level, tighter than *, right-associative. Hint: a named terminal and a right-recursive rule.ValueError: convert Lark's exceptions.
def run(text: str) -> str: ...
run("{1, 2, 3} | {3, 4}") # '{1, 2, 3, 4}'
run("{1, 2, 3} & {2, 3, 4}") # '{2, 3}'
run("{1, 2, 3} - {2}") # '{1, 3}'
run("{1, 2} | {3} & {3, 4}") # '{1, 2, 3}'
run("({1, 2} | {3}) & {3, 4}") # '{3}'
run("{} - {1}") # '{}'
run("{1, 2") # ValueError
Level 5 of 6 · take home
evaluate: set literals, | union, & intersection, - difference, parentheses.& binds tighter than | and -, which group from left to right.frozenset[int]. run returns the elements in increasing order, as in the examples.ValueError.The empty set {} must be accepted: in the grammar, the list of numbers is optional.
def run(text: str) -> list[tuple[int, int]]: ...
run("[1, 4] | [3, 8]") # [(1, 8)]
run("[1, 4] & [3, 8]") # [(3, 4)]
run("[1, 4] | [3, 8] & [6, 10]") # [(1, 4), (6, 8)]
run("[1, 5] | [6, 8]") # [(1, 8)]
run("[1, 2] & [5, 6]") # []
run("[5, 1]") # ValueError
Level 6 of 6 · take home
[1, 4] is 1, 2, 3, 4. Only | and &.normalize first: sort, then merge. Union is normalize(a + b).normalize.[1, 5] | [6, 8] is [(1, 8)]: between 5 and 6 there is no integer.