Rows — hash a whole row¶

import polars as pl
import polars_hash as plh

hash_rows(exprs: IntoExpr | Iterable[IntoExpr], *more_exprs: IntoExpr, version: int = 1) -> HExpr

>>> df = pl.DataFrame(
...     {
...         "foo": ["hello_world"],
...         "bar": [42],
...         "baz": [[1, 2, 3]],
...         "qux": [{"a": 1}],
...     }
... )
>>> df.select(plh.hash_rows(pl.all()).chash.sha2_256()).item()
'9055866af8d3c113e0a8fdb729ce8e6fa67ed5f6f51efa8235a588e88ea972f4'

>>> df.select(plh.hash_rows(pl.all())).item()
b'\r\x04\x05\x0bhello_world\x03\x01*\x0c\x03\x03\x01\x01\x03\x01\x02\x03\x01\x03\r\x01\x03\x01\x01'

>>> df.select(plh.hash_rows("foo", "bar").nchash.xxh3_64()).item()
9123089596710669414

The encoding¶

This is version 1. These bytes do not change. A user can keep a hash for longer than the release that made it. Therefore a new format takes a new version number, and version=1 always gives these bytes.

A row is a struct value. It contains each column in order, and nothing more.

An example¶

pl.DataFrame({"id": [7], "name": ["ok"]}).select(plh.hash_rows(pl.all()))

0d 02 03 01 07 05 02 6f 6b

Read the bytes from the left:

Each value starts with one tag byte, which gives the class of the value. The bytes after the tag are the payload. A payload has a fixed width, or it starts with its own length or count. Therefore the values need no separator, and a row has only one reading.

A null value is only its tag. A List and a Struct contain their values in the same form, one after the other. This row has a null and a list of 1 and 300:

pl.DataFrame({"id": pl.Series([None], dtype=pl.Int64), "tags": [[1, 300]]}).select(
    plh.hash_rows(pl.all())
)

0d 02 00 0c 02 03 01 01 03 01 ac 02

Numbers with a variable length¶

A count, a length, and the magnitude of an integer all use a varint. A varint takes one byte for a value below 128, and one more byte for each 7 bits after that.

Each byte of a varint holds 7 bits of the value, and the least significant 7 bits come first. Each byte has the high bit set, but the last byte does not:

An integer payload is a sign byte and then a varint of the magnitude. The sign byte is 01 for zero and more, and 00 for less than zero. Therefore 300 is 01 ac 02, and -300 is 00 ac 02.

The tags¶

How the encoder reads a value¶

Polars can store one value in more than one way. The encoder reads the value, and not the storage:

Which values stay different¶

What the encoder does not read¶

The column names. A new name for a column keeps all the hashes. A new order of the columns does not, because the order identifies the values. A column that you give two times is two values of the row.

The field names of a Struct. The encoder writes the fields in their order, as it writes the columns of a row. Therefore a new name for a field keeps the hash, and a new order of the fields does not.

Polars compares two Struct values by name, and the encoder reads the order. Therefore the two do not always agree:

first = pl.Series([{"a": 2, "b": 1}], dtype=pl.Struct({"a": pl.Int64, "b": pl.Int64}))
second = pl.Series([{"b": 1, "a": 2}], dtype=pl.Struct({"b": pl.Int64, "a": pl.Int64}))

first.equals(second)  # True: each name has the same value
# The hashes are different, because the values are in a different order.

The data type of an empty List. An empty List(Int64) and an empty List(String) both give the count 0 and no elements.

The chunks and the slices. One chunk, ten chunks, or a slice of a larger frame all give the same bytes for the same rows.