`nchash` — Non-cryptographic hash functions¶

Every example on this page starts here

import polars as pl
import polars_hash as plh

Non-cryptographic hash functions, on pl.Expr as .nchash.

These expressions are fast and their output is stable. Each one accepts Utf8 or Binary. A digest reads bytes, so the data type of the input does not change the result, and a Utf8 column gives the digest of its UTF-8 bytes. A null input gives a null output. Any other input type raises ComputeError: expected `String` or `Binary` input.

Every expression with a UInt128 output also takes return_binary=True, which writes the same hash as 16 Binary bytes.

Warning

Polars encodes UInt128 as a private Arrow type, so to_arrow() and to_pandas() raise ArrowInvalid on such a column, and to_numpy() fails. write_parquet, write_ipc, joins, group_by and sorting all work. Set return_binary=True if the column has to leave Polars: Binary travels everywhere. A cast to pl.Binary is not the same thing. It writes the decimal digits of the integer, not its 16 bytes.

wyhash ¶

wyhash() -> Expr

wyhash with 64-bit output.

This expression is very fast. You cannot set the seed. It is always 0.

Returns:

Type	Description
`Expr`	UInt64.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.wyhash()).item()
16737367591072095403

Each expression in this namespace also hashes the bytes of a Binary column:

>>> pl.select(pl.lit(b"my_bytes").nchash.wyhash()).item()
5112362246832359110

sha1 ¶

sha1() -> Expr

SHA-1, hexadecimal.

Returns:

Type	Description
`Expr`	Utf8 with 40 characters.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.sha1()).item()
'e4ecd6fc11898565af24977e992cea0c9c7b7025'

md5 ¶

md5() -> Expr

MD5, hexadecimal.

Returns:

Type	Description
`Expr`	Utf8 with 32 characters.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.md5()).item()
'99b1ff8f11781541f7f89f9bd41c4a17'

murmur32 ¶

murmur32(*, seed: int = 0) -> Expr

MurmurHash3, x86 32-bit variant.

Many systems have an implementation of this algorithm, for example Spark, Kafka and bloom filter libraries.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u32`, that is 0 to 4294967295.	`0`

Returns:

Type	Description
`Expr`	UInt32.

Note

With the default seed, an empty string gives 0. With a different seed, an empty string gives a value that is not 0. This is the correct MurmurHash3 result. It is not a null value in the output.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.murmur32()).item()
3531928679
>>> df.select(plh.col("foo").nchash.murmur32(seed=42)).item()
259561949

murmur128 ¶

murmur128(*, seed: int = 0, return_binary: bool = False) -> Expr

MurmurHash3, x64 128-bit variant.

The integer matches mmh3.hash128(..., signed=False).

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u32`. The 128-bit variant also uses a 32-bit seed.	`0`
`return_binary`	`bool`	Write the hash as the 16 digest bytes. MurmurHash3 writes its digest as two little-endian halves, so these are the bytes `mmh3.hash_bytes()` gives. Use this where the target of a write has no 128-bit integer. The bytes and the integer hold the same hash.	`False`

Returns:

Type	Description
`Expr`	UInt128, or Binary with `return_binary=True`.

Warning

Releases up to 0.7.0 returned the 16 digest bytes, and 0.8.0 changed the output to UInt128. Only the container changed. int.from_bytes(old, "little") converts a stored value. From 0.9.1, return_binary=True gives the 0.7.0 bytes again, byte for byte.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.murmur128()).item()
134986332493155497415370161450594282648
>>> df.select(plh.col("foo").nchash.murmur128(seed=42)).item()
128378975539535818103252123378652633995
>>> df.select(
...     plh.col("foo").nchash.murmur128(return_binary=True).bin.encode("hex")
... ).item()
'982cf39e1c1aa55d1b079716076c8d65'

xxhash32 ¶

xxhash32(*, seed: int = 0) -> Expr

XXH32, the original 32-bit xxHash.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u32`. A value outside that range, or `None`, raises `expected u32`.	`0`

Returns:

Type	Description
`Expr`	UInt32.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.xxhash32()).item()
1605956417
>>> df.select(plh.col("foo").nchash.xxhash32(seed=42)).item()
1544934469

xxhash64 ¶

xxhash64(*, seed: int = 0) -> Expr

XXH64, the 64-bit xxHash.

xxh3_64() is faster. Use xxhash64() when you must get the same values as a different system.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u64`.	`0`

Returns:

Type	Description
`Expr`	UInt64.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.xxhash64()).item()
5654987600477331689
>>> df.select(plh.col("foo").nchash.xxhash64(seed=42)).item()
17477110538672341566

xxh3_64 ¶

xxh3_64(*, seed: int = 0) -> Expr

XXH3 with 64-bit output.

For usual string lengths, this is the fastest expression in the namespace.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u64`.	`0`

Returns:

Type	Description
`Expr`	UInt64.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.xxh3_64()).item()
7060460777671424209
>>> df.select(plh.col("foo").nchash.xxh3_64(seed=42)).item()
827481053383045869

xxh3_128 ¶

xxh3_128(*, seed: int = 0, return_binary: bool = False, byte_order: Literal['little', 'big'] | None = None) -> Expr

XXH3 with 128-bit output.

The integer matches xxhash.xxh128_intdigest(). Formatted as 32 hexadecimal digits with f"{value:032x}", it is the canonical XXH128 digest, the string xxh128_hexdigest() gives.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u64`.	`0`
`return_binary`	`bool`	Write the hash as 16 Binary bytes. Use this where the target of a write has no 128-bit integer.	`False`
`byte_order`	`Literal['little', 'big'] \| None`	How those bytes read, with `return_binary=True`. `"big"` is the digest XXH3 itself writes, the bytes `xxhash.xxh128_digest()` gives. `"little"` is the integer least significant byte first, which is what releases up to 0.7.0 wrote. The default is `"little"` and it warns, because the compatible order is not the canonical one. Name an order to accept it silently.	`None`

Returns:

Type	Description
`Expr`	UInt128, or Binary with `return_binary=True`.

Warning

0.8.0 changed this output from Binary to UInt128. Up to 0.7.0 the 16 bytes held the value in the reverse of the canonical order. The integer now agrees with the reference. To read data hashed by an older release, reverse the old bytes: int.from_bytes(old, "little").

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.xxh3_128()).item()
253649469245435599925940275794906345219
>>> f"{253649469245435599925940275794906345219:032x}"
'bed31c5eaf3dc62267fb185e21fe6f03'
>>> df.select(plh.col("foo").nchash.xxh3_128(seed=42)).item()
314735830047873782861649874643137875266

The canonical digest bytes, and the bytes of releases up to 0.7.0:

>>> df.select(
...     plh.col("foo")
...     .nchash.xxh3_128(return_binary=True, byte_order="big")
...     .bin.encode("hex")
... ).item()
'bed31c5eaf3dc62267fb185e21fe6f03'
>>> df.select(
...     plh.col("foo")
...     .nchash.xxh3_128(return_binary=True, byte_order="little")
...     .bin.encode("hex")
... ).item()
'036ffe215e18fb6722c63daf5e1cd3be'

farmhash32 ¶

farmhash32() -> Expr

Google FarmHash fingerprint32.

The fingerprint functions give the same value on all platforms. BigQuery uses them for its FARM_FINGERPRINT function. This expression has no seed.

Returns:

Type	Description
`Expr`	UInt32.

Examples:

>>> df = pl.DataFrame({"foo": ["hello world"]})
>>> df.select(plh.col("foo").nchash.farmhash32()).item()
430397466

farmhash64 ¶

farmhash64() -> Expr

Google FarmHash fingerprint64, the 64-bit fingerprint.

This expression has no seed.

Returns:

Type	Description
`Expr`	UInt64.

Note

The FARM_FINGERPRINT function in BigQuery gives the same 64 bits as a signed INT64. To compare the two results, use .cast(pl.Int64) on the polars-hash output.

Examples:

>>> df = pl.DataFrame({"foo": ["hello world"]})
>>> df.select(plh.col("foo").nchash.farmhash64()).item()
6381520714923946011

cityhash32 ¶

cityhash32() -> Expr

Google CityHash CityHash32, from CityHash v1.1.1.

FarmHash replaced CityHash, so use farmhash32() for new work. Use cityhash32() when you must get the same values as a different system. This expression has no seed. CityHash32 takes none.

Returns:

Type	Description
`Expr`	UInt32.

Warning

Every CityHash expression here gives the values of v1.1.1, the last release Google published. CityHash64 changed during the v1.0 series and CityHash128 changed after v1.0.3, so a system built on an earlier release gives a different value for the same input. Check which release the other system uses before you compare.

Note

FarmHash reuses CityHash for short input, so cityhash32() and farmhash32() give the same value for input up to 12 bytes, as do cityhash64() and farmhash64() up to 32 bytes. They part above those lengths. The equal values are not a defect.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.cityhash32()).item()
1719156559

cityhash64 ¶

cityhash64(*, seed: int | None = None) -> Expr

Google CityHash CityHash64, from CityHash v1.1.1.

Parameters:

Name	Type	Description	Default
`seed`	`int \| None`	A value in the range of a `u64`, or `None` for the unseeded algorithm.	`None`

Returns:

Type	Description
`Expr`	UInt64.

Warning

A seed of 0 is not the same as no seed. Without a seed this expression is CityHash64. With one it is CityHash64WithSeed, a separate function that gives a different value for every seed, 0 included. This is why the seed defaults to None.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.cityhash64()).item()
15605398435621216523
>>> df.select(plh.col("foo").nchash.cityhash64(seed=42)).item()
10175920941468920074

cityhash128 ¶

cityhash128(*, return_binary: bool = False) -> Expr

Google CityHash CityHash128, from CityHash v1.1.1.

CityHash128WithSeed is not wrapped, so this expression has no seed.

Parameters:

Name	Type	Description	Default
`return_binary`	`bool`	Write the hash as 16 Binary bytes, least significant byte first. The bytes and the integer hold the same hash.	`False`

Returns:

Type	Description
`Expr`	UInt128, or Binary with `return_binary=True`.

Note

C++ returns CityHash128 as a pair. This expression packs it the way python-cityhash does, Uint128Low64(h) << 64 | Uint128High64(h), so the C++ low word is the high half of the integer. A system that composes the halves the other way round, or that stores the raw 16 bytes, needs a word swap before the values compare equal.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.cityhash128()).item()
133423608296839006301901834072762183026

crc32c ¶

crc32c(*, return_binary: bool = False, byte_order: Literal['little', 'big'] = 'little') -> Expr

CRC-32C (Castagnoli), the variant iSCSI, SCTP and libcsp use.

This is a checksum, not a general-purpose hash. It is fast to compute and it detects the usual transmission and storage errors, but it does not resist a deliberate collision.

Parameters:

Name	Type	Description	Default
`return_binary`	`bool`	Write the checksum as 4 Binary bytes.	`False`
`byte_order`	`Literal['little', 'big']`	The order of those bytes. `"little"` is the integer's own bytes. `"big"` is network byte order, the order a checksum field on the wire usually has.	`'little'`

Returns:

Type	Description
`Expr`	UInt32, or Binary with `return_binary=True`.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.crc32c()).item()
1680342080
>>> df.select(
...     plh.col("foo")
...     .nchash.crc32c(return_binary=True, byte_order="big")
...     .bin.encode("hex")
... ).item()
'6427fc40'

gxhash32 ¶

gxhash32(*, seed: int = 0) -> Expr

GxHash with 32-bit output.

GxHash reaches its speed through the AES block cipher, which the CPU runs as a single instruction. It is the fastest expression in the namespace for input above about a hundred bytes. Below that, xxh3_64() is the one to beat.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u64`. GxHash takes an `i64` seed upstream, and this namespace presents every 64-bit seed as a `u64`, so a seed at or above `263` is the upstream seed minus `264`. Both reach the same 64 bits.	`0`

Returns:

Type	Description
`Expr`	UInt32.

Warning

GxHash needs a CPU with AES instructions. The algorithm has no software fallback. The published wheels cover x86, x86-64 and aarch64, and every one of them is built with the instructions enabled, so a CPU without them stops the process the moment a GxHash expression runs. On x86 the instructions arrived with Westmere in 2010 and every processor since has them. On ARM they are an optional extension: Apple silicon and server parts have them, and some small boards, such as the Raspberry Pi 4, do not.

The instructions are enabled for the whole build and not for GxHash alone, so on x86 ahash, which the h3 namespace pulls in, switches to its AES-NI implementation as well, and the h3 expressions come to need them too. On aarch64 ahash keeps its portable path, so there GxHash is the only namespace affected. There are no linux-armv7 or linux-ppc64le wheels from 0.8.0 on, because GxHash cannot be built for either.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.gxhash32()).item()
2751540945
>>> df.select(plh.col("foo").nchash.gxhash32(seed=42)).item()
3382299372

gxhash64 ¶

gxhash64(*, seed: int = 0) -> Expr

GxHash with 64-bit output.

gxhash32() gives the requirements of the algorithm and the meaning of the seed.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u64`. Every GxHash expression is seeded and the default seed is 0. Unlike `cityhash64()`, there is no unseeded form to differ from.	`0`

Returns:

Type	Description
`Expr`	UInt64.

Warning

GxHash holds its output stable across platforms, but only within a major version. polars-hash pins GxHash 3 exactly, so the values here do not change without a release that says so. A system on GxHash 2 gives different values for the same input and seed.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.gxhash64()).item()
2180020304351407825
>>> df.select(plh.col("foo").nchash.gxhash64(seed=42)).item()
15254170022685821676

gxhash128 ¶

gxhash128(*, seed: int = 0, return_binary: bool = False) -> Expr

GxHash with 128-bit output.

gxhash32() gives the requirements of the algorithm.

Parameters:

Name	Type	Description	Default
`seed`	`int`	A value in the range of a `u64`.	`0`
`return_binary`	`bool`	Write the hash as 16 Binary bytes, least significant byte first. The bytes and the integer hold the same hash.	`False`

Returns:

Type	Description
`Expr`	UInt128, or Binary with `return_binary=True`.

Note

GxHash builds a single 128-bit state and each width reads the low part of it, so gxhash32() is the low 32 bits of gxhash64(), which is the low 64 bits of gxhash128(). Ask for the width you need. The narrow ones cost no less than the wide one.

Examples:

>>> df = pl.DataFrame({"foo": ["hello_world"]})
>>> df.select(plh.col("foo").nchash.gxhash128()).item()
56218077491375249900279963678916292305
>>> df.select(plh.col("foo").nchash.gxhash128(seed=42)).item()
11136336363892181958542060125951740652

nchash — Non-cryptographic hash functions¶

wyhash ¶

sha1 ¶

md5 ¶

murmur32 ¶

murmur128 ¶

xxhash32 ¶

xxhash64 ¶

xxh3_64 ¶

xxh3_128 ¶

farmhash32 ¶

farmhash64 ¶

cityhash32 ¶

cityhash64 ¶

cityhash128 ¶

crc32c ¶

gxhash32 ¶

gxhash64 ¶

gxhash128 ¶

`nchash` — Non-cryptographic hash functions¶