Wojciech Muła --- website

Rust is perfectly imperfect

Thu, 08 Jan 2026 12:00:00 +0100

My sampling of the real world is performed via X, formerly Twitter. And as of the end of 2025, it appears that Rust is the most hated language ever. I noticed that the C and C++ programmers constitute the majority of Rust critics.

Despite the critics, Rust is having its momentum — more and more companies are adopting Rust, trying to create new pieces of software using it or just rewriting their software. There are also sporadic opposite moves, that is getting back to C or C++.

I was thinking about these strong, anti-Rust sentiments and — more importantly — why people want to use Rust. The anti-Rust sentiments seem to me to be more of a social problem caused by individuals, and since I am rarely involved in "community" life and seldom read any comments, I don’t feel like I have anything interesting to add. Let me share my thoughts about the latter issue.

From the perspective of language programming theory, Rust added nothing. Literally, there’s nothing new in it. Of course, someone might shout “THE BORROW CHECKER” — but the borrow checker is just more advanced liveness analysis.

Rust did something that many other programming languages tried in the past but had failed to deliver. It is easy to use thanks to its consistency and predictability. And it was achieved by Rust as the whole, not only the language itself.

First of all, bringing the toolchain – which contains the compiler, linker, standard library, documentation and package manager — is seamless with rustup.
The package manager, cargo, makes it easy to tame dependencies. And it works with third-party packages as well as local ones. From the programmer point of view there’s no difference whether they use “local” or “remote” packages. It’s also easy to point out the path where the given package is located, so with established dependencies, the internet connection is not needed.
The standard library provides sufficiently powerful entities to produce a decent program, although it is not as rich as Python's one. But all basic, numeric data types come with a handful of useful methods and constants. Strings are by default UTF-8 encoded ones, also with a rich set of methods. There are also popular containers: resizable vectors, maps and sets (both key-ordered and hash-based). There are interfaces to the underlying OS: we can work with a file system, network, threading and timers.
The language comes with a small, strong type system. There are arrays, tuples, and structures and, the most important element of the type system, algebraic data types, called for an unknown reason “enums”. Algebraic data types remove the need of introducing hierarchies of types in 99% cases — complex structures are naturally expressed with “enums”. Even using interfaces (called in Rust “traits”) is not needed. Apart from libraries, when we work on regular software, we usually know all types in advance, or the rate of introducing new types is relatively slow.
The language supports two special kinds of enums: Option (value or null/nil/none) and Result (value or error). The types come with a couple of methods that eases their usage, for instance returning default value when option is none. The standard library uses these enums everywhere, and their usage is ubiquitous and feels natural.
Rust allows us to generate code based on annotations, this is a kind of compile-time reflection. These code generators are written in Rust. This feature allows programmers to extend their types with extra methods or implement some boring parts of code. Think about serialization, for instance.
Generics, called in C++ as “templates”, are built into the language and their use is mostly not different than using regular types/functions.
The language introduces the traits, called in Java as “interfaces”, that enable two important programming constructions: abstract data types and restricting template arguments, acting as C++’s concepts. When we need to interface with other-programmer-defined code, we use traits. Traits are also practical: they allow not only to define an abstract API, but also provide default implementations. Oh, and traits can extend other traits.
Feature flags are built in the whole ecosystem.
The tools for documenting code are built in. There’s no difference in documentation generated for standard library and any other package.
Rust is also aware of situations where we don’t work under any OS (so called “baremetal”) or we are an OS. Then a single declaration no_std disable linking with the standard library — yet keep providing some OS-independent types and functions via crate “core”.

These are the major factors that make Rust ecosystem composable. I’m not really convinced that Rust’s safety is the primary criteria of choosing the language. For sure, Rust makes it hard to introduce memory-related bugs, but there is still a huge area where bugs are possible. Well, if people really needed safety in their system, they would use functional languages, theorem provers and other sophisticated tools or methods to assure software is safe in all possible attack or malfunction directions. The C and C++ world already have quite a nice set of tools helping in catching different kinds of bugs, for instance fuzzers, runtime checkers like address sanitizers or valgrind.

Rust, similarly to functional languages, allows a programmer to focus on their problem: if your code compiles, it likely will work.

But in my opinion Rust brings back the joy of programming in an imperative paradigm.

Thank you for reading.

Acknowledgements: I’d like to thank Daniel Lemire, Dan Shechter and Romek Kurc for reading the draft of this text and sharing their corrections and thoughts.

Change case of UTF-32-encoded strings

Sun, 02 Feb 2025 12:00:00 +0100

Introduction

Changing letters case is something that appear handy in many situations, for instance input normalization, parsing, and other text-related tasks.

When we are dealing with ASCII, such conversion is straightforward. We check if a character code lies in the range 'a' .. 'z' (or 'A' .. 'Z') and when its true, we toggle the 5th bit (0x20).

But in the case of Unicode-encoded strings it is not that simple. There are many code points having upper- or lowercase counterparts, and they are not placed in any regular way in the Unicode code space. Although we may identify shorter or longer ranges of such codes, similarly to Latin letters in ASCII, this does not help much. See appendixes A, B and C, where we visualize how these characters are located.

Additionally, there are cases where an uppercase or lowercase version of the given character is not a single character, but a string. These strings are short, have two or three Unicode points. For example the uppercase of "Latin small ligature ffi" (ﬃ) is a three-character string "FFI".

How many characters are subject of case change (Unicode 15.0.0)
	1 code point	2 code points	3 code points	total
uppercase	1423	86	16	1525
lowercase	1432	1	0	1433

How many characters are subject of case change (Unicode 13.0.0)
	1 code point	2 code points	3 code points	total
uppercase	1383	86	16	1485
lowercase	1392	1	0	1393

There is one thing that makes our task easier: while the Unicode codes span the range 0..0x10ffff, we can check that only characters up to 0x1ffff may have different codes due to case change.

The outline of case change algorithm for UTF-32 encoded strings can be written as follows.

void utf32_uppercase(const uint32_t *input, size_t n, uint32_t *output) {
    size_t j = 0;
    for (size_t i=0; i < n; i++) {
        // fetch the character code
        const uint32_t src_code = input[i];

        if (has_uppercase_variant(src_code)) {
            switch (uppercase_letters_count(src_code)) {
                // the majority: single character
                case 1:
                    output[j++] = uppercase(src_code);
                    break;

                // rare cases: strings
                case 2:
                    output[j++] = uppercase(src_code)[0];
                    output[j++] = uppercase(src_code)[1];
                    break;

                case 2:
                    output[j++] = uppercase(src_code)[0];
                    output[j++] = uppercase(src_code)[1];
                    output[j++] = uppercase(src_code)[2];
                    break;
            }
        } else {
            // no upper
            output[j++] = src_code;
        }
    }
}

LoongArch64 subjective higlights

Sun, 19 Jan 2025 12:00:00 +0100

Introduction

I get back to work on simdutf recently, and noticed that the library gained support for LoongArch64. This is a custom design and custom ISA by Loongson from China. They provide documentation for scalar ISA, but not for the vector extension. Despite that, GCC, binutils, QEMU and other tools already support the ISA. To our luck, Jiajie Chen did an impressive work of reverse engineering the vector stuff and published results online as The Unofficial LoongArch Intrinsics Guide.

LoongArch comes with two vector extensions:

LSX, having 128-bit vector registers,
LSAX, having 256-bit vector registers.

These extensions are similar, especially most instructions present in LSX exist in LSAX. According to the Wikipedia entry, the ISA is mixture of RISC-V and MIPS.

ISA supports both integer and floating point instructions. There's support for 8-bit, 16-bit, 32-bit, 64-bit and also 128-bit integers. Floating point instructions cover single precision, double precision and half precision numbers.

Comparisons yield byte-masks, similarly to SSE.

Integer instructions are defined for most integer types, this makes the ISA regular.

My impression is that the ISA is well designed, but have not vectorized any code for that architecture. Below is the list of features I found interesting while browsing the intrinsics guide.

SIMD binary heap operations

Sat, 18 Jan 2025 12:00:00 +0100

Introduction

Binary heap is a binary tree data structure having some interesting properties. One of them is an array-friendly memory layout, achieved by building (almost) complete binary tree.

A binary heap keeps at index 0 the maximum value, or the minimum one depending on convention — let's stick to maximum heaps. There is exactly one invariant: a child node, if exist, keep a value less than the parent node. For comparison, in the case of binary search trees, we have more strict rules: the left child keeps a smaller value than the parent's value, and the right child keeps a bigger value.

A non-root node stored at index i have the parent node at index floor[(i − 1)/2], and children nodes at indices 2 ⋅ i + 1 and 2 ⋅ i + 2.

In this text we cover two procedures related to heaps:

is_heap — checks if an array is proper heap,
push_heap — adds a new element to the heap,

The procedure is_heap is vectorizable and using SIMD instructions brings profit. We also show that it is possible to define this function using forward iterators rather random iterators, as the C++ standard imposes.

The procedure push_heap can be expressed with gather and scatter, but performance is terrible. For the sake of completeness, we show the AVX-512 implementation.

There is also one more crucial method for heaps: removing the maximum value, pop_heap. However, it is difficult to vectorize, and benefits from vectorization would likely be worse than in the case of push_heap.

AVX512: printing u64 as binary

Sat, 18 Jan 2025 12:00:00 +0100

Problem

Printing 64-bit numbers in binary format can be done nicely with AVX-512 instructions. First, we populate each byte from the number into a separate 64-bit word of an AVX-512 register:

    ┌───┬───┬───┬───┬───┬───┬───┬───┐
x = │ h │ g │ f │ e │ d │ c │ b │ a │
    └───┴───┴───┴───┴───┴───┴───┴───┘
      |   |                   |   │
      │   │                   │   └──────────────────┐
      │   │                   └──────────┐           │
      │   └──────────┐                   │           │
      └──┐           │                   │           │
         │           │                   │           │
         ├─╴┈┈┈╶─┐   ├─╴┈┈┈╶─┐           ├─╴┈┈┈╶─┐   ├─╴┈┈┈╶─┐
         │       │   │       │           │       │   │       │
         ▼       ▼   ▼       ▼           ▼       ▼   ▼       ▼
       ┌───┬┈┈┈┬───┬───┬┈┈┈┬───┬┈┈┈┈┈┈┈┬───┬┈┈┈┬───┬───┬┈┈┈┬───┐
zmm0 = │ h │   │ h │ g │   │ g │       │ b │   │ b │ a │   │ a │
       └───┴┈┈┈┴───┴───┴┈┈┈┴───┴┈┈┈┈┈┈┈┴───┴┈┈┈┴───┴───┴┈┈┈┴───┘

       │           │                               │           │
       ╰─ word 7 ╶─╯                               ╰─ word 0 ╶─╯

Then, in each byte of 64-bit words we isolate i-th bit, where i is the byte position within a 64-bit word.

     ┈┈┈┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬┈┈┈
zmm0    │ 01010100 │ 01010100 │ 01010100 │ 01010100 │ 01010100 │ 01010100 │ 01010100 │ 01010100 │
     ┈┈┈┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴┈┈┈
     ┈┈┈┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬┈┈┈
zmm1    │ 10000000 │ 01000000 │ 00100000 │ 00010000 │ 00001000 │ 00000100 │ 00000010 │ 00000001 │
     ┈┈┈┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴┈┈┈

zmm0 & zmm1 =

     ┈┈┈┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬──────────┬┈┈┈
        │ 00010100 │ 01000000 │ 00000000 │ 00010000 │ 00000000 │ 00000100 │ 00000000 │ 00000000 │
     ┈┈┈┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴┈┈┈

Finally, we convert non-zero bytes into ASCII '1' (0x31) and zero bytes into ASCII '0' (0x30). This particular operation can be done in two different ways:

converting non-zero bytes to value 1 using min operation and performing a non-masked addition: byte[i] = min(byte[i], 1) + 0x30.
building a bitmask in k-register and performing standard masked instructions, like: byte[i] = mask[i] ? 0x31 : 0x30.

These two methods do not differ in performance, just the first one does not use mask registers.

Drawing trees

Sun, 12 Jan 2025 12:00:00 +0100

Problem

We have a tree of any degree and depth. Each node has assigned a bounding box of its graphical representation.

We want to draw such data structure, taking into account geometry of nodes.

Building full-text search in Javascript

Tue, 07 Jan 2025 12:00:00 +0100

Introduction

I dedicated the last few days of 2024 on refreshing my website. The project started around 2002, when the Internet was not widespread, there was no GitHub, Wikipedia or anything we know right now. Thus the website served also as a hosting platform for my open-source software.

I created custom python software to maintain both the articles and software. In the meantime things evolved. I started to write my texts in English, and publish them more in a blog style (although, I'm not a fan of the term "blog"), also GitHub allowed to easily distribute software. At some point of time my fancy system for static website become more cumbersome than helpful.

The decision was simple: drop the old build system, create a new one, uncomplicated and tailored to my current needs — focus only on publishing articles. A wise decision I made 20 years ago was picking reStructuedText to write texts. I prefer it over markdown. Not to mention that ReST allows to easily extend itself, which I found extremely handy.

Long story short, the new system allowed me to introduce tags, to maintain texts in draft mode and, last but not least, to let make perform all boring tasks.

But with the new build system, a new idea appeared: "how about searching?".

SIMD parallel bits deposit/extract

Sun, 05 Jan 2025 12:00:00 +0100

Introduction

The BMI2 extension introduced two complementary instructions: parallel bits deposit (PDEP) and parallel bits extract (PEXT).

The PDEP scatters continuous set of bits to positions denoted by the mask. The PEXT does the opposite: gathers/compresses selected bits into a continuous word.

SIMD instruction sets do not directly support this kind of operations. There is GF2P8AFFINEQB in AVX-512, that allows arbitrary bit shuffling at the byte level (see Use AVX512 Galois field affine transformation for bit shuffling).

In this text we show approaches suitable for implementing PEXT and PDEP for wider element widths on any SIMD ISA.

Dividing unsigned 16-bit numbers

Fri, 03 Jan 2025 12:00:00 +0100

Introduction

This is a follow-up text for Dividing unsigned 8-bit numbers. We checked if dividing 16-bit unsigned numbers is also feasible for SIMD instructions.

Apart from obvious path, where we use floating-point division (DIVPS), 8-bit numbers could also utilize the approximate reciprocal instruction RCPPS.

Unfortunately, for 16-bit numbers the latter instruction cannot be used directly. To properly divide 16-bit integers we need to perform a single step of the Newton-Raphson algorithm, which kills performance.

Dividing unsigned 8-bit numbers

Sat, 21 Dec 2024 12:00:00 +0100

Introduction

Division is quite an expensive operation. For instance, latency of the 32-bit division varies between 10 and 15 cycles on the Cannon Lake CPU, and for Zen4 this range is from 9 to 14 cycles. The latency of 32-bit multiplication is 3 or 4 cycles on both CPU models.

None of commonly used SIMD ISAs (SSE, AVX, AVX-512, ARM Neon, ARM SVE) provides the integer division, only RISC-V Vector Extension does. However, all these ISAs have floating point division.

In this text we present two approaches to achieve a SIMD-ized division of 8-bit unsigned numbers:

using floating point division,
using the long division algorithm.

We try to vectorize the following C++ procedure. The procedure cannot assume anything about dividends, especially if they are all equal. Thus, it is not possible to employ division by a constant.

void scalar_div_u8(const uint8_t* a, const uint8_t* b, uint8_t* out, size_t n) {
    for (size_t i=0; i < n; i++) {
        out[i] = a[i] / b[i];
    }
}

Compilers cannot vectorize it. For example GCC 14.1.0 produces the following assembly (stripped from code alignment junk):

2b40:       48 85 c9                test   %rcx,%rcx
2b43:       74 30                   je     2b75 <_Z13scalar_div_u8PKhS0_Phm+0x35>
2b45:       45 31 c0                xor    %r8d,%r8d
2b60:       42 0f b6 04 07          movzbl (%rdi,%r8,1),%eax
2b65:       42 f6 34 06             divb   (%rsi,%r8,1)
2b69:       42 88 04 02             mov    %al,(%rdx,%r8,1)
2b6d:       49 ff c0                inc    %r8
2b70:       4c 39 c1                cmp    %r8,%rcx
2b73:       75 eb                   jne    2b60 <_Z13scalar_div_u8PKhS0_Phm+0x20>
2b75:       c3                      ret

Myriad sequences of RISC-V code

Mon, 11 Nov 2024 12:00:00 +0100

Myriad sequences

The RISC-V assembler defines the pseudo-instruction li that load an immediate into a register. Unlike other pseudo-instructions, having one or a few expansions, li explodes into — as the spec says — myriad sequences.

RISC-V opcodes have 32 bits, it's impossible to encode 64-bit immediates. It's impossible to encode 32-bit immediates too, as we need to have some spare bits for the opcode itself (instruction + destination).

Assemblers have to do quite complex job, as they can only use a single register — the li argument; compilers have more freedom.

RISC-V comes with two instructions that are used to fill registers with the given value:

ADDI rd, rs1, imm12 — that adds a sign-extended 12-bit immediate to register rs1 and stores result in rd;
ADDIW rd, rs1, imm12 — likewise, but defined for RV64, i.e., the CPUs with 64-bit registers;
LUI rd, imm20 — that stores a sign-extend 20-bit immediate shifted left by 12 positions in rd; alternatively: 32-bit immediate with reset lowest 12 bits.

For easy cases we can use pick a single instruction from the above list. But when a constant fall off any of the ranges, more instructions have to be used.

RISC-V Vector Extension overview

Sat, 09 Nov 2024 12:00:00 +0100

Introduction

The goal of this text is to provide an overview of RISC-V Vector extension (RVV), and compare — when applicable — with widespread SIMD vector instruction sets: SSE, AVX, AVX-512, ARM Neon and SVE.

The RISC-V architecture defines four basic modes (32-bit, 32-bit for embedded systems, 64-bit, 128-bit) and several extensions. For instance, the support for single precision floating-point numbers is added by the F extension.

The vector extension is quite a huge addition. It adds 302 instructions plus four highly configurable load & store operations. The RVV instructions can be split into three groups:

related to masks,
integer operations,
and floating-point operations.

When a CPU does not support floating-point instructions, it still may provide the integer subset.

RVV introduces 32 vector registers v0, ..., v31, a concept of mask (similar to AVX-512), and nine control registers.

Unlike other SIMD ISAs, RVV does not explicitly define size of vector register. It is an implementation parameter (called VLEN): the size has to be a power of two, but not greater than 2¹⁶ bits. Likewise, the maximum vector element size is an implementation parameter (called ELEN, also a power of two and not less than 8 bits). For example, a 32-bit CPU might not support vectors of 64-bit values.

But generally, we may expect that a decent 64-bit CPU would support elements having 8, 16, 32 or 64-bit, interpreted as integers or floats.

Simple suggestions using popcount

Mon, 20 Nov 2023 12:00:00 +0100

Introduction

One great feature that some of CLI applications and compilers recently gained is providing suggestions in the case of misspellings arguments or options. For instance, Python suggests possible method/fields names, like:

>>> 'lower'.is_upper()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'is_upper'. Did you mean: 'isupper'?

If we want to include similar feature in our program, then a quite obvious solution is to use Levenshtein distance. Its basic implementation is simple and short. If we want suggestion for larger corpus, we may use tries to speedup matching — see: Fast and Easy Levenshtein distance using a Trie by Steve Hanov.

Despite that, I was curious if simpler algorithm would do the job. We may assume that our users know what they are doing. Their inputs will be sane, with an exception of minor mistakes. Like omitting a single letter, swapping two adjacent letters or maybe hitting a wrong key from time to time.

The two additional assumptions I made: we search suggestions in small sets, and we use only ASCII letters (128 possible bytes).

AVX-512 conflict detection without resolving conflicts

Sat, 06 May 2023 12:00:00 +0100

Introduction

One of the hardest problem in SIMD is dealing with non-continuous data accesses, that appear pretty common. Data structures based on indices, like graphs or trees, are a good example. CPU vendors introduced instructions GATHER and SCATTER to address these needs. A gather instruction builds a SIMD vector from N values loaded from N addresses. A scatter instruction stores N values from a SIMD vector at N addresses. Both instructions allow repeated indices.

Repeated indices are the real issue if an algorithm uses the SCATTER — that is, it either sets or updates values. Then we need to define how to handle repeated stores. To solve that particular problem AVX-512 introduced a complex instruction called conflict detection. The instruction builds a vector containing masks that mark repeated values in the input vector.

Intel proposed a pattern that uses the gather, scatter and conflict detection instructions to efficiently handle repeated indices. It is described in the freely available "Intel® 64 and IA-32 Architectures Optimization Reference Manual", in chapter "18.16.1 Vectorization with Conflict Detection". The problem of calculating a histogram is used there as an example.

The core of Intel's approach is a conflict resolution loop in which the repeated values are aggregated into a single element. The number of iterations varies, and depends on data: it is 0 to 4, when we process 16-item vectors (32-bit elements).

We propose a modified approach, that avoids any additional looping at the cost of additional storage. It is faster 1.4 times than the Intel algorithm when the input size is larger than 100,000 items.

The text contains a recap of AVX-512 instructions, a detailed overview of the Intel algorithm, the presentation of our procedure, and evaluation results.

All source codes are available.

Modern perfect hashing for strings

Sun, 30 Apr 2023 12:00:00 +0100

Introduction

Looking up in a static set of strings is a common problem we encounter when parsing any textual formats. Such sets are often keywords of a programming language or protocol.

Parsing HTTP verbs appeared to be the fastest when we use a compile-time trie: a series of nested switch statements. I could not believe that a perfect hash function is not better, and that led to a novel hashing approach that is based on the instruction PEXT (Parallel Bits Extract).

Briefly, when constructing a perfect hash function, we are looking for the smallest set of input bytes that can be then the input for some function combines them into a single value. The instruction PEXT allows to quickly construct any n-bit subword from a 64-bit word; the latency of the instruction is 3 CPU cycles on the current processors. This allows us to extend the schema for looking for the smallest subset of bits. This n-bit word is then the input for a function that translates the word into the desired value.

Instead of something like:

func hash(s []byte) uint64 {
    // read bytes at indices a, b, c
    // and push forward
    return hash_bytes(s[a], s[b], s[c])
}

We have:

func hash(s []byte) uint64 {
    // read bytes at indices d and e, and form a temp value
    uint64 tmp = (uint64(s[d]) << 8) | uint64(s[e])

    // from the temp value (bytes d and e) extract crucial bits
    uint64 val = PEXT(tmp, precomputed_mask)

    return hash_uint64(val)
}

Please note that depending on the strings set, the number of bytes read in both schemas can vary. It is not the rule that a bit-level hash function would touch fewer bytes than a byte-level hash function.

Apart from the above hashing schema, this text describes also constructing a compile-time hash table and compile-time switch.

All source codes are available.

PEXT recap

The instruction PEXT gets two arguments: the input word and the input mask. Bits from the input word for which the input mask is 1 are copied to the output. For example:

word:   0010101011010111
mask:   0011100100100010
masked: __101__0__0___1_
PEXT:   __________101001

SIMD-ized faster parse of IPv4 addresses

Sun, 09 Apr 2023 12:00:00 +0100

Introduction

Just for recap, an IPv4 address written in the textual form consists four decimal numbers separated by the dot character. Each number represents an octet (byte), that is in range from 0 to 255. Here are some examples: "10.1.1.12", "127.0.0.1", "255.255.255.0". An IPv4 address is stored in the big-endian byte order.

Parsing IPv4 addresses seems to be a trivial task. For example, the Go builtin module netip implements parsing with full validation in 35 lines. The inet_pton specialisation for IPv4 addresses, that can be found in Glibc, spans approx 40 lines of plain C code. For completeness, their full sources were put in appendix.

These two procedures share the same schema of parsing and validation:

Read the input string byte by byte.
When the current byte is an ASCII digit ('0' ... '9') add it to the current octet. If the octet value becomes larger than 255 or the leading zero was detected, then report an error.
When the current byte is the dot ('.'), check if we read at least one digit. If not, it's an error (for example ".1.1.20" or "192..0.12").
When the current byte is not a digit or the dot, report an error.
At the very end, check if exactly four octets were read.

I bet 99.999% of ever written IPv4 parsers use similar schema. And that is pretty hard to see any obvious inefficiency in this approach. Additionally, when we take into account that compilers are getting smarter and smarter, we may assume that a compiler would do a decent job for us.

However, it's possible to make the conversion faster, using SIMD instructions. The best solution is two-three times faster than a reference scalar procedure.

The actual C++ code snippets with SSE instructions are used to illustrate solution. Full source code is available. They include a SWAR implementation and also different SSE variants.

SWAR find any byte from set

Mon, 06 Mar 2023 12:00:00 +0100

Introduction

When I was browsing the source code of project Ada (WHATWG-compliant and fast URL parser written in modern C++) the following procedure caught my attention:

ada_really_inline size_t find_authority_delimiter_special(std::string_view view) noexcept {
  auto has_zero_byte = [](uint64_t v) {
    return ((v - 0x0101010101010101) & ~(v)&0x8080808080808080);
  };
  auto index_of_first_set_byte = [](uint64_t v) {
    return ((((v - 1) & 0x101010101010101) * 0x101010101010101) >> 56) - 1;,
  };
  auto broadcast = [](uint8_t v) -> uint64_t { return 0x101010101010101 * v; };
  size_t i = 0;
  uint64_t mask1 = broadcast('@');
  uint64_t mask2 = broadcast('/');
  uint64_t mask3 = broadcast('?');
  uint64_t mask4 = broadcast('\\');

  for (; i + 7 < view.size(); i += 8) {
    uint64_t word{};
    memcpy(&word, view.data() + i, sizeof(word));
    word = swap_bytes_if_big_endian(word);
    uint64_t xor1 = word ^ mask1;
    uint64_t xor2 = word ^ mask2;
    uint64_t xor3 = word ^ mask3;
    uint64_t xor4 = word ^ mask4;
    uint64_t is_match = has_zero_byte(xor1) | has_zero_byte(xor2) | has_zero_byte(xor3) | has_zero_byte(xor4);
    if (is_match) {
      return i + index_of_first_set_byte(is_match);
    }
  }

  if (i < view.size()) {
    uint64_t word{};
    memcpy(&word, view.data() + i, view.size() - i);
    word = swap_bytes_if_big_endian(word);
    uint64_t xor1 = word ^ mask1;
    uint64_t xor2 = word ^ mask2;
    uint64_t xor3 = word ^ mask3;
    uint64_t xor4 = word ^ mask4;
    uint64_t is_match = has_zero_byte(xor1) | has_zero_byte(xor2) | has_zero_byte(xor3) | has_zero_byte(xor4);
    if (is_match) {
      return i + index_of_first_set_byte(is_match);
    }
  }

  return view.size();
}

The above procedure finds the position of the first occurrence of a char from the set @, /, ? and /. It returns the length of input string if nothing was found.

The procedure uses SWAR techniques: it processes several bytes at once, taking advantage on the current CPUs architecture that process 64-bit values. The procedure implementation comes from src/helpers.cpp, and more function from that file follow exactly the same SWAR approach.

These two functions are crucial:

has_zero_byte is non-zero if a multi-byte word has at least one zero byte; note that the procedure also keeps only the most significant bits.
index_of_first_set_byte returns the index of first non-zero byte; it uses the fact it is called on word formed with bytes 0x00 and 0x80.

The pattern used is quite straightforward. If we bit-xor input bytes with a word filled with one of bytes from set, then the result has zero byte if the byte was there. We check then if it least one result of bit-xor has zero-byte and if it is true, we're looking for its position.

While the production code processes multi-word inputs, let's focus on a basic building block that processes a single 64-bit word.

int find_authority_delimiter_special_reference(uint64_t word) noexcept {
  auto has_zero_byte = [](uint64_t v) {
    return ((v - 0x0101010101010101) & ~(v)&0x8080808080808080);
  };
  auto index_of_first_set_byte = [](uint64_t v) {
    return ((((v - 1) & 0x101010101010101) * 0x101010101010101) >> 56) - 1;
  };
  auto broadcast = [](uint8_t v) -> uint64_t { return 0x101010101010101 * v; };
  uint64_t mask1 = broadcast('@');
  uint64_t mask2 = broadcast('/');
  uint64_t mask3 = broadcast('?');
  uint64_t mask4 = broadcast('\\');

  uint64_t xor1 = word ^ mask1;
  uint64_t xor2 = word ^ mask2;
  uint64_t xor3 = word ^ mask3;
  uint64_t xor4 = word ^ mask4;
  uint64_t is_match = has_zero_byte(xor1) | has_zero_byte(xor2) | has_zero_byte(xor3) | has_zero_byte(xor4);
  if (is_match) {
      return index_of_first_set_byte(is_match);
  }

  return -1;
}

The following assembly is produced by GCC 12.2.0 from Debian for the IceLake Server architecture (gcc -O3 -march=icelake-server).

movabs $0x2f2f2f2f2f2f2f2f,%rax
movabs $0xfefefefefefefeff,%rsi
xor    %rdi,%rax
movabs $0xd0d0d0d0d0d0d0d0,%rcx
xor    %rdi,%rcx
add    %rsi,%rax
and    %rcx,%rax
movabs $0x4040404040404040,%rcx
mov    %rdi,%rdx
xor    %rdi,%rcx
movabs $0xbfbfbfbfbfbfbfbf,%rdi
xor    %rdx,%rdi
add    %rsi,%rcx
and    %rdi,%rcx
or     %rcx,%rax
movabs $0x3f3f3f3f3f3f3f3f,%rcx
xor    %rdx,%rcx
movabs $0xc0c0c0c0c0c0c0c0,%rdi
add    %rsi,%rcx
xor    %rdx,%rdi
and    %rdi,%rcx
or     %rcx,%rax
movabs $0x5c5c5c5c5c5c5c5c,%rcx
xor    %rdx,%rcx
add    %rsi,%rcx
movabs $0xa3a3a3a3a3a3a3a3,%rsi
xor    %rsi,%rdx
and    %rcx,%rdx
or     %rdx,%rax
movabs $0x8080808080808080,%rdx
and    %rdx,%rax
je     <_Z42find_authority_delimiter_special_referencem+0xc0>
movabs $0x101010101010101,%rdx
dec    %rax
and    %rdx,%rax
imul   %rdx,%rax
shr    $0x38,%rax
dec    %eax
ret
mov    $0xffffffff,%eax
ret

The assembly contains:

11 x constants,
6 x xor,
6 x and,
4 x add,
3 x or,
1 x multiplication (imul),
1 x shift right,
1 x branch.

AVX512: finding first byte in lanes

Mon, 06 Feb 2023 12:00:00 +0100

Introduction

The problem is defined as follows: we have separate lanes (32-bit or 64-bit) and want to find the position of the first occurrence of the given byte in each lane.

For example, when we look for byte 0xaa in 32-bit lanes:

   lane 0        lane 1        lane 2       lane 4  ...
[00|aa|aa|11] [aa|aa|aa|aa] [aa|11|11|22] [11|22|33|44]
       ^^               ^^   ^^
  position 1    position 0   position 3    position 4 (not found)

The result should be a vector of uint32 = {1, 0, 3, 4, ...}.

With AVX512 an obvious solution would be producing a bitmask from byte-level comparison and then doing some permutations to convert parts of bitmasks into 32-bit values.

While it's feasible, I want to show a method that uses trick from my previous article AVX512: count trailing zeros.

Finding lowest common ancestor of two nodes

Sun, 05 Feb 2023 12:00:00 +0100

Introduction

There are several approaches to find lowest common ancestor (LCA).

The algorithm showed here does not need extra memory. There's an assumption that we can get the parent node of given node in constant time.

Faster fractional exponents

Sun, 05 Feb 2023 12:00:00 +0100

Introduction

A well known method of calculating powers of integers is based on the binary representation of an exponent. Let's consider a simple example. Exponent equals y = 9 = 0b1001; its value can be expressed as 1 ⋅ 2⁰ + 0 ⋅ 2¹ + 0 ⋅ 2² + 1 ⋅ 2³; after constant folding it simplifies to 2⁰ + 2³ = 1 + 8 = 9. Thus x⁹ can be expanded into x^{2⁰ + 2³} = x^2⁰ ⋅ x^2³ = x¹ ⋅ x⁸.

The main observation is that the product contains x^2ⁱ only if the i-th bit of exponent is 1.

An algorithm utilizing this observation is quite simple:

func powint(x float64, y uint) float64 {
    result := 1.0   // x^0
    tmp := x        // tmp is x^{2^i}

    for y > 0 {
        if y & 1 != 0 {             // i-th bit set?
            result = result * tmp   // result times x^{2^i}
        }

        // square in each iteration
        tmp = tmp * tmp

        // scan bits starting from the least significant one
        y >>= 1
    }

    return result
}

Exactly the same schema can be used if an exponent is fractional; for simplicity let's assume the exponent is positive and less than 1.

We also use binary representation of fraction, we just need to remember that weights of bits are fractions: x^{2^{− i}}. They equals 1/2, 1/4, 1/8, 1/16, 1/32, and so on, so forth.

The algorithm is almost identical: we scan bits starting from the most significant one — bit weights are decreasing by factor 1/2. Value x^½ is a square root.

Let's assume that we have a fraction expressed as a uint64, where the decimal dot is before the most significant bit. For instance fraction 0.1010111₂ would have the following representation as uint64:

decimal dot
|
[10101110|00000000|00000000|00000000|00000000|00000000|00000000|00000000]
 ||│                                                                   |
 ││└- bit 61, weight 1/8                                           bit 0
 │└─╴ bit 62, weight 1/4
 └──╴ bit 63, weight 1/2

In the next section we will see how to convert a normalized float into that representation.

The algorithm is:

func powfracaux(x float64, fraction uint64) float64 {
    res := 1.0              // res = 2^0
    sq := x
    for fraction != 0 {
        sq = math.Sqrt(sq)       // sq = x^(1/2^i)

        if int64(fraction) < 0 { // i-th bit set (MSB)
            res *= sq            // update result
        }

        fraction <<= 1
    }

    return res
}

Converting binary fraction to ratio

Sun, 05 Feb 2023 12:00:00 +0100

Introduction

Suppose we have a binary fraction, that is positive and less than 1:

0.101011000┈┈┈ = 0.671875
  | | |│
  │ │ │└─╴ 1/2^6
  │ │ └──╴ 1/2^5
  │ └────╴ 1/2^3
  └──────╴ 1/2^1

We want to express it as a ratio of two integer numbers.

AVX512: count trailing zeros

Tue, 31 Jan 2023 12:00:00 +0100

Introduction

AVX512 lacks of counting trailing zeros; it supports counting of leading zeros via instruction VPLZCNTD in 32- and 64-bit words. There is the scalar instruction BSF (Bit Scan Forward).

To recall how counting the leading zeros is supposed to work, let's see sample 32-bit word:

 bit 31                       bit 0
 |                                |
[0000|0001|0111|1000|0011|1100|0000|
                            ^^ ^^^^
                       6 trailing zeros

An obvious solution would be reversing bits in a word and then use the VPLZCNT instruction. Source code for this approach is shown in a following section. Basically, with PSHUFB we can reverse order of bytes. Then, with two additional invocations of that instruction, we swap order of bits within bytes.

However, a solution which is really clever uses population count; it is explained in the next section.

AVX512: check if value belongs to a set

Sat, 21 Jan 2023 12:00:00 +0100

Introduction

We want to check if a value belongs to a set. More formally, we want to evaluate the following expression: (x == word_0) or (x == word_1) or ... or (x == word_n), where x is a vector of words, and word_i is a constant vector.

For a four-element set, a naive version of AVX512 assembly code:

VPCMPD.BCST   $0, (AX),   Z1, K1        // K1 = Z1 == word_0
VPCMPD.BCST   $0, 4(AX),  Z1, K2        // K2 = Z1 == word_1
VPCMPD.BCST   $0, 8(AX),  Z1, K3        // K3 = Z1 == word_2
VPCMPD.BCST   $0, 12(AX), Z1, K4        // K4 = Z1 == word_3
KORW          K1, K2, K1                // K1 = K1 | K2
KORW          K3, K4, K3                // K3 = K3 | K4
KORW          K1, K3, K1                // K1 = K1 | K3

The above code tests a vector register Z1 against const values stored in an array pointed by AX, and sets result in kreg K1.

The tool uICA reports the following timings (for Skylake-X):

Throughput (in cycles per iteration): 4.00
Bottlenecks: Decoder, Ports

┌───────────────────────┬────────┬───────┬───────────────────────────────────────────────────────────────────────┐
│ MITE   MS   DSB   LSD │ Issued │ Exec. │ Port 0   Port 1   Port 2   Port 3   Port 4   Port 5   Port 6   Port 7 │
├───────────────────────┼────────┼───────┼───────────────────────────────────────────────────────────────────────┤
│  2                    │   2    │   2   │                     1                          1                      │ vpcmpeqd k1, zmm1, dword ptr [rax]{1to16}
│  2                    │   2    │   2   │                              1                 1                      │ vpcmpeqd k2, zmm1, dword ptr [rax+0x4]{1to16}
│  2                    │   2    │   2   │                     1                          1                      │ vpcmpeqd k3, zmm1, dword ptr [rax+0x8]{1to16}
│  2                    │   2    │   2   │                              1                 1                      │ vpcmpeqd k4, zmm1, dword ptr [rax+0xc]{1to16}
│  1                    │   1    │   1   │   1                                                                   │ korw k1, k2, k1
│  1                    │   1    │   1   │   1                                                                   │ korw k3, k4, k3
│  1                    │   1    │   1   │   1                                                                   │ korw k1, k3, k1
├───────────────────────┼────────┼───────┼───────────────────────────────────────────────────────────────────────┤
│  11                   │   11   │  11   │   3                 2        2                 4                      │ Total
└───────────────────────┴────────┴───────┴───────────────────────────────────────────────────────────────────────┘

AVX512: generating constants

Thu, 19 Jan 2023 12:00:00 +0100

Introduction

AVX512 code often needs constants that repeat in 64-, 32-, 16- or 8-bit lanes. Instead of pre-computing such constants as whole 64-byte arrays, we can reduce memory consumption by using explicit or implicit broadcasts.

While broadcasts have high throughput, their latencies are quite high. According to uops.info, latencies are 5 cycles (on Skylake-X, Cannon Lake, Ice Lake, Adler Lake-P) when broadcasting from either a 32-bit or 64-bit register. When the source is memory location, latencies are even higher.

When an AVX512 procedure is quite short, or often loads different constants from memory, broadcast latencies might become visible. To overcome this problem, we might compute some values using few cheap instructions.

We can quickly fill an AVX512 all zero bits (with XOR) or ones (with ternary log instruction) and then use shifts, bit operations and other instructions to construct desired value.

This article show some ways to calculate different simple constants. Examples are focused mostly on 32-bit values, although in most cases we might generalize the methods to other lane widths.

AVX512: histogram of sixteen nibbles

Fri, 06 Jan 2023 12:00:00 +0100

Introduction

At Sneller we had the following problem: there are sixteen 4-bit values, we need a histogram for that limited set.

Since the input set has the fixed size of 64 bits, the problem is not that difficult as a generic case.

The basic problem we solve is counting how many nibbles are present in the given 64-bit set. Then, the solution to the initial problem is performing the basic step for each possible nibble value. And it's is done in parallel.

Faster hack

Mon, 31 Jan 2022 12:00:00 +0100

The other day I came across the following line:

len * ("11124811248484"[type < 14 ? type:0]-'0') > 4;

My first reaction was "WTF", but later I realized that such a hackish code has to be a response to poor compiler optimizations. And taking into account that the code might be quite old, this is a perfect solution. Only a bit unreadable.

In fact we have something like len * coefficient(type) > 4, where coefficient is a value from the set {1, 2, 4, 8}.

I want to show that this problem can be solved not only cleaner but also faster.

First, let's examine the compiler output for the original expression. We assume both variables are unsigned integers.

bool fun1(unsigned int len, unsigned int type) {
    return len * ("11124811248484"[type < 14 ? type:0]-'0') > 4;
}

And the output from gcc -O3 -march=skylake -s; GCC version is 10.2.1.:

    cmpl    $13, %esi
    ja      .L2
    movl    %esi, %esi
    leaq    .LC0(%rip), %rax        # coef = "11124811248484"[type] if type < 14
    movsbl  (%rax,%rsi), %eax
    subl    $48, %eax               # coef -= '0'
    imull   %eax, %edi              # edi = len * coef
.L2:
    cmpl    $4, %edi                # len * coef > 4
    seta    %al
    ret

It's worth to note that the compiler knows that for type >= 14, we always fetch the value "11124811248484"[0] - '0' that equals one. Thus, jumps instantly to the comparison.

Optimization #1

We may omit subtraction of constant '0', if we don't use ASCII digits, but hex strings. Maybe for older compiler we need to use octal digits. Either way, we avoid a subtraction.

bool fun2(unsigned int len, unsigned int type) {
    if (type >= 14) {
        type = 0;
    }
    const unsigned coef = "\x01\x01\x01\x02\x04\x08\x01\x01\x02\x04\x08\x04\x08\x04"[type];
    return (len * coef) > 4;
}

The assembly code:

    cmpl    $13, %esi
    ja      .L5
    movl    %esi, %esi
    leaq    .LC1(%rip), %rax
    movsbl  (%rax,%rsi), %eax
    imull   %eax, %edi
.L5:
    cmpl    $4, %edi
    seta    %al
    ret

Optimization #2

We removed one instruction. Can we do it better? The comparison type < 14 cannot be avoided,b ut the constants are {1, 2, 4, 8}. All are powers of two, thus we can replace the multiplication with a binary shift left just by adjusting the constants.

bool fun3(unsigned int len, unsigned int type) {
    if (type >= 14) {
        type = 0;
    }
    const unsigned shift = "\x00\x00\x00\x01\x02\x03\x00\x00\x00\x01\x02\x03\x02\x03\x02"[type];
    return (len << shift) > 4;
}

The assembly code:

    cmpl    $13, %esi
    ja      .L7
    movl    %esi, %esi
    leaq    .LC2(%rip), %rax
    movsbl  (%rax,%rsi), %eax
    shlx    %eax, %edi, %edi
.L7:
    cmpl    $4, %edi
    seta    %al
    ret

Since we are targeting the Skylake, it was possible to use the instruction SHLX from the BMI extensions. The instruction performs shift left, but does not alter the CPU flags register. By not doing this, it does not create any indirect dependencies between instructions.

Optimization #3

Seems we reached the end? Not really. We are calculating four cases:

len * 1 > 4
len * 2 > 4
len * 4 > 4
len * 8 > 4

Remembering that len is an unsigned integer, we may rewrite the expressions:

len > 5     # 5 * 1 > 4
len > 3     # 3 * 2 > 4
len > 2     # 2 * 4 > 4
len > 1     # 1 * 8 > 4

We just figured out the minimum value of len for which the expression len * coefficient > 4 is true. As a result, we may get rid of multiplication/shift.

bool fun4(unsigned int len, unsigned int type) {
    if (type >= 14) {
        type = 0;
    }
    const unsigned bound = "\x05\x05\x05\x03\x02\x01\x05\x05\x03\x02\x01\x02\x01\x02"[type];
    return len > bound;
}

The compiler output:

    movl    $5, %eax            # bound = lookup[0]
    cmpl    $13, %esi           # type >= 14
    ja      .L9
    movl    %esi, %esi
    leaq    .LC3(%rip), %rax
    movsbl  (%rax,%rsi), %eax   # bound = lookup[type]
.L9:
    cmpl    %eax, %edi          # len > cond
    seta    %al
    ret

Conclusions:

All in all, the trick with subscripting a string is neat. This is what a compiler may do underneath for switch statements, but it is not guaranteed.
Clang uses conditional moves instead of jumps.

Fast parsing HTTP verbs

Sat, 29 Jan 2022 12:00:00 +0100

Introduction

When we started to use boost::beast library at work, obviously I downloaded its source code. Can't say I am good at navigation across boost libraries, I was just opening random files waiting for compilation completion. My attention was caught by procedure string_to_verb, that translates a HTTP verb into a number, in fact an enum. The most common verbs are GET POST, PUT and DELETE, however in total there are 33 verbs, the longest ones have 12 characters.

Why the boost implementation seemed to me to be odd? It's basically a hardcoded Trie. There is the main switch statement that selects a subtree based on the first character. Then:

When there is exactly one verb starting with given letter, just equality of the suffix is checked. For example "BIND" or "TRACE" are handled like this.
When more words start with the same letter but they don't share any prefix — there's a plain ladder of if checking equality of the suffix. For example pairs of verbs "LINK" and "LOCK" or "SEARCH" and "SUBSCRIBE" are matched in this way.
Otherwise, there are more switches resolving the subsequent letters. This is how a group of verbs "PATCH", "POST", "PROPFIND", "PROPPATCH", "PURGE" and "PUT" is matched.

This looks like a lot of branches and we do know that branches might be a source of performance problems. A mispredicted branch penalty is several CPU cycles. I thought it would be good to check if other solutions would be better.

The first solution is to match not a single letter prefix, but take into account four or eight characters at time. It costs exactly one load and comparison. The only drawback is padding with zeros strings shorter than 4 chars.
Since the set of verbs is small and given statically, we may build a minimal perfect hash function (MPH). GNU gperf can be used to generate a C++ program implementing a MPH. The major drawback of gperf is that it generates function "exists", while we need a "lookup". I had to manually edit the generated program.

DDoS-ed by a service runs on AWS

Sat, 29 Jan 2022 12:00:00 +0100

Every year in January I download the Apache logs for my home page. I run a simple script on the logs to see which articles were popular last year. Sometimes I'm surprised that some old stuff is still being read.

The script counts the total number of visits and the unique number of visitors. And this year something strange happened. Below is the head of list:

total  uniq
651522 7864   /notesen/2021-12-22-test-and-clear-bit.html
9008   5142   /articles/simd-strfind.html
5478   3521   /notesen/2019-01-07-cpp-read-file.html
4414   3348   /notesen/2021-01-18-autovectorization-gcc-clang.html
3773   2008   /articles/simd-parsing-int-sequences.html
3270   1897   /notesen/2018-10-03-simd-index-of-min.html
2948   1870   /notesen/2019-02-02-autovectorization-gcc-clang.html
3539   1825   /articles/index.html
3843   1617   /articles/sse-popcount.html
1908   1287   /notesen/2021-03-11-any-word-is-zero.html
1982   1244   /notesen/2016-01-12-sse-base64-encoding.html
2202   1229   /articles/simd-byte-lookup.html
2060   1220   /notesen/2018-10-24-sse-sumbytes.html
2072   1202   /notesen/2021-02-02-all-bytes-in-reg-are-equal.html
1998   1103   /articles/avx512-ternary-functions.html

The trend is clear — a thousand visits is normal, few thousands is something very popular. But the top article got visited 600 thousands times! It was added on Hacker News, but never hit its main page (gained only ~30 upvotes).

Below is a daily number of visits:

/notesen/2021-12-22-test-and-clear-bit.html
2021-12-22:     360
2021-12-23:   69468 =============================
2021-12-24:  141027 ============================================================
2021-12-25:  138335 ==========================================================
2021-12-26:  137856 ==========================================================
2021-12-27:  136672 ==========================================================
2021-12-28:   27736 ===========
2021-12-29:      37
2021-12-30:      17
2021-12-31:      14
total: 651522

For whole four days, every ~2 seconds, somebody performed HTTP GET on this very URL.

I collected the IPs visiting the article. Almost all of the IPs are registered by Amazon Web Services.

Below is the detailed histogram of visits per IP. An catching-eye pattern is that the number of visits is shared across groups of different IPs. Looks like somebody did a silly synchronisation mistake and let workers pick the same same job over and over.

count    IP
5724     44.201.39.42
5724     3.94.193.105
4770     44.201.33.147
3816     54.91.111.64
3816     54.84.75.180
3816     54.221.21.59
3816     54.196.220.147
3816     54.175.245.212
3816     44.192.16.27
3816     3.87.247.68
3816     3.80.213.10
3816     3.237.21.87
3816     3.236.190.167
3816     3.236.104.169
3816     3.235.181.101
3816     18.204.227.120
2862     54.87.77.182
2862     54.85.161.10
2862     54.221.43.86
2862     54.209.115.198
2862     54.164.131.116
2862     54.161.64.49
2862     52.91.89.169
2862     52.87.191.130
2862     52.21.158.202
2862     44.200.72.132
2862     44.200.26.192
2862     44.200.187.40
2862     44.200.143.177
2862     44.197.188.220
2862     3.90.183.142
2862     3.88.173.14
2862     3.85.125.218
2862     3.84.155.160
2862     34.239.146.120
2862     34.228.40.63
2862     3.239.96.83
2862     3.239.219.229
2862     3.236.241.160
2862     3.236.207.16
2862     3.236.141.1
2862     3.236.131.194
2862     3.236.119.211
2862     3.233.219.239
2862     3.230.127.79
2862     3.227.254.57
2862     3.227.235.211
2862     3.227.11.80
2862     3.226.76.105
2862     3.220.231.2
2862     3.216.79.148
2862     18.233.148.19
2862     18.215.147.137
2862     18.214.37.67
2862     18.207.252.143
2862     18.205.159.11
2862     100.26.61.210
2862     100.26.185.13
1909     3.236.27.213
1908     54.242.255.17
1908     54.242.228.33
1908     54.242.163.248
1908     54.236.36.193
1908     54.205.82.147
1908     54.175.4.154
1908     54.174.240.173
1908     54.174.150.40
1908     54.172.41.64
1908     54.167.8.206
1908     54.167.39.17
1908     54.166.44.219
1908     54.161.206.139
1908     54.160.234.43
1908     54.159.84.255
1908     54.147.117.226
1908     50.19.161.70
1908     44.201.62.226
1908     44.201.58.131
1908     44.200.86.209
1908     44.200.65.76
1908     44.200.40.176
1908     44.200.239.183
1908     44.200.230.223
1908     44.200.227.211
1908     44.200.205.85
1908     44.200.188.45
1908     44.200.173.245
1908     44.200.171.203
1908     44.199.232.199
1908     44.197.133.168
1908     44.195.38.186
1908     44.193.83.155
1908     44.192.82.83
1908     44.192.58.57
1908     44.192.38.32
1908     44.192.131.229
1908     3.95.182.93
1908     3.95.179.4
1908     3.92.88.127
1908     3.92.190.61
1908     3.91.63.171
1908     3.91.132.53
1908     3.89.228.171
1908     3.89.141.217
1908     3.88.7.101
1908     3.88.66.222
1908     3.88.40.185
1908     3.87.80.192
1908     3.86.216.188
1908     3.85.59.14
1908     3.85.224.21
1908     3.85.162.102
1908     3.85.13.203
1908     3.84.19.250
1908     3.83.133.228
1908     3.80.68.17
1908     35.175.110.3
1908     35.174.4.162
1908     35.173.1.105
1908     35.171.146.181
1908     35.170.198.64
1908     35.168.58.80
1908     35.168.23.194
1908     35.153.39.124
1908     34.238.193.239
1908     34.236.171.38
1908     34.236.144.152
1908     34.231.21.251
1908     34.230.79.25
1908     34.229.193.190
1908     34.228.141.13
1908     34.227.111.192
1908     34.205.4.165
1908     34.201.24.65
1908     34.201.21.89
1908     34.200.237.89
1908     3.239.98.150
1908     3.239.27.216
1908     3.239.236.9
1908     3.239.195.119
1908     3.238.30.227
1908     3.238.28.93
1908     3.238.239.14
1908     3.238.233.181
1908     3.238.149.106
1908     3.237.30.128
1908     3.237.28.58
1908     3.237.197.197
1908     3.236.88.202
1908     3.236.29.198
1908     3.236.172.81
1908     3.236.113.37
1908     3.235.232.232
1908     3.234.141.7
1908     3.231.60.92
1908     3.231.219.59
1908     3.231.208.60
1908     3.227.252.65
1908     3.221.155.113
1908     3.210.205.79
1908     3.209.12.110
1908     23.20.132.70
1908     184.73.60.65
1908     184.73.57.197
1908     18.234.236.146
1908     18.234.111.35
1908     18.213.118.61
1908     18.212.159.84
1908     18.209.237.95
1908     18.208.249.16
1908     18.208.209.143
1908     18.207.104.102
1908     18.207.100.231
1908     18.206.228.51
1908     18.205.238.85
1908     18.205.107.9
1908     18.204.35.218
1908     18.204.215.122
1908     18.204.17.190
1908     100.26.147.44
1908     100.24.45.26
1908     100.24.18.194
 955     44.200.107.134
 954     54.91.114.177
 954     54.90.80.165
 954     54.90.201.186
 954     54.90.124.119
 954     54.89.101.235
 954     54.88.23.164
 954     54.86.217.46
 954     54.86.115.15
 954     54.83.241.217
 954     54.82.90.177
 954     54.82.164.242
 954     54.242.227.105
 954     54.242.140.162
 954     54.236.217.111
 954     54.227.40.231
 954     54.226.62.158
 954     54.226.122.88
 954     54.226.0.34
 954     54.221.180.78
 954     54.211.152.69
 954     54.209.207.116
 954     54.209.189.47
 954     54.208.171.144
 954     54.175.53.23
 954     54.175.195.174
 954     54.174.44.66
 954     54.166.7.228
 954     54.166.159.234
 954     54.163.187.47
 954     54.160.0.105
 954     54.157.193.120
 954     54.152.146.248
 954     54.145.24.108
 954     54.145.223.209
 954     52.91.93.152
 954     52.91.61.246
 954     52.91.226.148
 954     52.91.196.215
 954     52.90.54.249
 954     52.90.3.62
 954     52.90.192.144
 954     52.87.250.143
 954     52.87.249.92
 954     52.72.121.162
 954     52.55.65.234
 954     52.23.167.2
 954     52.23.158.119
 954     52.207.62.132
 954     52.207.250.70
 954     52.207.228.111
 954     52.202.9.106
 954     50.19.41.32
 954     44.201.7.198
 954     44.201.5.78
 954     44.201.56.67
 954     44.201.34.223
 954     44.201.31.199
 954     44.201.3.1
 954     44.201.18.138
 954     44.201.10.172
 954     44.201.0.117
 954     44.200.53.196
 954     44.200.50.199
 954     44.200.40.24
 954     44.200.39.160
 954     44.200.32.176
 954     44.200.251.208
 954     44.200.228.186
 954     44.200.218.184
 954     44.200.195.52
 954     44.200.144.107
 954     44.200.143.106
 954     44.200.137.243
 954     44.200.112.188
 954     44.199.250.90
 954     44.199.236.15
 954     44.198.59.240
 954     44.197.250.137
 954     44.197.245.138
 954     44.193.2.184
 954     44.192.84.100
 954     44.192.76.80
 954     44.192.57.38
 954     3.94.202.83
 954     3.93.81.203
 954     3.92.45.247
 954     3.92.217.211
 954     3.92.207.122
 954     3.92.132.193
 954     3.91.55.178
 954     3.91.45.166
 954     3.91.219.210
 954     3.89.55.35
 954     3.89.223.189
 954     3.89.101.5
 954     3.88.173.142
 954     3.87.219.112
 954     3.87.201.92
 954     3.87.152.148
 954     3.85.241.232
 954     3.85.13.102
 954     3.84.30.31
 954     3.83.161.181
 954     3.82.139.5
 954     3.80.79.219
 954     3.80.5.219
 954     3.80.220.39
 954     3.80.118.234
 954     35.175.200.233
 954     35.173.48.63
 954     35.173.47.212
 954     35.173.250.34
 954     35.173.134.240
 954     35.172.219.110
 954     35.172.217.161
 954     35.170.73.138
 954     35.170.64.84
 954     34.239.175.219
 954     34.239.130.54
 954     34.239.125.189
 954     34.238.83.186
 954     34.238.247.118
 954     34.238.151.112
 954     34.236.143.245
 954     34.234.223.149
 954     34.230.80.248
 954     34.229.175.140
 954     34.229.172.210
 954     34.229.168.118
 954     34.228.143.140
 954     34.227.11.197
 954     34.227.110.19
 954     34.224.41.142
 954     34.224.21.214
 954     34.207.93.237
 954     34.204.200.137
 954     34.204.189.163
 954     34.204.186.183
 954     34.203.245.120
 954     34.203.200.103
 954     34.203.199.145
 954     34.203.14.139
 954     34.200.222.190
 954     3.239.79.175
 954     3.239.76.99
 954     3.239.51.238
 954     3.239.119.61
 954     3.239.116.62
 954     3.239.11.11
 954     3.238.249.185
 954     3.238.244.136
 954     3.238.23.31
 954     3.238.228.216
 954     3.238.221.170
 954     3.238.194.160
 954     3.238.177.225
 954     3.238.150.175
 954     3.238.13.205
 954     3.238.113.12
 954     3.237.238.142
 954     3.237.10.223
 954     3.237.10.2
 954     3.237.10.145
 954     3.236.80.163
 954     3.236.54.153
 954     3.236.16.119
 954     3.236.142.65
 954     3.236.121.4
 954     3.236.120.156
 954     3.236.115.165
 954     3.236.101.79
 954     3.235.226.5
 954     3.235.199.76
 954     3.235.193.114
 954     3.235.192.58
 954     3.235.182.96
 954     3.235.178.120
 954     3.234.244.141
 954     3.234.239.170
 954     3.233.241.219
 954     3.233.234.70
 954     3.231.21.251
 954     3.230.159.197
 954     3.227.242.133
 954     3.227.11.186
 954     3.224.147.147
 954     3.223.3.71
 954     3.223.195.119
 954     3.219.217.120
 954     3.215.22.124
 954     3.209.10.196
 954     3.208.15.210
 954     184.72.114.127
 954     18.234.227.246
 954     18.234.139.137
 954     18.232.54.171
 954     18.232.177.100
 954     18.215.63.197
 954     18.215.189.70
 954     18.215.163.85
 954     18.215.159.83
 954     18.214.26.230
 954     18.212.242.2
 954     18.209.59.182
 954     18.208.195.208
 954     18.208.148.189
 954     18.207.241.17
 954     18.207.160.105
 954     18.207.143.8
 954     18.207.129.148
 954     18.206.254.83
 954     18.206.16.17
 954     18.205.105.108
 954     107.22.155.204
 954     107.22.134.56
 954     107.21.166.150
 954     100.27.49.48
 954     100.27.21.180
 954     100.26.35.56
 954     100.26.185.73
 954     100.24.25.254
 954     100.24.21.216
 954     100.24.209.103
 240     34.226.188.26
 238     3.218.212.35
 228     158.69.195.206
 173     35.203.245.218
 124     136.243.70.68

The end. :)

AVX512VBMI2 and packed varuint format

Mon, 24 Jan 2022 12:00:00 +0100

Introduction

A quite popular varuint format lets to save an arbitrary integer number on a sequence of bytes. Each byte stores seven bits of information, and the most significant bit indicates whether the given byte is the last one.

Decoding such numbers is quite easy, but is not fast. This is the reason why Google came up with their packed varint format, that stores four numbers (from 1 to 4 byte each). In this format control bits and data bits are separated. The control bits are grouped into single byte: four pairs of bits encode lengths of four numbers.

Handling this format is way easier and is vectorizable. The control byte is used to fetch a shuffle pattern, which is then issued to PSHUFB. Then, this single instruction expands 4-16 data bytes into sixteen 32-bit numbers. Details are shown in the next section.

The packed format can be slightly modified to utilize the instruction VPEXPANDB defined in AVX512VBMI2. The instruction expands bytes according to an AVX512 write mask — it's exactly what the packed varint format needs.

Parsing hex numbers with validation

Mon, 17 Jan 2022 12:00:00 +0100

Introduction

A non-validating parsing from hexadecimal string can be vectorized: Using SSE to convert from hexadecimal ASCII to number. This text shows two validating and vectorized approaches.

Parsing algorithms convert 16-byte inputs. They consists two major parts:

Validate and convert ASCII digits and letters into 4-bit values (value [0..15]) stored on separate bytes.
Merge the 4-bit words into a continues 64-bit word.

Below are the valid ASCII hexadecimal digits:

0 ... 9 — 0x30 ... 0x31,
a ... f — 0x61 ... 0x66,
A ... F — 0x41 ... 0x46.

We can note that:

Valid characters have higher nibbles equal to 0x3, 0x4 or 0x6.
Values of decimal digits equal to the lower nibble.
Values of hex digits (above 9) equal to the lower nibble plus 9.

Bit test and reset vs compilers

Wed, 22 Dec 2021 12:00:00 +0100

Problem

The problem: there is a bitmask (16-, 32-, 64-bit). We need to scan it backward, starting from the most significant bit. We finish at the last set bit. For instance we scan bits from 15 to 11 in a 16-bit mask 0b1100'1000'0000'0000. Depending on the bit's value we perform different tasks.

Since x86 has instruction BTR it was obvious for me that I should use the idiom bit-test-and-reset. Thus my initial code was straightforward.

void loop_v1(uint64_t mask) {
    for (int i=63; i >= 0 && mask != 0; i--) {
        if (test_and_clear_bit(mask, i))
            func_true(i);
        else
            func_false(i);
    }
}

Function test_and_clear_bit wraps the BTR instruction. Below is an example how this function behaves.

uint16_t w = 0b0000'0010'1110'0100;
bool b;

// bit #1 is zero:
// - b == false
// - w == 0b0000'0010'1110'0100 — unchanged
b = test_and_clear_bit(w, 1);

// bit #2 is set:
// b == true
// - w == 0b0000'0010'1110'0000
b = test_and_clear_bit(w, 2);

// bit #10 is set:
// - b == true
// - w == 0b0000'0000'1110'0000
b = test_and_clear_bit(w, 2);

Conversion uint32 into decimal without division nor multiplication

Tue, 23 Nov 2021 12:00:00 +0100

Introduction

This is follow up to Daniel Lemire's Converting integers to fix-digit representations quickly.

The method described here does not use multiplication nor division instructions. It relies only on addition and byte-level comparison. It's weird and slow, though.

The main idea is to work directly on the BCD representation. First, we pre-calculate BCD images (16-byte arrays) for individual bytes of a 32-bit number. The following values are considered:

0x00, 0x01, ..., 0xff;
0x0000, 0x0100, ..., 0xff00;
0x000000, 0x010000, ..., 0xff0000;
0x00000000, 0x01000000, ..., 0xff000000;

Then, when converting a number, we fetch the BCD images and add them together.

The next step of algorithm is fixing up the sum, as some bytes might be greater then 9. After this step all bytes are in range 0 .. 9.

The last step is simple conversion into ASCII by adding ord('0') = 0x30.

Sample scalar implementation is shown below.

void itoa_divless(uint32_t x, char* buffer) {
    union {
        uint64_t qword[2];
        uint8_t bytes[16];
    };

    qword[0] = qword[1] = 0;

    // 1. obtain BCD representation of all bytes
    {
        const uint8_t byte0 = x & 0xff;
        qword[1] += *(uint64_t*)&lookup0[byte0][0];
    }
    {
        const uint8_t byte1 = (x >> 8) & 0xff;
        qword[1] += *(uint64_t*)&lookup1[byte1][0];
    }
    {
        const uint8_t byte2 = (x >> 16) & 0xff;
        qword[0] += *(uint64_t*)&lookup2[byte2][0];
        qword[1] += *(uint64_t*)&lookup2[byte2][8];
    }
    {
        const uint8_t byte3 = (x >> 24) & 0xff;
        qword[0] += *(uint64_t*)&lookup3[byte3][0];
        qword[1] += *(uint64_t*)&lookup3[byte3][8];
    }

    // 2. fixup BCD & store result
    uint8_t carry = 0;
    for (int i=15; i >= 0; i--) {
        const uint8_t byte = bytes[i] + carry;
        if (byte >= 30) {
            buffer[i] = byte - 30 + '0';
            carry = 3;
        } else if (byte >= 20) {
            buffer[i] = byte - 20 + '0';
            carry = 2;
        } else if (byte >= 10) {
            buffer[i] = byte - 10 + '0';
            carry = 1;
        } else {
            buffer[i] = byte + '0';
            carry = 0;
        }
    }
}

Example

Let x = 20211121 = 0x13465b1. We split the value into separate bytes 0xb1, 0x65, 0x34 and 0x01.

Then, for each byte, we fetch the appropriate BCD image:

0xb1 => [ 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 1| 7| 7] (177)
0x65 => [ 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 5| 8| 5| 6] (101 * 256 = 25'856)
0x34 => [ 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 4| 0| 7| 8| 7| 2] (52 * 65536 = 3'407'872)
0x01 => [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 6| 7| 7| 7| 2| 1| 6] (1 * 16777216 = 16'777'216)

This step requires six 64-bit loads. For byte #0 and byte #1 the higher 8 bytes of BCD image are always zero. For bytes #2 and #3 all 16 bytes of images are required.

Once we have all the BCD images, we simply add them together. We have four inputs, where none of bytes exceed 9, thus it's safe to perform 64-bit additions.

For our sample data we have:

[ 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 1| 7| 7]
[ 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 2| 5| 8| 5| 6] +
[ 0| 0| 0| 0| 0| 0| 0| 0| 0| 3| 4| 0| 7| 8| 7| 2] +
[ 0| 0| 0| 0| 0| 0| 0| 0| 1| 6| 7| 7| 7| 2| 1| 6] +
--------------------------------------------------
[ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|11| 9|19|19|20|21]

There are some bytes greater than 9, we need to fix them up:

t0 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|11| 9|19|19|20|21]
t1 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|11| 9|19|19|22| 1] -- carry 2 from #0 to #1
t3 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|11| 9|19|21| 2| 1] -- carry 2 from #1 to #2
t4 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|11| 9|21| 1| 2| 1] -- carry 2 from #2 to #3
t5 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|11|11| 1| 1| 2| 1] -- carry 2 from #3 to #4
t6 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1| 9|12| 1| 1| 1| 2| 1] -- carry 1 from #4 to #5
t7 = [ 0| 0| 0| 0| 0| 0| 0| 0| 1|10| 2| 1| 1| 1| 2| 1] -- carry 1 from #5 to #6
t8 = [ 0| 0| 0| 0| 0| 0| 0| 0| 2| 0| 2| 1| 1| 1| 2| 1] -- carry 1 from #6 to #7

The carry value between bytes never exceeds 3. Since we have four inputs, then maximum value of byte at 0th position is 4*9 = 36. Any subsequent carry value cannot be greater than 3, as 4*9 + 3 is 39.

This means that the carry value can be obtained with a series of comparisons.

How to check if any word is zero

Thu, 11 Mar 2021 12:00:00 +0100

Problem

We want to check if at least one integer value is zero. In other words, we are evaluating following expression (for reasonably small N):

if (x0 == 0 or x1 == 0 or ... or xN == 0)
    ...

For a three-argument expression GCC 10.2 produces (with -O3 switch) the following x86 code:

testl   %esi, %esi  ; esi == 0?
sete    %al         ; al = 1 if the above condition is true, 0 otherwise
testl   %edx, %edx
sete    %dl
orl     %edx, %eax  ; plain `or`
testl   %edi, %edi
sete    %dl
orl     %edx, %eax

Clang 11.0 generates almost identical code:

testl   %edi, %edi
sete    %al
testl   %esi, %esi
sete    %cl
orb     %al, %cl
testl   %edx, %edx
sete    %al
orb     %cl, %al

ICC and MSVC added some jumps, but generally also use basic building block: test followed by a conditional set sete.

Autovectorization status in MSVC in 2021

Wed, 17 Feb 2021 12:00:00 +0100

Introduction

This year I re-checked the status of autovectorization in the latest GCC and Clang. MSVC was omitted because I didn't see any new version of this compiler on godbolt. More precisely, I didn't believe that there is a difference between versions 19.28 and 19.16 (that was tested two years ago).

Harold Aptroot pointed out that there are some differences in code generated for the AVX2 target. Additionally, in 2020 MSVC started to support AVX512. These two reasons forced me to recheck MSVC too.

Counting byte in byte stream with AVX512BW instructions

Sun, 14 Feb 2021 12:00:00 +0100

Introduction

This is a follow up to article SIMDized counting byte in byte stream. In this article only AVX512BW variants are discussed. Performance is analyzed only for the Skylake-X CPU.

We want to count how many times given byte appears in a byte stream. The following C++ code shows the naive algorithm:

size_t countbyte(uint8_t* data, size_t size, uint8_t b) {
    size_t result = 0;
    for (size_t i=0; i < size; i++)
        result += (data[i] == b);

    return result;
}

How to detect if all bytes in SIMD register are the same?

Tue, 02 Feb 2021 12:00:00 +0100

Introduction

We want to detect if all bytes stored in a SIMD register (SSE, AVX2, AVX512, Neon etc.) are the same. For example for byte layout in an SSE register like this:

[42|42|42|42|42|42|42|42|42|42|42|42|42|42|42|42]

We see that all bytes are equal to 42. For this one not all bytes have the same value:

[42|42|42|42|42|42|42|42|42|42|42|42|03|42|42|42]

The algorithm which uses basic vector operations:

broadcast the 0th byte of register into a new vector:

input     = [42|42|42|42|42|42|42|42|42|42|42|42|03|42|42|42]
broadcast = [42|42|42|42|42|42|42|42|42|42|42|42|42|42|42|42]

perform a vector-wide compare for equality:

cmp       = (input == broadcast)
          = [ff|ff|ff|ff|ff|ff|ff|ff|ff|ff|ff|ff|00|ff|ff|ff]

check whether all elements of cmp vector are "true".

Depending on a SIMD flavour, these simple steps may not be that simple.

Autovectorization status in GCC & Clang in 2021

Mon, 18 Jan 2021 12:00:00 +0100

Introduction

Almost two years ago I did an in-depth comparison of autovectorization abilities of popular compilers: GCC, clang, ICC and MSVC. In this text only GCC and clang are considered, as I don't see any new versions of ICC nor MSVC on godbolt.org (drop me a line if I got lost in the multitude of compiler versions). Update 2021-02-17: MSVC 19.28 status.

The question is: "what has changed between GCC 9 and GCC 10, and between clang 9 and clang 11?".

Use AVX512 to calculate binomial coefficient

Sat, 21 Mar 2020 12:00:00 +0100

Introduction

The value of binomial coefficient kovern can be expressed as n!/(k! ⋅ (n − k)!). This can be simplified to [(n − p) ⋅ (n − p + 1) ⋅ … ⋅ (n)]/p!, where p = max(k, n − k). Daniel Lemire showed in article Fast divisionless computation of binomial coefficients how efficiently evaluate the latter expression.

Can SIMD instructions be utilized to get binomial coefficients? I wish I could write "yes, they can", but the answer is not optimistic. SIMD instructions can be used to perform in parallel several pairs of multiplications, however it's a quest to properly setup registers and deal with different numbers of arguments that depend on the n and k. That's the first option, which I didn't check.

What I checked and described here is utilization of AVX512 with a different numeric system. An important fact is that calculation of binomial coefficients involves only multiplication and (integer) division.

We do know that all natural numbers can be factorized, i.e. represent as a product of prime numbers raised to some non-negative powers. For instance, 20200320 is equal to 2⁷ ⋅ 3³ ⋅ 5 ⋅ 7 ⋅ 167, similarly 7812 is 2² ⋅ 3² ⋅ 7 ⋅ 31. Now a product of two factored numbers can be calculated by adding the exponents of corresponding primes. For example 20200320 ⋅ 7812 = 2^{(7 + 2)} ⋅ 3^{(2 + 3)} ⋅ 5 ⋅ 7^{(1 + 1)} ⋅ 31 ⋅ 167 = 2⁹ ⋅ 3⁵ ⋅ 5 ⋅ 6² ⋅ 31 ⋅ 167. Likewise, division requires subtraction of exponents.

The core idea is to represent input integers as factored numbers. We setup the fixed number of primes and then operate only on exponents. We make sure that exponents fit in 8-bit values, thus in case of AVX512 we have 64-element products. With such representation, the vector addition and subtraction instructions are sufficient to calculate binomial coefficients.

So far everything might sounds nice, but unfortunately there are some serious problems:

Factorization is not cheap, we must cache exponents. Since we must cache intermediate values, why not pre-calculate binomial coefficients?
Getting back from factorized representation is also not cheap. It requires multiplication and getting integer powers (also multiplication).
To my surprise, the range of inputs covered by a SIMD-ized algorithm is smaller than a scalar version. I supposed that cancellation of primes present in both nominator and denominator would be beneficial.

Use AVX512 Galois field affine transformation for bit shuffling

Sun, 19 Jan 2020 12:00:00 +0100

Introduction

This article was inspired by Geoff Langdale's text Why Ice Lake is Important (a bit-basher’s perspective). I'm also grateful Zach Wegner for an inspiring discussion.

The AVX512 extension GFNI adds three instructions related to Galois field:

VGF2P8MULB (_mm512_gf2p8mul_epi8) — multiply 8-bit integers in the field GF(2⁸);
VGF2P8AFFINEINVQB (_mm512_gf2p8affineinv_epi64_epi8) — inverse affine transformation in the field GF(2⁸);
VGF2P8AFFINEQB (_mm512_gf2p8affine_epi64_epi8) — affine transformation in the field GF(2⁸).

While the two first instructions perform quite specific algorithms, the third one is the most generic and promising.

AVX512 8-bit positional population count procedure

Tue, 31 Dec 2019 12:00:00 +0100

Introduction

Positional population count (pospopcnt) is a procedure that calculates the histogram for bits placed at given position in a byte, word or double word etc. from larger stream of such entities.

This is a very naive implementation of 8-bit pospopcnt:

void pospopcnt(const uint8_t* data, size_t n, uint64_t histogram[8]) {
    for (size_t i=0; i < n; i++) {
        const uint8_t byte = data[i];
        for (int bit=0; bit < 8; bit++)
            histogram[bit] += (byte >> bit) & 0x01;
    }
}

For example [3, 3, 2, 4, 1, 2, 3, 1] is the pospopcnt result for following five bytes:

0b0110'1001,
0b1100'1000,
0b0000'1111,
0b0001'0011,
0b0110'1110.

A few 16-bit pospopcnt procedures were described in article Efficient Computation of Positional Population Counts Using SIMD Instructions by Marcus D. R. Klarqvist, Daniel Lemire and me. The library maintained by Marcus provides pospopcnt procedures for 8, 16 and 32-bit words.

This article shows a neat utilization of SAD instruction to calculate 8-bit pospopcnt. It's not the fastest one, but I really like the whole algorithm.

SIMDization of switch statements

Sun, 03 Feb 2019 12:00:00 +0100

Introduction

There are two main purposes of a switch statement:

Express simple function that translate from one set of values into another, like getting a string representation of enum values.
Dispatch different code sequences based on switch argument, as an alternative to "if-ladder".

Compilers usually transform switch statements using following approaches:

Binary search on constant keys: a compiler emits series of comparisons and jumps interleaved with case code.
When the key values span a small range (even non-continuous one), the values are used to index a lookup table of jump targets.

Of course, a compiler might optimize some specific cases, for some neat examples look at tree-switch-conversion.cc from GCC.

However, switch statements can be expressed also with SIMD instructions. Vector instructions are used used to translate from the argument value into case ordinal number. Then, the index is used either to 1) fetch a value from a precalculated table, or 2) get a jump target (the address) which is used to dispatch code fragments.

Example of binary search code

Following C++ code is compiled by GCC 7.3.0 into a binary search procedure.

enum class Colour : uint32_t {
    RED     = 0x00ff0000,
    GREEN   = 0x0000ff00,
    BLUE    = 0x000000ff,
    WHITE   = 0x00ffffff,
    GRAY0   = 0x00333333,
    GRAY1   = 0x00aaaaaa,
    GRAY2   = 0x00dddddd,
    BLACK   = 0x00000000
}

int palette(Colour col) {
    switch (col) {
        case Colour::RED:
            return 0;
        case Colour::GREEN:
            return 1;
        case Colour::BLUE:
            return 2;
        case Colour::WHITE:
            return 3;
        case Colour::GRAY0:
            return 4;
        case Colour::GRAY1:
            return 5;
        case Colour::GRAY2:
            return 6;
        case Colour::BLACK:
            return 7;
        default:
            return -1;
    }
}

_Z7palette6Colour:
    cmpl    $3355443, %edi
    je      .L3
    jbe     .L24
    movl    $6, %eax
    cmpl    $14540253, %edi
    je      .L21
    jbe     .L25
    xorl    %eax, %eax
    cmpl    $16711680, %edi
    je      .L21
    movl    $3, %eax
    cmpl    $16777215, %edi
    jne     .L2
.L21:
    ret
.L24:
    movl    $2, %eax
    cmpl    $255, %edi
    je      .L21
    movl    $1, %eax
    cmpl    $65280, %edi
    je      .L21
    movl    $7, %eax
    testl   %edi, %edi
    je      .L26
.L2:
    movl    $-1, %eax
    ret
.L25:
    movl    $5, %eax
    cmpl    $11184810, %edi
    jne     .L2
    ret
.L3:
    movl    $4, %eax
    ret
.L26:
    ret

Example of jump lookup

#include <cstdio>

int code_block(int x) {

    int result = -1;

    switch (x) {
        case 0:
            puts("zero");
            break;

        case 3:
            puts("one");
            result = 42;
            break;

        case 4:
            puts("two");
            result = 42;
            break;

        case 7:
            puts("three");
            result = 1024;
            break;

        case 8:
            puts("four");
            result = 42;
            break;

        case 11:
            puts("five");
            result = 1024;
            break;
    }

    return result;
}

.LC0:
    .string "zero"
.LC1:
    .string "one"
.LC2:
    .string "two"
.LC3:
    .string "three"
.LC4:
    .string "four"
.LC5:
    .string "five"
    .text
_Z10code_blocki:
    cmpl    $11, %edi
    ja      .L13
    leaq    .L4(%rip), %rdx
    movl    %edi, %edi
    subq    $8, %rsp
    movslq  (%rdx,%rdi,4), %rax
    addq    %rdx, %rax
    jmp     *%rax
    .section        .rodata
.L4:
    .long   .L3-.L4
    .long   .L10-.L4
    .long   .L10-.L4
    .long   .L5-.L4
    .long   .L6-.L4
    .long   .L10-.L4
    .long   .L10-.L4
    .long   .L7-.L4
    .long   .L8-.L4
    .long   .L10-.L4
    .long   .L10-.L4
    .long   .L9-.L4
    .text
.L9:
    leaq    .LC5(%rip), %rdi
    call    puts@PLT
    movl    $1024, %eax
.L11:
    addq    $8, %rsp
    ret
.L3:
    leaq    .LC0(%rip), %rdi
    call    puts@PLT
    movl    $-1, %eax
    addq    $8, %rsp
    ret
.L5:
    leaq    .LC1(%rip), %rdi
    call    puts@PLT
    movl    $42, %eax
    addq    $8, %rsp
    ret
.L6:
    leaq    .LC2(%rip), %rdi
    call    puts@PLT
    movl    $42, %eax
    addq    $8, %rsp
    ret
.L7:
    leaq    .LC3(%rip), %rdi
    call    puts@PLT
    movl    $1024, %eax
    addq    $8, %rsp
    ret
.L8:
    leaq    .LC4(%rip), %rdi
    call    puts@PLT
    movl    $42, %eax
    addq    $8, %rsp
    ret
.L10:
    movl    $-1, %eax
    jmp     .L11
.L13:
    movl    $-1, %eax
    ret

Malloc internal memory fragmentation footprint

Sun, 03 Feb 2019 12:00:00 +0100

Introduction

When we allocate memory using malloc or another interface, like operator new in C++, we get a pointer and promise that nobody else would acquire the same memory area. But underneath, more memory is needed. For instance the allocator has to keep the size of block to implement realloc.

More important is that the allocator unlikely allocate the exact number of bytes we requested, rather it will round the size up. Internal memory fragmentation is how we call this extra memory — which is allocated, but not legally available for a program.

This text shows internal memory fragmentation effects in different synthetic scenarios. However, memory fragmentation is a real problem. I came across this issue when was developing pyahcorasic module, that is built around multi-way trees — tries. To build a trie we need to allocate a large number of quite small fixed-size nodes associated with (also rather small) edge structures of variable size. It appeared that while theoretical size of all structures is smaller than memory I had available in my system, malloc reported no memory.

Auto-vectorization status in GCC, Clang, ICC and MSVC

Sat, 02 Feb 2019 12:00:00 +0100

Introduction

Update 2021-02-17: please check the newest status of GCC & Clang, and MSVC.

The term "auto-vectorization" means the ability of a compiler to transform given scalar algorithm into vectorized one, i.e. express dominating operation(s) using SIMD instruction.

I'm sure nobody would argue that auto-vectorization is as important as scalar optimizations performed by compilers. Now vectorization is a must.

From what I can gather, one of the first commonly used compiler of C/C++ which automatically vectorized code was Intel compiler; by the time, luckily for us, GCC and Clang caught up.

SIMDized counting byte in byte stream

Tue, 29 Jan 2019 12:00:00 +0100

Introduction

We want to count how many times given byte occurs in a byte stream; here is a C program doing this:

#include <stdint.h>
#include <stddef.h>

size_t countbyte(uint8_t* data, size_t size, uint8_t b) {
    size_t result = 0;
    for (size_t i=0; i < size; i++)
        result += (data[i] == b);

    return result;
}

GCC vectorization

The current GCC vectorization algorithm is able to handle the presented procedure, but its output is not optimal. For an AVX2 target GCC keeps four 64-bit sub-counters which are updated in every iteration and then added in the end.

In a single iteration 32 bytes are loaded and then compared with vector filled with given byte:

vpcmpeqb    (%rax), %ymm7, %ymm0

Result of this operation is vector of bytes filled with either ones (0xff) or zeros. Then 0xff are converted to ones by bit-and operation:

vpand       %ymm6, %ymm0, %ymm0 ; ymm6 = _mm256_set1_epi8(0x01)

The last step of algorithm is casting these 8-bit numbers to 64-bit value and updating the mentioned counters.

The conversion is done gradually: first from 8-bit numbers to 16-bit ones; then from 16-bit to 32-bit and finally from 32-bit to 64-bit values. This conversion must be done for each 4-byte subarray of input register. It means that following code is repeated eight times:

vpmovzxbw       %xmm0, %ymm1        ; 8-bit -> 16-bit numbers
vpmovzxwd       %xmm1, %ymm4        ; 16-bit -> 32-bit numbers
vpmovzxdq       %xmm4, %ymm2        ; 32-bit -> 64-bit numbers
vpaddq          %ymm4, %ymm2, %ymm2 ; update the sub-counters

Below is the full dissasembly from GCC 9 (snapshot from 2019-01-17, options: -O3 -march=cannonlake).

vpmovzxbw       %xmm0, %ymm1
vpmovzxwd       %xmm1, %ymm4
vpmovzxdq       %xmm4, %ymm2
vextracti128    $0x1, %ymm1, %xmm1
vextracti128    $0x1, %ymm4, %xmm4
vpmovzxwd       %xmm1, %ymm1
vpmovzxdq       %xmm4, %ymm4
vpaddq          %ymm4, %ymm2, %ymm2
vextracti128    $0x1, %ymm0, %xmm0
vpmovzxdq       %xmm1, %ymm4
vextracti128    $0x1, %ymm1, %xmm1
vpmovzxbw       %xmm0, %ymm0
vpmovzxdq       %xmm1, %ymm1
vpmovzxwd       %xmm0, %ymm3
vpaddq          %ymm1, %ymm4, %ymm1
vpaddq          %ymm1, %ymm2, %ymm1
vextracti128    $0x1, %ymm0, %xmm0
vpmovzxdq       %xmm3, %ymm2
vextracti128    $0x1, %ymm3, %xmm3
vpmovzxwd       %xmm0, %ymm0
vpmovzxdq       %xmm3, %ymm3
vpaddq          %ymm3, %ymm2, %ymm3
vpmovzxdq       %xmm0, %ymm2
vextracti128    $0x1, %ymm0, %xmm0
vpmovzxdq       %xmm0, %ymm0
vpaddq          %ymm3, %ymm1, %ymm1
vpaddq          %ymm0, %ymm2, %ymm0
vpaddq          %ymm0, %ymm1, %ymm0
vpaddq          %ymm0, %ymm5, %ymm5

std::function and overloaded functions

Wed, 23 Jan 2019 12:00:00 +0100

Overloaded functions

Let us consider this simple use case, where we want to invoke a function:

void invoke_callback(std::function<void(int, int)>);

Everything works fine when a callback is a lambda.

auto callback = [](int, int){};
invoke_callback(callback);

When we have overloaded functions, there are problems, as the compiler is not able to select a proper overload.

void overloaded_function(int, int);
void overloaded_function(int, const std::string&);

// ...

invoke_callback(overloaded_function);

The above code leads does not compile, error report from GCC is:

error: cannot resolve overloaded function ‘overloaded_function’
based on conversion to type ‘std::function<void(int, int)>’

To make this compilable we need to insert a weird casting to pointer to function. As far I know it's not possible to obtain from std::fuction any member type for this, so retyping the whole function type is required.

invoke_callback(static_cast<void(*)(int, int)>(&overloaded_function));
//                           ^^^           ^^^

This is pretty verbose.

I ended up with bare pointers to function in the signature of invoke_callback.

pyahocorasick stabilisation story

Tue, 08 Jan 2019 12:00:00 +0100

Introduction

pyahocorasick is a python module I started in 2011. That time I was interested in stringology and the Aho-Corasick algorithm appeared to be quite challenging. It was a sufficient reason to program it. However, I also decided that the result shouldn't be another proof-of-concept, that nobody --- except me — would use. Since I like Python, I chose form of a C extension, which nicely combines a friendly Python API with an efficient C implementation.

Moving fast forward, the module gained a few users worldwide. Maybe this is not the most popular package on pypi, but people keep installing it. Many users contributed to the code, documentation and infrastructure, or reported bugs and helped with debugging. Philippe Ombredanne helped a lot with different aspects of development — without him the project wouldn't be so great.

This text is a result of recent work on stabilisation the module, that was propelled by fixing a long-standing bug. The bug was driving me crazy for more than a year. I want to show what means were used to eliminate this and many other bugs. And also how the code quality was improved as a side effect. I hope some of you find an inspiration or solution.

The bug

Before we start I have to describe the bug, nobody should repeat my stupid mistake.

The bug was caused by misuse of python function PyBuild_Value, which is used by a pickling mechanism. Basically, pickling used to be done as a simple memory dump — the module created single memory area filled with some binary data.

The invocation Py_BuildValue("y#", ptr, size) constructs a bytes object with a copy of memory pointed by ptr, having given size. The problem is that such a format string gets size of type int. I wrongly assumed that on 64-bit machines int is a 64-bit number. It's not true, int has only 32 bits. Because of that, when size of the memory area was larger than 2GB, strange things happen, as shown in the table below.

64-bit size	outcome
range 0 to `0x7fffffff` (up to 2GB)	no errors
from `0x80000000` to `0xffffffff`	`int` is negative, empty buffer created but no error is reported
anything larger than 4GB, but bit 32th equals zero	created buffer of `size & 0x7fffffff`
anything larger than 4GB, but bit 32th one	`int` is negative, empty buffer created but no error is reported

The solution was pretty simple: a huge memory area is split into several smaller regions and the list of such regions is pickled. The size of single region is limited to a few megabytes, it will never be close to the 2GB boundary (although all data still can be larger than 2GB).

C++ --- how to read a file into a string

Mon, 07 Jan 2019 12:00:00 +0100

Introduction

To my surprise I quite often need to read the whole contents of a file into a string. Sometimes it's easier to generate data with an external program, sometimes unittests require to read generated file, etc.

A signature of such loader function is:

std::string load_file(const std::string& path);

In C++ an official way to deal with files are streams. There are at least two methods to load data into a string:

use streambuf iterators to construct a string,
use an auxilary string stream to handle a streambuf object.

std::string load1(const std::string& path) {
    std::ifstream file(path);
    return std::string((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
}


std::string load2(const std::string& path) {
    auto ss = std::ostringstream{};
    std::ifstream file(path);
    ss << file.rdbuf();
    return ss.str();
}

Both functions do their jobs, but reportedly are slow. While C++ still exposes good old C API, i.e. fread (libc) or read (POSIX), I compared performance of all solutions. Although the C solution using fread — which is shown below — is much longer than the C++ counterparts, its performance is significantly better than anything based on C++ streams. Performance of read is almost identical to fread, differences are negligible.

Of course, the performance boost highly depends on a machine type, hard drive, etc., but clearly the overhead of C++ streams is really huge compared to libc and POSIX calls.

Implementation using fread:

std::string load3(const std::string& path) {

    auto close_file = [](FILE* f){fclose(f);};

    auto holder = std::unique_ptr<FILE, decltype(close_file)>(fopen(path.c_str(), "rb"), close_file);
    if (!holder)
      return "";

    FILE* f = holder.get();

    // in C++17 following lines can be folded into std::filesystem::file_size invocation
    if (fseek(f, 0, SEEK_END) < 0)
      return "";

    const long size = ftell(f);
    if (size < 0)
      return "";

    if (fseek(f, 0, SEEK_SET) < 0)
        return "";

    std::string res;
    res.resize(size);

    // C++17 defines .data() which returns a non-const pointer
    fread(const_cast<char*>(res.data()), 1, size, f);

    return res;
}

Implementation using read:

std::string load4(const std::string& path) {

    int fd = open(path.c_str(), O_RDONLY);
    if (fd < 0)
        return "";

    struct stat sb;
    fstat(fd, &sb);

    std::string res;
    res.resize(sb.st_size);

    read(fd, const_cast<char*>(res.data()), sb.st_size);
    close(fd);

    return res;
}

AVX512VBMI --- remove spaces from text

Sat, 05 Jan 2019 12:00:00 +0100

Introduction

Removing spaces from a string is a common task in text processing. Instead of removing single character we often want to remove all the white space characters or the punctuation characters etc.

In this article I show an AVX512VBMI implementation. The algorithm is not faster than the scalar code for all cases. But for many it can be significantly faster, and what is more important, in tests on real-world texts it performs better.

Update 2019-01-13: this article pop up on twitter and Hacker News where provoked an incredibly fruitful discussion. Travis Downs noticed that branch mispredictions can be compensated by unrolling the loop in the initial algorithm. Zach Wegner came up with an algorithm which works in constant time by using PEXT instruction. Michael Howard shared with his scalar and AVX2 variants of "despacing" procedure. I'd like to thank all people discussed this topic both on HN and twitter.

Python --- file modification time perils

Sat, 24 Nov 2018 12:00:00 +0100

I need to copy a file to another directory whenever it got changed. The easiest way to do this is to check the modification time of file, a number of seconds since epoch:

import os
def get_mtime(path)
    return os.stat(path).st_mtime

It's not the most reliable way, but in my case it was good enough. Up to the time when I noticed that sometimes files didn't get updated.

I figured out that for a reason get_mtime returned an integer value, while the file system was able to deal with higher resolution than a second; system command stat printed microseconds.

The culprit was the setting of os module. It is possible to select in runtime, by os.stat_float_times(boolean), whether the module reports times as integers or floats. For an unknown reason my python installation defaulted to integers. Thus it was possible to have two files with different modification times having the same integer parts.

Finally, I forced float times everywhere (os.stat_float_times(True)).

SIMDized sum of all bytes in the array --- part 2: signed bytes

Sun, 18 Nov 2018 12:00:00 +0100

Introduction

This is the second part of SIMDized sum of all bytes in the array. The first part describes summing unsigned bytes, here we're going to experiment with summing of signed bytes.

The baseline C implementation is:

int32_t sumbytes(int8_t* array, size_t size) {

    int32_t result = 0;

    for (size_t i=0; i < size; i++)
        result += int32_t(array[i]);

    return result;
}

And the C++ implementation:

#include <numeric>

int32_t sumbytes(int8_t* array, size_t size) {
    return std::accumulate(array, array + size, int32_t(0));
}

Algorithm used by GCC

Below is the assembly code of the main loop compiled for Skylake by GCC 7.3.0 with flags -O3 -march=skylake:

vpmovsxbw       %xmm1, %ymm2
vextracti128    $0x1, %ymm1, %xmm1
vpmovsxwd       %xmm2, %ymm3
vextracti128    $0x1, %ymm2, %xmm2
vpmovsxbw       %xmm1, %ymm1
vpaddd          %ymm0, %ymm3, %ymm3
vpmovsxwd       %xmm2, %ymm0
vpaddd          %ymm3, %ymm0, %ymm2
vpmovsxwd       %xmm1, %ymm0
vextracti128    $0x1, %ymm1, %xmm1
vpaddd          %ymm2, %ymm0, %ymm0
vpmovsxwd       %xmm1, %ymm1
vpaddd          %ymm0, %ymm1, %ymm0

The approach used here by GCC is exactly the same as for summing unsigned bytes. There are multiple 32-bit sub-accumulators in single register, i.e. eight in case of AVX2 (four in SSE code), which are added together in the end, forming the scalar result.

To get 32-bit values there's two-step casting from int8_t to int32_t:

First extend a vector of int8_t into two vectors of int16_t numbers (VPMOVSXBW).
Then, get four vectors of int32_t from the vectors obtained in the previous step (VPMOVSXWD).

The cast instruction VPMOVSX extends the lower part of a register, in this case the lower half. This is the reason why extractions of helves (VEXTRACTI128) are needed.

How many uops are there?

Sun, 18 Nov 2018 12:00:00 +0100

The current Intel CPUs translate instructions into so called uops (micro-ops), which is a kind of internal ISA. For simple operations, like addition or bitops, translation is one-to-one, i.e. there's exactly one uop for given instruction. When an instruction gets a memory argument we usually will get two uops: one for load, another for actual operation; please note that most instructions has many forms, usually reg, reg and reg, mem.

I was curious how it looks in case of SIMD instructions. I used data from uops.info, and picked recent SkylakeX architecture; results are from IACA 3.0,

Observations:

90% of SIMD instructions are directly (or almost directly) translated into simple uops. It means they're likely supported by dedicated circuits.
AVX512 scatter, gather and conflict instructions seem not to be backed by hardware.
STNI is very dead.

uops	number of CPU instructions	%	CPU instructions
0	8	0.17	vgatherdps, vgatherdps, vgatherqps, vpgatherdd, vpgatherdd, vpgatherqq, vpscatterqd, vscatterqps
1	1752	36.17	too many, omitted
2	2616	54.00	too many, omitted
3	234	4.83	too many, omitted
4	140	2.89	too many, omitted
5	38	0.78	dpps, vdpps, vdpps, vgatherdpd, vgatherdpd, vgatherdpd, vgatherdpd, vgatherdpd, vgatherdps, vgatherdps, vgatherdps, vgatherqpd, vgatherqpd, vgatherqpd, vgatherqpd, vgatherqpd, vgatherqps, vgatherqps, vgatherqps, vgatherqps, vmovdqu8, vpgatherdd, vpgatherdd, vpgatherdd, vpgatherdq, vpgatherdq, vpgatherdq, vpgatherdq, vpgatherdq, vpgatherqd, vpgatherqd, vpgatherqd, vpgatherqd, vpgatherqd, vpgatherqq, vpgatherqq, vpgatherqq, vpgatherqq
7	4	0.08	vpscatterdq, vpscatterqq, vscatterdpd, vscatterqpd
8	10	0.21	pcmpestri, rex.w pcmpestri, rex.w vpcmpestri, vpcmpestri, vpconflictd, vpconflictd, vpscatterqd, vpscatterqd, vscatterqps, vscatterqps
9	8	0.17	pcmpestri, pcmpestrm, rex.w pcmpestri, rex.w pcmpestrm, rex.w vpcmpestri, rex.w vpcmpestrm, vpcmpestri, vpcmpestrm
10	4	0.08	pcmpestrm, rex.w pcmpestrm, rex.w vpcmpestrm, vpcmpestrm
11	6	0.12	vaeskeygenassist, vaeskeygenassist, vpscatterdq, vpscatterqq, vscatterdpd, vscatterqpd
12	2	0.04	vpscatterdd, vscatterdps
14	2	0.04	vpconflictd, vpconflictq
15	2	0.04	vpconflictq, vpconflictq
16	1	0.02	vzeroall
19	4	0.08	vpscatterdq, vpscatterqq, vscatterdpd, vscatterqpd
20	2	0.04	vpscatterdd, vscatterdps
21	2	0.04	vpconflictd, vpconflictq
22	4	0.08	vpconflictd, vpconflictd, vpconflictq, vpconflictq
35	1	0.02	vpconflictd
36	4	0.08	vpconflictd, vpconflictd, vpscatterdd, vscatterdps

Scripts used to collect the data are available.

A short report from code::dive 2018

Sun, 18 Nov 2018 12:00:00 +0100

Introduction

In November this year there was new edition of code::dive, an IT conference in Wrocław, Poland. Practically nothing has changed from the previous edition: it is still great. The place is perfect, it's in a huge cinema located in the city center. There were free snack and water, and amazing coffee at decent price (the only downside was huge queues). As always, there were a lot of interesting talks; a new thing were "lighting talks", run during lunch break.

This edition was a bit different, though. Although Nokia still sponsors the conference, the organizers asked participants to buy tickets (25 złotych, 5 euro) and then gave all income to Polish Association of the Blind; I totally love this approach. BTW, approx 38,000 złotych were collected.

Disclaimer: I'm Nokia employee right now, but am writing this text in my spare time. The employer was so kind that I went to the conference during working hours.

The talks I attended:

"The Hitchhiker's Guide to Faster Builds" (two parts) by Viktor Kirilov
"Clean code in Go" by Mateusz Dymiński
"GoLand Tips & Tricks" by Florin Pățan
"Taming dynamic memory - An introduction to custom allocators in C++" by Andreas Weis
"Python as C++’s limiting case" by Brandon Rhodes
"How to do practical Data Science? From real-world examples to recommendations" by Artur Suchwałko
"C/C++ vs Security!" by Gynvael Coldwind
"A trusted trip in the cloud — working with trusted hardware in practice" by Gabriela Limonta
"Augmented Reality - The State of Play" by Rafał Legiędź
"Why algebraic data types are important" by Bartosz Milewski

Speeding up multiple vector operations using SIMD

Wed, 14 Nov 2018 12:00:00 +0100

Introduction

One step of k-means algorithm is calculating the distance between all centroids and all samples. Then centroids are recalculated and samples re-assigned. Centroids and also samples are vectors of fixed size.

I was curious how SIMD might help in this task (or similar ones).

SIMD --- why you shouldn't use static vector constants

Sun, 28 Oct 2018 12:00:00 +0100

Introduction

When work with SSE/AVX2/AVX512 it's virtually impossible not to use some vector constants, which are defined by _mm_set_epi32 or similar intrinsic functions.

If your program is written in C++ NEVER EVER use static const for such constants. Why? From what I can gather, a compiler treats vector types not as PODs (Plain-Old-Data), but as fully-featured classes that have to be constructed and destructed by some additional code.

I checked this on GCC 7.3.0 from Debian, and then confirmed on GCC 8.2.0 and Clang 7.0.0 on godbolt.org.

SIMDized sum of all bytes in the array

Wed, 24 Oct 2018 12:00:00 +0100

Introduction

I was curious how GCC vectorizes function that sums bytes from an array. Below is a loop-based implementation.

uint32_t sumbytes(uint8_t* array, size_t size) {

    uint32_t result = 0;

    for (size_t i=0; i < size; i++)
        result += uint32_t(array[i]);

    return result;
}

The same algorithm can be expressed with following C++ code.

#include <numeric>

uint32_t sumbytes(uint8_t* array, size_t size) {
    return std::accumulate(array, array + size, uint32_t(0));
}

When I saw the assembly generated by GCC I was sure that it's possible to make it better and faster. This text summarizes my findings.

I focus solely on Skylake performance and AVX2 code. The sources have got also implementations of SSE procedures and experiments include timings from an older CPU.

Algorithm used by GCC

Below is the assembly code of the main loop compiled for Skylake by GCC 7.3.0 with flags -O3 -march=skylake:

vpmovzxbw    %xmm0, %ymm2
vextracti128 $0x1, %ymm0, %xmm0
vpmovzxwd    %xmm2, %ymm1
vextracti128 $0x1, %ymm2, %xmm2
vpmovzxbw    %xmm0, %ymm0
vpmovzxwd    %xmm2, %ymm2
vpaddd       %ymm2, %ymm1, %ymm1
vpmovzxwd    %xmm0, %ymm2
vextracti128 $0x1, %ymm0, %xmm0
vpaddd       %ymm2, %ymm1, %ymm1
vpmovzxwd    %xmm0, %ymm0
vpaddd       %ymm0, %ymm1, %ymm0
vpaddd       %ymm0, %ymm3, %ymm3

GCC nicely vectorized the algorithm: it keeps multiple 32-bit sub-accumulators in single register, i.e. eight in case of AVX2 (four in SSE code). These 32-bit numbers are added together in the end, forming the scalar result.

Now, let's look how the type casting is done. Although AVX2 has variant of instruction VPMOVZXBD that converts directly from uint8_t to uint32_t (intrinsic _mm256_cvtepu8_epi32) the compiler does the conversion in two steps:

First, it extends a vector of uint8_t into two vectors of uint16_t numbers (VPMOVZXBW).
Then, gets four vectors of uint32_t from the vectors obtained in the previous step (VPMOVZXWD).

The cast instruction VPMOVZX extends the lower part of a register, in this case the lower half. This is the reason why extractions of helves (VEXTRACTI128) are needed.

SIMDized check which bytes are in a set

Thu, 18 Oct 2018 12:00:00 +0100

Introduction

The problem is defined as follows: there's a stream of bytes and we want to get a byte-mask (or a bit-mask) that indicates which bytes are in the predefined set.

Thanks to SIMD instructions this task can be performed faster than scalar code. Jobs like input validation or parsing (for instance CSV files), might benefit from a vectorized approach.

In this text I show several SIMD methods:

The universal algorithm that can handle arbitrary sets (from 1 to 255 elements) with a few instructions.
Several specialized algorithms, that handle small sets of peculiar properties. They require fewer instructions than the universal algorithm. However, the algorithms are rather meant for compilers/code generators, where we can statically determine the best code sequence for a predefined set.
For sake of completeness I describe basic SIMD methods. If the set has a few elements then no fancy algorithm is needed. Likewise, if the set can be represented as a union of ranges the code is also not complicated.

Finding index of the minimum value using SIMD instructions

Wed, 03 Oct 2018 12:00:00 +0100

Introduction

The goal is to find the first index of the minimum value in a non-empty sequence.

Following C code shows the idea.

size_t min_index_scalar(int32_t* array, size_t size) {

    assert(array != nullptr);
    assert(size > 0);

    size_t minindex = 0;
    int32_t minvalue = array[0];

    for (size_t i=1; i < size; i++) {
        if (array[i] < minvalue) {
            minvalue = array[i];
            minindex = i;
        }
    }

    return minindex;
}

The C++ standard library allows to express the same algorithm in one line.

#include <algorithm> // for std::min_element
#include <iterator>  // for std::distance

template <typename T>
size_t min_index(const T& v) {

    assert(!v.empty());

    return std::distance(v.begin(), std::min_element(v.begin(), v.end()));
}

The current versions of compilers (GCC 8.2, clang 7.0.0) are not able to autovectorize the code. However, they do autovectorize finding the minimum value, i.e. statement like return *std::min_element(v.begin(), v.end()).

AVX512 mask registers support in compilers

Fri, 18 May 2018 12:00:00 +0100

Introduction

AVX-512 introduced the set of 64-bit mask registers, called in assembler k0 ... k7. A mask can be used to:

Conditionally update elements in a destination register; it's an incredibly powerful feature, as virtually all vector instructions support it.
Hold the result of vector comparison.

The latter is also useful, as there is instruction ktest that updates the flags register, EFLAGS. Prior to AVX512 an extra instruction — like pmovmskb or ptest (SSE 4.1) — has to be used in order to alter control flow based on vectors content.

There are four variants of ktest kx, ky that operates on 8, 16, 32 or 64 bits of mask registers, but basically they perform the same operation:

ZF := (kx AND ky) == 0
CF := (kx AND NOT ky) == 0

2018-05-22 update: unfortunately the instruction is not available in AVX512F; 8- and 16-bit variants are available in AVX512DQ, 32- and 64-bit in AVX512BW.

AVX512 implementation of JPEG zigzag transformation

Sun, 13 May 2018 12:00:00 +0100

Introduction

One of steps in JPEG compression is zigzag transformation which linearises pixels from 8x8 block into 64-byte array. It's possible to vectorize this transformation. This short text shows SSE implementation, then its translation into AVX512BW instruction set, and finally AVX512VBMI code.

The order of items in a block after transformation is shown below:

[  0  1  5  6 14 15 27 28 ]
[  2  4  7 13 16 26 29 42 ]
[  3  8 12 17 25 30 41 43 ]
[  9 11 18 24 31 40 44 53 ]
[ 10 19 23 32 39 45 52 54 ]
[ 20 22 33 38 46 51 55 60 ]
[ 21 34 37 47 50 56 59 61 ]
[ 35 36 48 49 57 58 62 63 ]

Be careful with directory_iterator

Sat, 28 Apr 2018 12:00:00 +0100

C++17 finally introduced filesystem library, which is pretty nice.

This text shows a caveat my colleague bumped into recently. He wanted to perform a set of different operations on files from a directory; it was something like:

#include <filesystem>

using fs = std::filesystem;

class Foo {
public:
    void perfrom(const fs::path& dir) {
        perform_impl(fs::directory_iterator{dir}, fs::directory_iterator{});
    }

private:
    void perform_impl(fs::directory_iterator first, fs::directory_iterator end) {

        for (fs::directory_iterator it = first; it != end; ++it)
            do_foo(*it);

        for (fs::directory_iterator it = first; it != end; ++it)
            do_bar(*it);
    }

    void do_foo(const fs::directory_entry& de);
    void do_bar(const fs::directory_entry& de);
};

In method perform_impl we iterate twice over the range defined by two directory_iterators. Well, we suppose so. Although iterator first is copied, the copy operation is peculiar. It doesn't copy the state of iterator, we get merely a new "handle" to an existing, single state. Standard libraries from GCC and Clang keep a std::shared_ptr, which holds an instance of internal class responsible for iterating.

What it means? When the first loop executes, then first == end. Thus, the second loop never runs.

In my opinion this behaviour is counter-intuitive. If the copy operator doesn't really make a copy, it should be disabled in API (it can be done with = delete put next to the operator declaration). People will be forced to pass the iterator by reference and, thanks to that, will be aware of the iterator traits.

A funny side-effect of the current language feature is that even iterators passed by const reference change their visible state.

Parsing series of integers with SIMD

Thu, 19 Apr 2018 12:00:00 +0100

1 Introduction

While conversion from a string into an integer value is feasible with SIMD instructions, this application is unpractical. For typical cases, when a single value is parsed, scalar procedures — like the standard atoi or strtol — are faster than any fancy SSE code.

However, SIMD procedures can be really fast and convert in parallel several numbers. There is only one "but": the input data has to be regular and valid, i.e. the input string must contain only ASCII digits. Recently, I updated article about SSE parsing with the benchmark results. The speed-ups are really impressive, for example the SSSE3 parser is 7 to 9 times faster than a naive, scalar code.

The obvious question is how these powerful SIMD procedures can be used to convert real data? By real I mean possibly broken inputs that contain series of numbers of different length separated with characters from a predefined set.

In this text I try to answer that question; major contributions of this article are:

Methods to efficiently parse and validate such strings using SSE instructions. There is a special variant that handles only unsigned numbers and also a fully featured variant for signed numbers.
Boosting the SSE procedure with AVX2 or AVX512 instructions when possible.
A way to combine some of SIMD techniques with scalar conversions.

The article starts with unsigned conversion, because it is easier than a signed one. The signed conversion shares the core idea, it just adds some extra steps.

The text is accompanied with BSD-licensed software, that includes fully functional implementations alongside the programs which validate and benchmark the procedures.

Contents

1 Introduction
2 Parser specification
3 SSE conversion capabilities
- 3.1 Range checking
- 3.2 Sample implementation
4 Parsing and conversions of unsigned numbers
- 4.1 Algorithm overview
- 4.2 Normalizing input
- 4.3 Precalculating data
  - 4.3.1 Example
- 4.4 SSE algorithm outline
- 4.5 Detecting invalid inputs
  - 4.5.1 Detecting digits
  - 4.5.2 Detecting characters from set
    - 4.5.2.1 SSE & AVX2
    - 4.5.2.2 SSE4.2
- 4.6 Caveats
5 Parsing and conversion of signed numbers
6 Processing larger inputs
7 Scalar hybrid
- 7.1 Example
8 Appendix A — conversion of three-digit numbers
9 Appendix B — conversion of two four-digit numbers
10 Reference scalar procedures
11 Experiments
12 Conclusions
13 Acknowledgements
14 See also
15 Source code

Accidental recursion

Sat, 14 Apr 2018 12:00:00 +0100

Another bug I bumped into. There is the enum:

enum class Color {
    red,
    green
};

And the associated ostream operator, which is merely a switch; all operators for enums look like this.

std::ostream& operator<<(std::ostream& s, const Color c) {
    switch (c) {
        case Color::red:
            s << "red";
            break;

        case Color::green:
            s << "green";
            break;

        default:
            s << "invalid " << c;
            break;
    }

    return s;
}

The author clearly wanted to handle all possible cases, including invalid enum values, that might appear due to memory corruption or simply as a result of invalid casting. Sadly, the handler would fail in such an erroneous case --- there is an infinite recursion in the default case.

The correct solution would be extract enumeration type and print it as a numeric value:

default:
    s << "unknown " << static_cast<std::underlying_type<Color>::type>(c);
    break;

This is not the nicest code in the world, but does the job well.

Is sorted using SIMD instructions

Wed, 11 Apr 2018 12:00:00 +0100

Introduction

Recently, I came across a function that checks whether an array is sorted, i.e. if there is no element which would be greater than its successor. Below is a sample implementation:

bool is_sorted(const int32_t* input, size_t n) {
    if (n < 2) {
        return true;
    }

    for (size_t i=0; i < n - 1; i++) {
        if (input[i] > input[i + 1])
            return false;
    }

    return true;
}

I was sure that such a trivial loop is autovectorized by all decent compilers. I checked this on Compiler Explorer and to my surprise none of compilers does it. This is the state for GCC 7.3 (and upcoming GCC 8.0), clang 6.0 and ICC 19.

This text explores possible vectorization schemas.

When lock does not lock --- C++ story

Mon, 26 Mar 2018 12:00:00 +0100

A few days ago I had compiled fresh GCC 8.0 from trunk and then compiled our product, just to see what we will have to fix in the future. And I found a nasty mistake.

Lets start with a class that perhaps every project has.

class Resource {
    int resource;
    std::shared_mutex mutex;
public:
    int get() {
        std::shared_lock<std::shared_mutex>(mutex);
        return resource;
    }
};

At first glance get() protects the shared resource; thanks to RTTI we don't have to care about locking and unlocking our mutex, as std::shared_lock internals care about this.

However, the code doesn't work that way. Indeed, the line

std::shared_lock<std::shared_mutex>(mutex);

declares std::shared_lock variable. But the name of variable is mutex and the lock is constructed with no associated mutex. Thus, the lock doesn't lock anything.

Of course, the correct declaration should be:

std::shared_lock<std::shared_mutex> lock(mutex);

Can we detected this kind of mistake? Yes, GCC has flag -Wshadow that warns when "a local variable or type declaration shadows another variable, parameter, type, or class member (in C++), or whenever a built-in function is shadowed":

file.cpp: warning: declaration of ‘mutex’ shadows a member of ‘Resource’ [-Wshadow]
     std::shared_lock<std::shared_mutex>(mutex);

However, I found the mistake thanks to more aggressive -Wparentheses flag in GCC 8:

file.cpp: warning: unnecessary parentheses in declaration of ‘mutex’ [-Wparentheses]
     std::shared_lock<std::shared_mutex>(mutex);

An awful part of C++17

Fri, 16 Mar 2018 12:00:00 +0100

The current state

I was really happy when saw that C++17 finally introduced standard functions to parse integers and floats. It is a group of functions std::from_chars defined in the header charconv. Unfortunately, it was a fleeting moment of happiness. The proposed API quickly appeared to be awful. Lets look how the integer parser is defined (the floating-point parsers are similar):

struct from_chars_result {
    const char* ptr;
    std::errc ec;
};

from_chars_result from_chars(const char* first, const char* last,
                             /*integer type*/& value, int base = 10);

The API resembles old good C, with one important exception: it's not good at all. How one is supposed to use it in C++?

#include <string>
#include <charconv> // from_chars

// ...

std::string input;

long int result;
auto ret = std::from_chars(&*input.begin(), &*input.end(), result);
if (ret.ec) {
   const auto error = std::make_error_code(ret.ec);
   std::cout << error.message();
}

Yes, to get a char pointer from a string iterator one need to write &*it. The alternative invocation is std::from_chars(input.c_str(), input.c_str() + input.size(), ...). Both are ugly, aren't they?

To make things funnier, from_chars recognizes the minus character, but not the plus character. Yes, it's not a misake — you can convert string "-42" but for "+42" you'll get an error.

Compare this with strtol, which was defined several decades ago:

#include <string>
#include <cstdio>
#include <cstdlib>

// ...

std::string input;
char* err;
long int result = strtol(input.data(), &err, 0);
if (errno) {
    fprintf(stderr, "%s\n", strerror(errno));
}

Intersection of ordered sets

Wed, 14 Mar 2018 12:00:00 +0100

Introduction

The intersection of two sets represented by sorted collections (like lists or arrays) can be done in linear time. If we label with k the size of one collection, and with n the size of another collection, then the complexity of intersection is O(n + k).

Below is a naive C++ implementation; the C++ standard library comes with std::set_intersection.

template <typename INSERTER>
void custom_set_intersection(const vec& A, const vec& B, INSERTER output) {

    size_t i = 0;
    size_t j = 0;

    while (i < A.size() && j < B.size()) {
        if (A[i] < B[j]) {
            i += 1;
        } else if (B[j] < A[i]) {
            j += 1;
        } else {
            // A[i] == B[j]
            *output++ = A[i];

            i += 1;
            j += 1;
        }
    }
}

If there is a huge difference between sizes of collections, then we might use two different approaches described below. I use terms "smaller" (k items) and "larger" collection/set (n items).

SSE/AVX: absolute value of difference of unsigned integers

Sun, 11 Mar 2018 12:00:00 +0100

With signed numbers it is really easy. We can subtract two numbers (instruction psub) and then calculate abs directly, as all Intel's SIMD instruction sets support the abs operation.

To calculate the abs of difference of two unsigned numbers we can use the saturated arithmetic. The saturated subtract (instructions psubusX) is equivalent to max(a - b, 0); it means that whenever subtraction would yield a negative result, the final result is zero.

We need to calculate two saturated subtracts, one for a - b, another for b - a; then merge them with bitwise or -- it's safe, because one of the subtract results is zero.

Below is an SSE implementation; full source code is available.

__m128i abs_sub_epu8(const __m128i a, const __m128i b) {

    const __m128i ab = _mm_subs_epu8(a, b);
    const __m128i ba = _mm_subs_epu8(b, a);

    return _mm_or_si128(ab, ba);
}

Is power of two --- BMI1 version

Sun, 11 Mar 2018 12:00:00 +0100

To check if a number is a power of two, the instruction BLSR from BMI1 extension can be used. The instruction resets the least set bit of a number, i.e. calculates (x - 1) and x. A sample C procedure that use the bit-trick:

bool is_power_of_two(int x) {
    return (x != 0) && (x == ((x - 1) & x));
}

If a number has exactly one bit set then BLSR yields zero. However, when input of BLSR is zero, the instruction also yields zero. Fortunately, BLSR sets CPU flags in following way:

ZF is set when result is zero,
CF is set when input is zero (note that if CF is set, ZF is also set).

Thanks to that we can properly handle all cases. Below is an assembly code:

blsr %eax, %eax

// result = (ZF == 1) and (CF == 0)
setz %al                // al = ZF
sbb  $0, %al            // al = ZF - CF
movzx %al, %eax         // cast

Sample program is available.

A short report from code::dive 2017

Sun, 26 Nov 2017 12:00:00 +0100

FPGA for a software developer

It was one of my favourite talks. Miodrag presented an architecture of typical FPGA, he also showed a number of open source tools that enables us to play with FPGAs. He also got through core parts of Verilog, using as an example a simple 8-bit CPU. The CPU has a fully functional design -- it has got an ALU, a control unit; it was able to decode variable-length opcodes and execute them. At the end the speaker run a sample program on the CPU realized on a real, cheap FPGA hardware. Wow!

Although I have already known some bits of FPGA and Verilog, the presentation gave me a grasp of the whole stack needed to use the programmable arrays in practice.

ARM Neon and Base64 encoding & decoding

Sat, 07 Jan 2017 12:00:00 +0100

Introduction

Base64 algorithms were subject of my interest last year and I proposed a few different SIMD approaches for both encoding and encoding. This text sums up my experiments with ARM Neon implementation. For algorithms' details please refer to Base64 encoding with SIMD instructions and Base64 decoding with SIMD instructions.

Contents

Introduction
ARM Neon
Base64 encoding
Base64 decoding
Performance results
- Encoding (quadwords)
- Decoding (doublewords)
See also
Source code

AVX512 --- first bit set in a large array

Thu, 22 Dec 2016 12:00:00 +0100

Problem

There is an array of 64-bit values, we want to find the first bit set; the array is sparse.

SWAR check if all chars are digits

Wed, 21 Dec 2016 12:00:00 +0100

Problem

We have a string and want to check if all its characters are ASCII digits.

It's a remnant of my experiments in number parsing: I was curious if separating validation from actual conversion would be profitable. The answer is no in a generic case. For really long numbers there might be some improvement, but in reality inputs are usually short.

Population count using XOP instructions

Fri, 16 Dec 2016 12:00:00 +0100

Introduction

AMD XOP defines instruction VPTERNB which does lookup in a pair of SSE registers. The instruction is similar to PSHUFB, but apart of wider, 5-bit index, it allows to perform several extra operations based on the higher 3-bits.

I showed that PSHUFB can be used to implement population count procedure. With 4-bit indices such procedure is straightforward. We split a byte vector into two halves, and invoke the instruction twice, getting popcount for both nibbles. Next these popcounts are added together, forming 8-bit counters, which are added in the end.

Similar procedure can be build around VPTERNB. However, to fully utilize 5-bit indices a slightly different strategy is needed. We process two vectors in one step treating byte pairs as 16-bit words. We call VPTERNB to calculate popcount of three 5-bit fields, one remaining bit of the 16-bit word is counted separately.

SIMD-friendly algorithms for substring searching

Mon, 28 Nov 2016 12:00:00 +0100

Introduction

Popular programming languages provide methods or functions which locate a substring in a given string. In C it is the function strstr, the C++ class std::string has the method find, Python's string has methods pos and index, and so on, so forth. All these APIs were designed for one-shot searches. During past decades several algorithms to solve this problem were designed, an excellent page by Christian Charras and Thierry Lecroq lists most of them (if not all). Basically these algorithms could be split into two major categories: (1) based on Deterministic Finite Automaton, like Knuth-Morris-Pratt, Boyer Moore, etc., and (2) based on a simple comparison, like the Karp-Rabin algorithm.

The main problem with these standard algorithms is a silent assumption that comparing a pair of characters, looking up in an extra table and conditions are cheap, while comparing two substrings is expansive.

But current desktop CPUs do not meet this assumption, in particular:

There is no difference in comparing one, two, four or 8 bytes on a 64-bit CPU. When a processor supports SIMD instructions, then comparing vectors (it means 16, 32 or even 64 bytes) is as cheap as comparing a single byte.
Thus comparing short sequences of chars can be faster than fancy algorithms which avoids such comparison.
Looking up in a table costs one memory fetch, so at least a L1 cache round (~3 cycles). Reading char-by-char also cost as much cycles.
Mispredicted jumps cost several cycles of penalty (~10-20 cycles).
There is a short chain of dependencies: read char, compare it, conditionally jump, which make hard to utilize out-of-order execution capabilities present in a CPU.

Contents

What does AVX512 conflict detection do?

Sun, 23 Oct 2016 12:00:00 +0100

AVX512CD

AVX512CD, or conflict detection, is a subset of AVX512 introducing following instructions:

broadcast bit-mask to byte-mask (vpbroadcastmw2d and vpbroadcastmb2q);
parallel lzcnt, i.e. counting leading zeros (vplzcntd and vplzcntq);
conflict detection (vpconflictd and vpconflictq).

The first two are not very interesting. Converting bit-mask into byte-mask saves in fact one move (vmovdqa32 with zero mask and a constant), it isn't too innovative. I can't find any real usage for lzcnt. Well, I wrote parallel popcount using this instruction (yes, 16 while-loops...), but it was just to show that I could.

In my opinion the most interesting are conflict detection instructions.

Detecting bit patterns with series of zeros followed by ones

Sun, 16 Oct 2016 12:00:00 +0100

Problem

We want to detect if a pattern is a sequence of zeros followed by ones. Table below lists all 8-bit patterns having this form.

bin	hex
1000_0000	80
1100_0000	c0
1110_0000	e0
1111_0000	f0
1111_1000	f8
1111_1100	fc
1111_1110	fe
1111_1111	ff

Byte-wise alignr in AVX512F

Sun, 16 Oct 2016 12:00:00 +0100

Introduction

The instruction alignr in Intel SIMD builds a new vector from a subrange of two concatenated vectors; its downside is accepting only compile-time constants. AVX512F lacks of byte-wise instructions, an available variant of alignr works at level of 32-bit words.

Byte-wise alignr is viable in AVX512F, using techniques used to handle so called long-numbers. We can do shifts at 32-bit word granulation using vpalignr (_mm512_alignr_epi32), then byte-wide shift inside each 32-bit word is possible. To perform the latter shift we need bytes from the next 32-bit word.

This force us to build two vectors, having current and next words at corresponding positions. Then these words are shifted accordingly and finally merged into one 32-bit word.

GNU std::string::find is very slow

Sat, 08 Oct 2016 12:00:00 +0100

Sorting an AVX512 register

Sat, 08 Oct 2016 12:00:00 +0100

Introduction

Presented method allows to sort a whole AVX512 register or its subrange, it is a variant of counting sort. The time complexity is linear, moreover method works entirely on registers, no extra memory operations are done. It may also be easily extended to sorting more than one register.

The method is suitable for sorting 32- and 64-bit integers, and also floating point numbers, both single and double precision.

AVX512F base64 coding and decoding

Sat, 17 Sep 2016 12:00:00 +0100

Introduction

Both base64 coding and decoding algorithms can be vectorized, i.e. SIMD instructions can be utilized, gaining significant speed-up over plain, scalar versions. I've shown different vectorization approaches in a series of articles:

AVX-512 is the recent extension to the Intel's ISA, unfortunately the extension is split into several subextensions. In the last article from the list I described usage of subextension AVX512BW (Byte-Word), at the moment of writing both articles AVX512BW was not available.

However, in 2016 on the market has appeared processors having subextension AVX512F (Foundation). Among many advantages of AVX512F there is one serious problem: lack of instructions working at byte and word level. The minimum vector element's size is 32 bits.

This article is a study of base64 algorithms realisation with foundation instructions AVX512F, major contributions are:

The new, binary search vectorized lookup for base64 encoding.
Evidence that SWAR techniques, even seem not optimistic at the first glance, are beneficial.
Use of a ternary logic instruction makes code simpler and faster.
Evaluation of gather instruction in the context of lookup-based methods.
Measurements from a real machine.

The text is split into four parts:

description of SWAR techniques required for algorithms;
details of base64 encoding;
details of base64 decoding;
experiment results and final remarks.

As I don't want to repeat myself too much, please refer to the linked articles for other algorithms for SSE and AVX2 and their analysis.

2016-12-18 note: in the initial version of this text I wrongly assumed order of input words, Alfred Klomp noted that the standard imposes a specific order. Today's change fixes this error.

Table of contents

SIMD bit mask

Wed, 14 Sep 2016 12:00:00 +0100

Problem

There is a SIMD register (128-, 256-, 512-bit width), we want to set all bits above the given position k; k is in range from 0 to the register's width.

Of course a lookup table could be used, but it's not a interesting (maybe a little.)

Building a bitmask

Wed, 14 Sep 2016 12:00:00 +0100

The problem

There is an array of 32-bit integers and a key — a specific value. The result have to be a bit vector with bits set on these position where the key is equal to array items. Pseudocode:

-- n - array size
for i = 0 .. n - 1 loop
    if key = array[i] then
        bitvector[i] = 1
    else
        bitvector[i] = 0
end for

A C++ interface:

void bitmask(const uint32_t* array, size_t n, const uint32_t key, uint8_t* bitvector);

Base64 encoding & decoding using AVX512BW instructions

Sun, 03 Apr 2016 12:00:00 +0100

Introduction

The SIMD versions of base64 conversion algorithms were described in Base64 encoding with SIMD instructions and Base64 decoding with SIMD instructions. I also described realization of both encoding and decoding using AVX512F (Foundation) instructions.

AVX512BW (Byte & Word) comes with a great number of new instructions; following instructions can help base64-related problems:

vpshufb (intrinsic _mm512_shuffle_epi8) — does a lookup in 128-bit lanes. For base64 algorithm it's sufficient;
vpermd (_mm512_permutexvar_epi32) — moves 32-bit words across the 128-bit lanes;
vpsllvw (_mm512_sllv_epi16) and vpsrlvw (_mm512_srlv_epi16) --- shifts individual 16-bit words by a variable amount, saved in a ZMM register.

The extension AVX512VBMI adds even more powerful instructions:

vpermb (_mm512_permutexvar_epi8) — does a lookup in a 64-byte table (a ZMM register). Unlike pshufb it doesn't destroy the lookup register;
vpermi2b (_mm512_permutex2var_epi8) — does a lookup in a 128-byte table formed by two ZMM registers.

The extension AVX512VL adds just one, but really nice instruction:

vpmultishiftqb (_mm512_multishift_epi64_epi8) — moves 8-bit subwords onto selected bytes.

2018-04-18: In the earlier versions of this text I wrongly assumed that instructions vpermb and vpermi2b are part of AVX512BW. Sorry for that.

Implementing byte-wise lookup table with PSHUFB

Sun, 13 Mar 2016 12:00:00 +0100

Introduction

In articles about base64 encoding and decoding I've showed how to implement SIMD version of a lookup table using basic vector instruction. This text describes another technique which employs my favourite instruction pshufb.

The task is defined as follows:

input range (0..255) is split into several subranges;
for each subrange a predefined value is assigned.

Now, for an input value a proper subrange is determined, and the value associated to the subrange is returned. Of course, everything is done in parallel.

Base64 decoding with SIMD instructions

Sun, 17 Jan 2016 12:00:00 +0100

Introduction

Surprisingly good results of base64 encoding with SIMD instructions forced me to check the opposite algorithm, i.e. the decoding. The decoding is slightly more complicated as it has to check the input's validity.

A decoder must also consider character '=' at the end of input, but since it's done once, I didn't bother with this in a sample code.

2016-12-18 note: in the initial version of this text I wrongly assumed order of input words, Alfred Klomp noted that the standard imposes a specific order. Today's change fixes this error.

Base64 encoding with SIMD instructions

Tue, 12 Jan 2016 12:00:00 +0100

Introduction

I had supposed that SIMD-ization of a base64 encoder is not worth to bother with, and I was wrong. When compared to scalar code, an SSE code is 2 times faster on Core i5 (Westmere), and around 2.5 times faster on Core i7 (Haswell & Skylake). An AVX2 code is 3.5 times faster on Core i7 (Skylake).

2016-12-18 note: in the initial version of this text I wrongly assumed order of input words, Alfred Klomp noted that the standard imposes a specific order. Today's change fixes this error.

Speeding up letter case conversion

Wed, 06 Jan 2016 12:00:00 +0100

Introduction

The aim of this text is to show how simple procedure which change case of letters could be rewritten to SWAR version gaining significant boost. In the article method "to lower case" is explored, however the opposite conversion is very easy to derive.

To be honest I have no idea if changing latter case is crucial task in any problem. My knowledge and experiences suggest that the answer is "no", but who knows.

Fast conversion of floating-point values to string

Tue, 29 Dec 2015 12:00:00 +0100

Introduction

The conversion of floating-point numbers to a string representation is not an easy task. Such procedure must deal with different special FP values, perform proper rounding and so on. The paper Printing Floating-Point Numbers Quickly and Accurately [PDF] by Robert G. Burger & R. Kent Dybvig describes a procedure which solves the problem correctly.

However, in some applications (mostly logging, debugging) rounding and accuracy are not as important as the speed. Sometimes we simply want to know if a number was 1000.5 or 0.5 and even if we read "0.499999" nothing wrong would happen.

Base64 encoding --- implementation study

Sun, 27 Dec 2015 12:00:00 +0100

Introduction

In a basic step of the Base64 encoding three bytes are processed producing four output bytes. The step consist following stages:

load 3 bytes;
split these 3 bytes to four 6-bit indices;
translate the indices using a lookup table;
save four bytes.

In this text different implementations of the procedure are examined.

Benefits from the obsession

Sun, 13 Dec 2015 12:00:00 +0100

Case 1

On the other day I saw such innocent code:

string decode_base64(string& base64) {

    // some validation

    string result;
    const size_t n = 3*base64.size() / 4;
    result.resize(n);

    // decoding stuff

    return result;
}

The size of result is three fouth of the base64's size, so it is obviously smaller than the maximum value of size_t which is used to store the string size. However, the overflow can occur during multiplying by 3. On 64-bit machines the size of base64 have to be really huge to trigger the error — it is 1.23e+19. But on 32-bit machines it's "merely" 28 GB (try to imagine base64-encoded movie sent via e-mail...) Despite the CPU architecture, the problem still exists. And the solution is not very complicated.

Expression 3/4 * x is equivalent to x - 1/4 * x. Dividing x by 4 never cause an overflow, but since we operate on integers these two expressions are not equal. The latter expression have to be corrected (rounded up) with following conditional expression (x % 4 != 0) ? 1 : 0. Thus the final expression is:

const size_t x = base64.size();
const size_t n = x - x/4 + ((x % 4 != 0) ? 1 : 0);

The expression requires: 2 additions, 1 right shift, 1 comparison and 1 condition expression. And it's perfectly safe.

Implicit conversion --- the enemy

Sat, 28 Nov 2015 12:00:00 +0100

I wrote:

result += string_utils::pad_left(string, '0');

I forget that pad_left signature is string, int, char and the char parameter has a default value. My mistake, without doubts.

This is another example of dark sides of the implicit conversions. C++ converts between characters and integers seamlessly. These two beast are distinct in the nature. Of course characters are represented by the numbers, however it's an implementation detail.

One can say: you made a mistake and now blame the language. No, I blame language's design. I'm afraid that we end up with something like Integer and int to overcome such problems.

Lesson learned: never use default parameters in public API (surprise!)

Another C++ nasty feature

Sun, 22 Nov 2015 12:00:00 +0100

I'm fond of C++ weirdness, really. This language is full of traps, and it shocks me once in a while.

Let's look at this piece of code, a part of a larger module:

void validate_date() {

    // ...

    boost::optional<unsigned> clock_hour;
    boost::optional<unsigned> am_pm_clock;

    // ... fill these fields

    if (some sanity check failed) {

        report_error("user has entered wrong time: %d %s",
            *clock_hour
            *am_pm_clock ? "AM" : "PM");
    }
}

We would expect that in case of an error following line will be reported: "user has entered wrong time: 123 PM". Obvious. But please look closer at the code, do you see any mistake? There is one... dirty... hard to notice. I'll give you a minute.

So, the mistake is lack of comma between expressions *clock_hour and *am_pm_clock. However, the code is valid! It compiles! And it took me a little longer than a minute to understand what happened. Explanation is:

*clock_hour evaluates to expression of type unsigned;
then a compiler sees * - a multiplication operator;
so the compiler checks if multiplication of unsigned (on the left side) with boost::optional<unsigned> (on the right side) is possible;
it is, because boost::optional<T> has conversion operator to type T.

We can rewrite the whole expression, now it should be clear:

((*clock_hour) * unsigned(am_pm_clock)) ? "AM" : "PM"

In result method is called with a single parameter of type const char*.

It's bizarre, it's terrible. A language should help a programmer. In my opinion implicit conversions is the worst feature of C++.

Short report from code::dive 2015

Sun, 15 Nov 2015 12:00:00 +0100

Writing Fast Code

Andrei performed a nice show, however some part were... hm, confusing. For example he claimed that code x = x/10 emits a division instruction. This is not true, all compilers run in so called "release mode" will emit multiplication by a reciprocal of a constant. Check this for your own. Another big misunderstanding of the speaker was the cause of slow writing operations on the modern hardware. He claimed that after a write request a CPU loads a line of cache, then modifies it contents according to the request, and finally CPU writes the cache line back to the memory. No. It doesn't work like this, simply. Slow down is caused mostly by multicore architecture and required synchronization among cache subsystems. But this is not very important, I think people should remember simple fact: fewer writes means faster programs.

Update 2015-11-22: my friend Piotr reminded me another funny fact, a speculation why division of floats is faster than division of integers. Andrei claimed that the reason is... exponential format of floats. It's just a subtraction of exponents, a division of mantissa and viola. I'm pretty sure that it's not the real reason.

Andrei showed how he has optimized procedure converting a number to an ASCII representation. He used few tricks, and one of them is worth to mention. He minimize the number of "real" conversions by introducing a specialized path for smaller values. Do the same in your program, analyze your data and use an if instruction to select a fast path. It usually works. Andrei gained 3-5 speedup without big effort.

From perspective of a programmer who has never worked on code optimization Andrei's advice were very valuable. For example: never measure time of debug compilation. Compare your program with good, standard & proved existing solutions. Your optimization of one module could have a negative impact on the whole application. When you measure a time, run tests many times and get the minimum measurement. Pretty obvious, but precious for newbies (the conference was full of students from local universities).

Boolean function for the rescue

Sun, 25 Oct 2015 12:00:00 +0100

The problem is defined as follows: a set of features is saved using bit-sets (usually large), and there is a list/map/whatever of sets containing features of different objects. We have to find which features are unique.

Naive solution is to use nested loops:

bit_set find_unique(list) {
    unsigned size = FEATURES_COUNT;
    bit_set  result(size);

    for (auto i=0; i < size; i++) {

        unsigned count = 0;
        for (auto& set: list) {
            if (set.is_set(i)) {
                count += 1;
            }
        }

        if (count == 1) {
            result.set(i);
        }
    }

    return result;
}

Not good, the method is_set of bit-set is called size * list.size() times. Even if a compiler is able to inline the call and use simple bit tests instructions it's still too expansive. Bit-set implementations always use arrays of integers to store the data, thanks to that bit-operations (and, or, xor, etc.) are executed very fast. We try to exploit this with boolean functions.

Each feature could be described as:

non-existing (count is 0),
unique (count is 1),
non-unique (count greater than 1).

Now these states are encoded using two bits:

H	L	value
0	0	non-existing
0	1	unique
1	1	non-unique

Then we define a transition table. For example if a feature is present and the current value is 'unique' then the next value is 'non-unique' (row 5th).

H	L	feature	H'	L'
0	0	0	0	0
0	1	0	0	1
1	1	0	1	1
0	0	1	0	1
0	1	1	1	1
1	1	1	1	1

Boolean expressions are: L' = L or feature; H' = H or (feature and L).

We also need to get single bit-set from H and L at the end:

H	L	result
0	0	0
0	1	1
1	1	0

The final boolean expression is: result = L and not H.

Now we can rewrite the code:

class unique_checker {

    unsigned  size;
    bit_set   L, H;

public:
    unique_checker(unsigned size)
        : size(size)
        , L(size)
        , H(size) {}

    void update(const bit_set& set) {

        assert(set.size() == size);

        H = H | (set & L);  // we suppose that bit_set overloads bit-operator
        L = L | set;
    }

    bit_set finalize() const {

        return L | ~H;
    }
};

bit_set find_unique2(list) {

    unique_checker checker(FEATURES_COUNT);

    for (auto& set: list) {
        checker.update(set);
    }

    return checker.finalize();
}

I really like this approach.

Tricky mistake

Mon, 25 May 2015 12:00:00 +0100

A programmer wrote:

class container;

class IndexOutOfBounds {
public:
    IndexOutOfBounds(const std::string& msg);
};

void container::remove(int index) {

    if (index < 0 || index >= size()) {
        throw new IndexOutOfBounds("Invalid index: " + index);
    }

    // the rest of method
}

Do you see the mistake? The programmer thought that the expression "Invalid index: " + index evaluates to std::string("Invalid index: 5").

In fact the type of the expression "Invalid index: " is char[15], so char[15] + integer results in — more or less — char*. For index in range [0, 15] an exception will carry the tail of the message, for example when index=10 then message assigned to the exception object will be "dex: ". For indexes larger than 15 and less than 0 a program likely crash.

This is why I hate C++, this language has many dark corner, stupid conventions, implicit conversion and not mention UB ("just" 150 UB, if you're curious).

Speeding up bit-parallel population count

Mon, 13 Apr 2015 12:00:00 +0100

Introduction

This well know method requires logarithmic number of steps in term of the word width. For example the algorithm run on a 64-bit word executes 6 steps:

const uint64_t t1 = fetch data

const uint64_t t2 = (t1 & 0x5555555555555555llu) + ((t1 >>  1) & 0x5555555555555555llu);
const uint64_t t3 = (t2 & 0x3333333333333333llu) + ((t2 >>  2) & 0x3333333333333333llu);
const uint64_t t4 = (t3 & 0x0f0f0f0f0f0f0f0fllu) + ((t3 >>  4) & 0x0f0f0f0f0f0f0f0fllu);
const uint64_t t5 = (t4 & 0x00ff00ff00ff00ffllu) + ((t4 >>  8) & 0x00ff00ff00ff00ffllu);
const uint64_t t6 = (t5 & 0x0000ffff0000ffffllu) + ((t5 >> 16) & 0x0000ffff0000ffffllu);
const uint64_t t7 = (t6 & 0x00000000ffffffffllu) + ((t6 >> 32) & 0x00000000ffffffffllu);

In each step k-bit fields are summed together; we start from 1-bit fields, then 2, 4, 8, 16 and finally 32 bits. Single step requires:

2 bit-ands;
shift right by constant amount;
addition.

SIMD-ized searching in unique constant dictionary

Wed, 08 Apr 2015 12:00:00 +0100

Introduction

The problem: there is an ordered dictionary containing only unique keys. The dictionary is read only, and keys are 32-bit (SSE) or 64-bit (AVX2).

The obvious solution is to use binary search. Keys can be stored in a contiguous memory thanks to that there is no internal fragmentation, and data has cache locality. And of course indexing the keys is done in constant time (in the terms of computational complexity) or a single memory fetch (hardware).

The time complexity of binary search is O(log₂(n)), i.e. for one million elements single lookup takes up to 20 operations. Single operation is fetching a value and comparing with the given key.

Another algorithm is linear search which seems to be suitable for small dictionaries. Linear search could be easily SIMD-ized.

SIMD: detecting a bit pattern

Sun, 22 Mar 2015 12:00:00 +0100

Introduction

The problem: there are 64-bit values with some data bits and some metadata bits; the metadata includes a k-bit field describing a "type" (k >= 0). The type field is located in a lower 32-bits.

Procedure processes two "types", one denoted with the code 3 and another with 5. When all items are of type 3 then we can use a fast AVX2 path. If there are some types 5, we have to call an additional function (a virtual method to be precise). Pseudocode:

for (size_t i = 0; i < size; i += 8) {

    // two AVX registers, or 8 elements, are processed in a single loop

    __m256i A_lo = {A64[i + 0], A64[i + 1], A64[i + 2], A64[i + 3]};
    __m256i A_hi = {A64[i + 4], A64[i + 5], A64[i + 6], A64[i + 7]};

    __m256i B_lo = {B64[i + 0], B64[i + 1], B64[i + 2], B64[i + 3]};
    __m256i B_hi = {B64[i + 4], B64[i + 5], B64[i + 6], B64[i + 7]};

    if (any element of A or B vector has type 5) { // ***

        // slow path
        for (int k = 0; k < 4; k++) {
            result[i + k]     = function(A_lo[i + k], B_lo[i + k]);
            result[i + k + 4] = function(A_hi[i + k], B_hi[i + k]);
        }
    } else {

        // fast path
        ...
    }

    // further processing
}

We have to fill condition of the if-statement.

Compiler warnings are your future errors

Sun, 22 Mar 2015 12:00:00 +0100

Lesson learned

Because we always have very large build logs I didn't notice the new warning.

In order to prevent such errors in the future I've written a script that extracts all warnings from the logs and prints them in a easy-to-read form. I also fight with warnings in so called spare time.

AVX512: ternary functions evaluation

Sun, 22 Mar 2015 12:00:00 +0100

Introduction

Intel's version of SIMD offers following 2-argument (binary) boolean functions: and, or, xor, and not. There isn't a single argument not, this function can be expressed with xor reg, ones, however it requires additional, pre-set register.

AVX512F will come with a very interesting instruction called vpternlog. There are two variants of the instruction operating on a packed 32-bit (vpternlogd) or a 64-bit vector (vpternlogq), however they do exactly the same thing — evaluate a 3-argument (ternary) boolean function on each bit of arguments, the function is given as a truth table.

The pattern of a truth table:

inputs			result
A	B	C	result
0	0	0	a
0	0	1	b
0	1	0	c
0	1	1	d
1	0	0	e
1	0	1	f
1	1	0	g
1	1	1	h

A programmer supplies only the result column, i.e. defines values of bits a through h, this is a single 8-bit value.

Depending on function complexity, a single vpternlog instruction can replace from one up to eight SIMD instructions.

According to Agner Fog's documentation on SkylakeX vpternlog has 1 cycle latency and 0,5 cycle reciprocal throughput (there are two execution units able to handle the instruction). It's pretty fast, though.

Ternary logic function is available as the intrinsic function _mm512_ternarylogic_epi32(a, b, c, imm8), where the argument a carries most significant bits, and c least significant bits.

SSE/AVX2: Generating mask where n leading (trailing) bytes are set

Sat, 21 Mar 2015 12:00:00 +0100

Introduction

Informal specification:

__m128i mask_lower(const unsigned n) {

    assert(n < 16);

    /*
        __m128i result = 0;
        for (unsigned int i=0; i < n; i++) {
            result.byte[i] = 0xff;
        }
    */

    switch (n) {
        case 0: return {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
        case 1: return {0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
        case 2: return {0xff, 0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
        // ...
        case 14: return {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00};
        case 15: return {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff};
    }
}

__m128i mask_higher(const unsigned n) {

    assert(n < 16);

    return ~mask_lower(15 - n);
}

Not everything in AVX2 is 256-bit

Sat, 21 Mar 2015 12:00:00 +0100

AVX2 has added support for 256-bit arguments for many operations on packed integers, although not all. Some instructions accept the 256-bit registers, but operates on 128-bit lanes rather the whole register.

There are three major groups of instructions:

packing (narrowing conversion),
unpacking (interleave),
and permutations

Below is a full list of instructions (with intrinsics):

valignr (_mm256_alignr_epi8)
vpslldq (_mm256_bslli_epi128)
vpsrldq (_mm256_bsrli_epi128)
vmpsadbw (_mm256_mpsadbw_epu8)
vpacksswb (_mm256_packs_epi16)
vpackssdw (_mm256_packs_epi32)
vpackuswb (_mm256_packus_epi16)
vpackusdw (_mm256_packus_epi32)
vperm2i128 (_mm256_permute2x128_si256)
vpermq (_mm256_permute4x64_epi64)
vpermpd (_mm256_permute4x64_pd)
vpshufd (_mm256_shuffle_epi32)
vpshufb (_mm256_shuffle_epi8)
vpshufhw (_mm256_shufflehi_epi16)
vpshuflw (_mm256_shufflelo_epi16)
vpslldq (_mm256_slli_si256)
vpsrldq (_mm256_srli_si256)
vpunpckhwd (_mm256_unpackhi_epi16)
vpunpckhdq (_mm256_unpackhi_epi32)
vpunpckhqdq (_mm256_unpackhi_epi64)
vpunpckhbw (_mm256_unpackhi_epi8)
vpunpcklwd (_mm256_unpacklo_epi16)
vpunpckldq (_mm256_unpacklo_epi32)
vpunpcklqdq (_mm256_unpacklo_epi64)
vpunpcklbw (_mm256_unpacklo_epi8)

For me the most surprising are packing instructions (vpack*) as they require additional shuffling (after or before the instruction) if we want to keep the order of values. In some cases the order is crucial.

Using SSE to convert from hexadecimal ASCII to number

Wed, 22 Oct 2014 12:00:00 +0100

Introduction

SSE procedure can convert 16- and 32-digits inputs producing 8- and 16-bytes results.

To get correct result's order of input, characters have to be reversed. In SSSE3 it can be done with pshufb, but in the earlier versions of SSE this is quite hard. When byte shuffling is not available then reversing can be done on the result word using bswap instructions.

Parsing decimal numbers --- part 2: SSE

Wed, 15 Oct 2014 12:00:00 +0100

PMADDWD details

This instruction is specialized and I guess isn't often used. PMADDWD operates on packed words and performs following algorithm:

temp = array of 8 signed dwords

-- 1. multiply 16-bit signed numbers vertically
src = packed_word(...)
dst = packed_word(...)

for i in 0 .. 7 loop

    -- temp is 32-bit signed number

    temp[i] := signed_multiplication(src[i], dst[i])

end loop

-- 2. add adjacent 32-bit words of temp and save result to src
--    now src has type packed_dword
for i in 0 .. 3 loop

    src[i] = temp[2 * i] + temp[2 * i + 1]

end loop

Parsing decimal numbers --- part 1: SWAR

Sun, 12 Oct 2014 12:00:00 +0100

Introduction

Naive parsing of unsigned decimal numbers can be coded as:

uint32_t parse(char* s) {

    uint32_t result = 0;

    while (*s) {
        const uint8_t digit = *s++;

        result = result * 10 + (digit - '0');
    }

    return result;
}

The procedure parse does not check validity of a string nor check for overflow — these problems won't be discussed in this text.

Processing single letter require just 3 operations:

subtract,
addition,
multiplication by 10.

On x86 multiplication by 10 is cheap; multiplication x * 10 is equivalent to x << 3 + x << 1. Shift by 8 is coded with LEA (it's executed by addressing unit, not ALU), and shift by 1 is simple addition or another LEA.

Using PEXT to convert from hexadecimal ASCII to number

Thu, 09 Oct 2014 12:00:00 +0100

Naive conversion from an ASCII digit to a value could be coded with a switch instruction or a lookup table — too simple, right? This is another note where I try to exploit PEXT — parallel extract, the new instruction introduced in extension BMI2 (Bit Manipulation Instructions). By the way I present nice branchless algorithm to convert an ASCII letter to a number.

Let see which codes are assigned to hexadecimal digits:

digit	code	value	letter	code	value
'0'	0x30	0	'a'	0x61	10
'1'	0x31	1	'b'	0x62	11
'2'	0x32	2	'c'	0x63	12
'3'	0x33	3	'd'	0x64	13
'4'	0x34	4	'e'	0x65	14
'5'	0x35	5	'f'	0x66	15
'6'	0x36	6	'A'	0x41	10
'7'	0x37	7	'B'	0x42	11
'8'	0x38	8	'C'	0x43	12
'9'	0x39	9	'D'	0x44	13
			'E'	0x45	14
			'F'	0x46	15

Observations:

for digits the value is equal to lower nibble;
for letter the value is equal to lower nibble plus 9;
both small and big letters has set 7-th bit (mask 0x40).

Branchless code converting single letter to numeric value:

uint8_t hex_letter2value(uint8_t chr) {

    const uint8_t letter = chr & 0x40;
    const uint8_t shift  = letter >> 3 | letter >> 6; // 9 if chr is letter, 0 otherwise

    // this sum is safe -- if shift = 9, then max value in lower
    // nibble is 6, and there won't be an overflow
    const uint8 adjusted = chr + shift;

    return adjusted & 0xf;
}

Following operations are performed:

2 bit-and,
2 shifts,
1 bit-or,
1 addition.

For a single letter it's too expansive, fortunately this algorithm could be easily translated to SIMD and SWAR code (I hope SIMD version appear soon).

32-bit SWAR version:

#define packed32(b) (uint8_t(b) * 0x01010101)

uint16_t four_letters_to_value(char* str) {

    const uint32_t input = bswap(*(uint32_t*)str); // bswap is required

    const uint32_t letter = input & packed32(0x40);
    const uint32_t shift  = letter >> 3 | letter >> 6;

    const uint32_t adjusted = input + shift;

    // for example:
    //     adjusted    = 0x0b0a0d07
    //     pext result = 0x000000bad7

    return pext(adjusted, 0x0f0f0f0f);
}

Much better — now conversion from 4 letters (or 8, when operate on 64-bit words) requires:

1 byte swap,
2 bit-and,
2 shifts,
1 bit-or,
1 addition,
1 pext.

Sample implementation is available at github.

Using PEXT to convert from binary ASCII to number

Mon, 06 Oct 2014 12:00:00 +0100

Suppose we have a string containing ASCII zeros and ones, for example "11100100", and we want to interpret this text as a binary number and get value (0xe4).

New instruction PEXT from BMI2 (Binary Manipulation Instructions) is perfect for this task. PEXT — parallel extract — forms a word from source bits selected by a mask, for example (32-bit arguments):

         MSB                               LSB
         ┌────────┬────────┬────────┬────────┐
src    = │00101010│11101101│00011011│11110000│
         └────────┴────────┴────────┴────────┘
         ┌────────┬────────┬────────┬────────┐
mask   = │10000011│10000001│11110000│00000111│
         └────────┴────────┴────────┴────────┘

         ┌────────┬────────┬────────┬────────┐
result = │00000000│00000000│00000101│10001000│
         └────────┴────────┴────────┴────────┘

This is exactly what the conversion needs — since the code of ASCII '0' is 0x30 and '1' is 0x31 we need to extract the lowest bit of each byte (of course if we're sure that input is valid).

Example string "11100100" is encoded as 0x3131313030313030:

src  = 0x3131313030313030 // 64-bit word
mask = 0x0101010101010101 // 64-bit word

result = pext(src, mask)

The value of result is 0xe4 = 0b11100100.

Working example is available at github (see parse_string.c).

Conversion numbers to octal representation

Thu, 02 Oct 2014 12:00:00 +0100

Introduction

Conversion to octal isn't very computer-friendly, each digit occupy 3 bits, it isn't power of two. The smallest number of bits to convert worth to consider is 12 (3 nibbles and 4 digits):

┌──────────────┐
│dddc ccbb baaa│
└──────────────┘
8┈┈11 4┈┈7 0┈┈3

Then output is a 32-bit word:

┌────────┬────────┬────────┬────────┐
│00000ddd│00000ccc│00000bbb│00000aaa│
└────────┴────────┴────────┴────────┘

Conversion to ASCII require single add of 0x30303030 (0x30 = ord('0')).

Determining if an integer is a power of 2 --- part 2

Wed, 01 Oct 2014 12:00:00 +0100

The idea is simple:

precondition: x is not zero;
isolate lowest bit set: x & -x;
check if this number is equal to x: x == (x & -x).

Sample code:

uint32_t isolate_lowest_set(uint32_t x) {
    return x & -x;
}

int is_power_of_two_non_zero(uint32_t x) {
    return isolate_lowest_set(x) == x;
}

int is_power_of_two(uint32_t x) {
    return x != 0 && is_power_of_two_non_zero(x);
}

GCC 4.8.2 generates for is_power_of_two following code:

8b 54 24 04     mov    0x4(%esp),%edx
31 c0           xor    %eax,%eax
85 d2           test   %edx,%edx
74 0e           je     8048488 <is_power_of_two+0x18>
89 d0           mov    %edx,%eax
f7 d8           neg    %eax
21 d0           and    %edx,%eax
39 c2           cmp    %eax,%edx
0f 94 c0        sete   %al
0f b6 c0        movzbl %al,%eax
f3 c3           repz ret

Unfortunately compiler inserted a jump. But when we are sure that arguments are non-zero then only 5 basic instruction are required to perform this check.

Conditionally fill word (for limited set of input values)

Wed, 01 Oct 2014 12:00:00 +0100

"Limited" means a value where at most one bit is set. I.e. values are zero and all powers of two. Specification:

if x is zero then result is also zero,
if x is power of two result is word full of zeros.

Naive implementation:

uint32_t fill_word_naive(uint32_t x) {

    return x ? 0xffffffff : 0x00000000;
}

GCC 4.8.2 produces following code:

89 c2        mov    %eax,%edx
f7 da        neg    %edx
21 c2        and    %eax,%edx
39 c2        cmp    %eax,%edx
0f 94 c0     sete   %al
0f b6 c0     movzbl %al,%eax

Not bad, but this can be done much simpler. We know that a value is zero or have exactly one bit set — first we have to copy this bit to the highest position, and then populate the MSB using arithmetic shift right.

Copying a bit can be done using single operation — arithmetic negation:

x	-x
`00000000`	`00000000`
`00000001`	`ffffffff`
`00000002`	`fffffffe`
`00000004`	`fffffffc`
`00000008`	`fffffff8`
`00000010`	`fffffff0`
`00000020`	`ffffffe0`
`00000040`	`ffffffc0`
`00000080`	`ffffff80`
`00000100`	`ffffff00`
`00000200`	`fffffe00`
`00000400`	`fffffc00`
`00000800`	`fffff800`
`00001000`	`fffff000`
`00002000`	`ffffe000`
`00004000`	`ffffc000`
`00008000`	`ffff8000`
`00010000`	`ffff0000`
`00020000`	`fffe0000`
`00040000`	`fffc0000`
`00080000`	`fff80000`
`00100000`	`fff00000`
`00200000`	`ffe00000`
`00400000`	`ffc00000`
`00800000`	`ff800000`
`01000000`	`ff000000`
`02000000`	`fe000000`
`04000000`	`fc000000`
`08000000`	`f8000000`
`10000000`	`f0000000`
`20000000`	`e0000000`
`40000000`	`c0000000`
`80000000`	`80000000`

Now the procedure could be saved as:

uint32_t fill_word_naive(uint32_t x) {

    return ((int32_t)(-x) >> 31);
}

The compilation result:

f7 d8        neg    %eax
c1 f8 1f     sar    $0x1f,%eax

Just two simple instructions.

2018-03-11: GCC 7.2 still compiles fill_word_naive the same way, but clang 6.0 produces the final, two-instruction sequence.

Small win over compiler

Tue, 30 Sep 2014 12:00:00 +0100

There are some places where a low-level programmer can beat a compiler. Consider this simple code:

#include <stdint.h>

uint32_t bsr(uint32_t x) {
    // xor, because this builtin returns 31 - bsr(x)
    return __builtin_clz(x) ^ 31;
}

uint32_t min1(uint32_t x) {
    if (x != 0) {
        return bsr(x) + 1;
    } else {
        return 1;
    }
}

Function min1 is compiled to (GCC 4.8 with flag -O3):

min1:
    movl    4(%esp), %edx
    movl    $1, %eax
    testl   %edx, %edx
    je  .L3
    bsrl    %edx, %eax
    addl    $1, %eax
.L3:
    rep ret

There is a conditional jump, not very good. When we rewrite the function:

uint32_t min2(uint32_t x) {
    return bsr(x | 1) + 1;
}

Result is this nice branchless code:

min2:
    movl    4(%esp), %eax
    orl $1, %eax
    bsrl    %eax, %eax
    addl    $1, %eax
    ret

Conclusion: it's worth to check a compiler output. Sometimes.

Interpolation search revisited

Thu, 25 Sep 2014 12:00:00 +0100

Introduction

Interpolation search is the modification of binary search, where the index of a "middle" key is obtained from linear interpolation of values at start & end of a processed range:

a := start
b := end
key := searched key

while a <= b loop
    t := (key - array[a])/(array[b] - array[a])
    c := a + floor(t * (b - a))

    -- in binary search just: c := (a + b)/2

    if key = array[c] then
        return c
    else if key < array[c] then
        b := c - 1
    else
        a := c + 1
    endif

end loop

The clear advantage over basic binary search is complexity O(loglogn). When size of array is 1 million, then average number of comparison in binary search is log₂n = 20. For interpolation search it's log₂log₂n = 4.3 — 4.5 times faster.

However, this property is hold only when the distribution of keys is uniform. I guess this the reason why the algorithm is not well known — enforcing uniform distribution on real data is hard. Also calculating the index c is more computational expansive.

Software emulation of PDEP

Tue, 23 Sep 2014 12:00:00 +0100

Introduction

PDEP is a new instruction from BMI2 (Bit Manipulation Instruction), pseudocode for 32-bit PDEP variant:

src    - a source word
mask   - a mask

result := 0
k := 0
for i := 0 .. 31 loop
    if mask.bit[i] then
        result.bit[i] := src.bit[k]
        k := k + 1
    end if
end loop

Quite complicated, but it is really fast — latency is just 3 cycles and throughput is only one cycle. I've showed how to use this instruction in conversion integers to binary and haxadecimal representation.

I was wondering how this algorithm would execute on an old hardware.

Conversion numbers to hexadecimal representation

Sun, 21 Sep 2014 12:00:00 +0100

Branchless converting single nibble

For a nibble (0..15) stored in a byte conversion could be coded in this way:

char nibble_to_hex(uint8_t byte) {
    assert(byte >= 0 && byte <= 15);

    char c = byte + '0';
    if (byte > 9)
        c += 'a' - '0' - 10;

    return c;
}

If a nibble is greater than 9, then resulting letter have to be ASCII 'a' .. 'f' (or 'A' .. 'F'). It's done with simple correction of the code; value of correction 'a' - 10 - '0' is 39, and for uppercase letters it is 7.

The condition have to be replaced by branchless expression. First code is changed to:

char nibble_to_hex2(uint8_t byte) {
    assert(byte >= 0 && byte <= 15);

    const char corr = 'a' - '0' - 10
    const char c    = byte + '0';

    uint8_t mask    = (byte > 9) ? 0xff : 0x00;

    return c + (mask & corr);
}

We're bit closer. Now the question is: how to get a mask from condition byte > 9? Let's examine simple addition: 128 - 10 + x. For values x = 0 .. 9 the result is in range 128 - 10 .. 128 - 9 and for values x = 10 .. 15 the result is in range 128 .. 128 + 5.

Observation: for x in range 10 .. 15 the result has the highest bit set, otherwise it's clear. In other words we get 0x80 or 0x00 depending on condition, and now mask could be calculated as:

uint8_t tmp  = 128 - 10 + byte;
uint8_t msb  = tmp & 0x80;

uint8_t mask = msb - (msb >> 7) | msb; // 3 operations

Since correction's value is 39 or 7, i.e. is less than 128, the mask could be calculated simpler, yielding values 0x7f/0x00:

uint8_t mask = msb - (msb >> 7); // 2 operations

The final version:

char nibble_to_hex3(uint8_t byte) {
    assert(byte >= 0 && byte <= 15);

    const char corr = 'a' - '0' - 10
    const char c    = byte + '0';

    uint8_t tmp  = 128 - 10 + byte;
    uint8_t msb  = tmp & 0x80;

    uint8_t mask = msb - (msb >> 7); // 0x7f or 0x00

    return c + (mask & corr);
}

Conversion numbers to binary representation

Thu, 11 Sep 2014 12:00:00 +0100

Converting byte — SWAR version 1

Algorithm

Populate byte v (bits: abcdefgh) in a 64-bit word:

const uint64_t r1 = v * 0x0101010101010101

┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│abcdefgh│abcdefgh│abcdefgh│abcdefgh│abcdefgh│abcdefgh│abcdefgh│abcdefgh│
└────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Isolate one bit per byte:

const uint64_t r2 = r1 & 0x8040201008040201

┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│a┈┈┈┈┈┈┈│.b┈┈┈┈┈┈│┈┈c┈┈┈┈┈│┈┈┈d┈┈┈┈│┈┈┈┈e┈┈┈│┈┈┈┈┈f┈┈│┈┈┈┈┈┈g.│┈┈┈┈┈┈┈h│
└────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Clone each bit to the highest position

const uint64_t r3 = r2 + 0x00406070787c7e7f

     ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
r3 = │a┈┈┈┈┈┈┈│.b┈┈┈┈┈┈│┈┈c┈┈┈┈┈│┈┈┈d┈┈┈┈│┈┈┈┈e┈┈┈│┈┈┈┈┈f┈┈│┈┈┈┈┈┈g.│┈┈┈┈┈┈┈h│
     └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

     ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
   + │00000000│01000000│01100000│01110000│01111000│01111100│01111110│01111111│
     └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

     ┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
r3 = │a┈┈┈┈┈┈┈│bx┈┈┈┈┈┈│cxx┈┈┈┈┈│dxxx┈┈┈┈│exxxx┈┈┈│fxxxxx┈┈│gxxxxxx.│hxxxxxxx│
     └────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Move 7th bits to the 0th position and mask the lowest bits:

const uint64_t r4 = (r3 >> 7) & 0x0101010101010101

┌────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│┈┈┈┈┈┈┈a│┈┈┈┈┈┈┈b│┈┈┈┈┈┈┈c│┈┈┈┈┈┈┈d│┈┈┈┈┈┈┈e│┈┈┈┈┈┈┈f│┈┈┈┈┈┈┈g│┈┈┈┈┈┈┈h│
└────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

Finally convert to ASCII:

const uint64_t result = 0x3030303030303030 + r4  // ord('0') == 0x30

Notes

Sample implementation:

uint64_t convert_swar(uint8_t v) {

    const uint64_t r1 =  v * 0x0101010101010101;
    const uint64_t r2 = r1 & 0x8040201008040201;
    const uint64_t r3 = r2 + 0x00406070787c7e7f;
    const uint64_t r4 = (r3 >> 7) & 0x0101010101010101;

    return 0x3030303030303030 + r4;  // ord('0') == 0x30
}

This algorithm requires:

1 multiplication,
1 right shift,
1 bit-and,
1 bit-or,
2 additions.

Multiplication and shift are the slowest operations.

C++ bitset vs array

Sat, 22 Mar 2014 12:00:00 +0100

The C++ bitset conserves a memory, but at cost of speed access. The bitset must be slower than a set represented as a plain old array, at least when sets are small (say a few hundred elements).

Lets look at this simple functions:

// set_test.cpp

#include <stdint.h>
#include <bitset>

const int size = 128;

typedef uint8_t byte_set[size];

bool any_in_byteset(uint8_t* data, size_t size, byte_set set) {
        for (auto i=0u; i < size; i++)
                if (set[data[i]])
                        return true;

        return false;
}


typedef std::bitset<size> bit_set;

bool any_in_bitset(uint8_t* data, size_t size, bit_set set) {
        for (auto i=0u; i < size; i++)
                if (set[data[i]])
                        return true;

        return false;
}

The file was compiled with g++ -std=c++11 -O3 set_test.cpp; Assembly code of the core of any_in_byteset:

28: 0f b6 10              movzbl (%eax),%edx
2b: 83 c0 01              add    $0x1,%eax
2e: 80 3c 11 00           cmpb   $0x0,(%ecx,%edx,1)
32: 75 0c                 jne    40
34: 39 d8                 cmp    %ebx,%eax
36: 75 f0                 jne    28

Statement if (set[data[i]]) return true are lines 28, 2e and 32, i.e.: load from memory, compare and jump. Instructions 2b, 34 and 36 handles the for loop.

Now look at assembly code of any_in_bitset:

5f: 0f b6 13              movzbl (%ebx),%edx
62: b8 01 00 00 00        mov    $0x1,%eax
67: 89 d1                 mov    %edx,%ecx
69: 83 e1 1f              and    $0x1f,%ecx
6c: c1 ea 05              shr    $0x5,%edx
6f: d3 e0                 shl    %cl,%eax
71: 85 44 94 18           test   %eax,0x18(%esp,%edx,4)
75: 75 39                 jne    b0

All these instructions implements the if statement! Again, we have a load from memory (5f), but checking which bit is set requires much more work. The input (edx) is split to the lower part — i.e. bit number (67, 6c) and the higher part — i.e. word index (6c). The last step is to check if a bit is set in a word — GCC used variable shift left (6f), but x86 has BT instruction, so in the perfect code we would have two instructions less.

However, as we see a simple access in the bitset is much more complicated than simple memory fetch from byteset. For small sets memory fetches are well cached and smaller number of instruction improves performance. For really large sets cache misses would kill performance, then bitset is much better choice.

Quick and dirty ad-hoc git hosting

Wed, 19 Mar 2014 12:00:00 +0100

Recently I needed to synchronize my local repository with a remote machine, just for full backup. It's really simple if you have standard Linux tools (Cygwin works too, of course).

in a working directory run:

$ pwd
/home/foo/project
#         ^^^^^^^
$ git update-server-info

in the parent directory start HTTP server:

$ cd ..
$ pwd
/home/foo
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

on a remote machine clone/pull/whatever:

$ git clone http://your_ip:8000/project/.git
                                ^^^^^^^

Step 1 have to be executed manually when the local repository has changed.

Is const-correctness paranoia good?

Wed, 19 Mar 2014 12:00:00 +0100

Yes, definitely. Lets see this simple example:

$ cat test.cpp
int test(int x) {
 if (x = 1)
  return 42;
 else
  return 0;
}
$ g++ -c test.cpp
$ g++ -c -Wall test.cpp
int test(int x) {
 if (x = 1)
  return 42;
 else
  return 0;
}

Only when we turn on the warnings, a compiler tell us about a possible error. Making the parameter const shows us error:

$ cat test2.cpp
int test(int x) {
 if (x = 1)
  return 42;
 else
  return 0;
}
$ g++ -c test.cpp
test2.cpp: In function ‘int test(int)’:
test2.cpp:2:8: error: assignment of read-only parameter ‘x’
  if (x = 1)
        ^

All input parameters should be const, all write-once variables serving as a parameters for some computations should be also const.

Scalar version of SSE move mask instruction

Sun, 16 Mar 2014 12:00:00 +0100

Sample implementation

C function for 32-bit numbers:

#include <stdint.h>

uint32_t movmask(const uint32_t input) {

    assert((input & 0x7f7f7f7f) == 0);

    const uint32_t mult = 0x02040810;

    const uint64_t result = (uint64_t)input * mult;

    return (result >> 32) & 0xff;
}

GCC generated the best possible code:

Disassembly of section .text:

00000000 :
   0: b8 10 08 04 02        mov    $0x2040810,%eax
   5: f7 64 24 04           mull   0x4(%esp)
   9: 0f b6 c2              movzbl %dl,%eax
   c: c3                    ret

C function for 64-bit numbers (the type __int128 is the GCC extension):

uint64_t movmask64_unsafe(const uint64_t input) {

    assert((input & 0x7f7f7f7f7f7f7f7flu) == 0);

    const uint64_t mult = (1lu << (1*8 - 0))
                        | (1lu << (2*8 - 1))
                        | (1lu << (3*8 - 2))
                        | (1lu << (4*8 - 3))
                        | (1lu << (5*8 - 4))
                        | (1lu << (6*8 - 5))
                        | (1lu << (7*8 - 6))
                        | (1lu << (8*8 - 7))
                        ;

    const unsigned __int128 result = (unsigned __int128)mult * (unsigned __int128)input;

    return (result >> 64) & 0xff;
}

And disassembly, GCC generated also the shortest possible code:

48 bf 00 81 40 20 10 08 04 02    movabs $0x204081020408100,%rdi
48 f7 e7                         mul    %rdi
0f b6 c2                         movzbl %dl,%eax

Full source code is available, including the proof written in Python.

SIMD-friendly Rabin-Karp modification

Tue, 11 Mar 2014 12:00:00 +0100

Algorithm

Rabin-Karp algorithm uses a weak hash function to locate possible substring positions. This modification uses merely equality of the first and the last char of searched substring, however equality of chars can be done very fast in parallel, even without SIMD instruction.

Let packed_byte(x) is a function that fills a CPU register with byte x, for example on 32-bit architecture:

// packed_byte(0xab) = 0xabababab
uint32_t packed_byte(uint8_t byte) {
        return 0x01010101 * byte;
}

In a single iteration two registers are filled with part of string:

const size_t n = strlen(string);
const size_t k = strlen(substring);

const size_t first = packed_byte(substring[0]);
const size_t last  = packed_byte(substring[k - 1]);

for (size_t i = 0; i < n - k; i += 4) {
        const uint32_t block_first = string[i     .. i + 4];
        const uint32_t block_last  = string[i + k .. i + k + 4];

        ...
}

Then parallel comparison is done with simple xor:

const uint32_t first_zeros = block_first ^ first;
const uint32_t last_zeros  = block_last  ^ last;

Zero bytes in first_zeros and last_zeros indicate equality of chars. The positions of zero bytes have to be equal, so an additional bit or is needed:

const uint32_t zeros = first_zeros | last_zeros;

Getting zeros requires only:

two memory fetches,
two bit-xor,
one bit-or.

Now we have to check if zeros has any zero bytes, then iterate through zeros and perform byte-wise comparisons for all zero bytes:

if (has_zero_byte(zeros)) {
        for (int j = 0; j < 4; j++) {
                if (is_zero(zeros, j) && memcmp(&string[i + j], substring, k) == 0) {
                        return i + z;
                }
        }
}

Function has_zero_byte(uint32_t word) could be implemented using algorithm from Bit Twiddling Hacks — Determine if a word has a zero byte. Function is_zero(uint32_t word, int k) may use results from has_zero_byte (warning: this method has a drawback).

The worst case complexity is O(n*m). However the method minimize memory fetches during a string scan, also comparisons are performed in parallel, and no preprocessing is required (except two packed_byte calls before the main loop).

C++ standard inaccuracy

Tue, 11 Mar 2014 12:00:00 +0100

First we read:

21.4.1 basic_string general requirements [string.require]

[...]

3 No erase() or pop_back() member function shall throw any exceptions.

... a few pages later:

21.4.6.5 basic_string::erase [string::erase]

basic_string<charT,traits, Allocator>& erase(size_type pos = 0, size_type n = npos);

[...]

2 Throws: out_of_range if pos > size().

Integer log 10 of an unsigned integer --- SIMD version

Sun, 09 Mar 2014 12:00:00 +0100

Fast calculate ceil(log₁₀x) of an unsigned number is described on Bit Twiddling Hacks, this text show the SIMD solution for 32-bit numbers.

Algorithm:

populate value in XMM registers. Since maximum value of this function is 10 we need three registers:

movd   %eax, %xmm0          // xmm0 = packed_dword(0, 0, 0, x)
pshufd $0, %xmm0, %xmm0 \n" // xmm0 = packed_dword(x, x, x, x)
movapd %xmm0, %xmm1
movapd %xmm0, %xmm2

compare these numbers with sequence of powers of 10:

// powers_a = packed_dword(10^1 - 1, 10^2 - 1, 10^3 - 1, 10^4 - 1)
// powers_c = packed_dword(10^5 - 1, 10^6 - 1, 10^7 - 1, 10^8 - 1)
// powers_c = packed_dword(10^9 - 1, 0, 0, 0)
pcmpgtd powers_a, %xmm0
pcmpgtd powers_b, %xmm1
pcmpgtd powers_c, %xmm2

result of comparisons are: 0 (false) or -1 (true), for example:

xmm0 = packed_dword(-1, -1, -1, -1)
xmm1 = packed_dword( 0, 0, -1, -1)
xmm2 = packed_dword( 0, 0, 0, 0)

calculate sum of all dwords:

psrld $31, %xmm0       // xmm0 = packed_dword( 1, 1, 1, 1) - convert -1 to 1
psubd %xmm1, %xmm0     // xmm0 = packed_dword( 1, 1, 2, 2)
psubd %xmm2, %xmm0     // xmm0 = packed_dword( 1, 1, 2, 2)

// convert packed_dword to packed_word
pxor %xmm1, %xmm1
packssdw %xmm1, %xmm0 // xmm0 = packed_word(0, 0, 0, 0, 1, 1, 2, 2)

// max value of word in xmm0 is 3, so higher
// bytes are always zero
psadbw %xmm1, %xmm0   // xmm0 = packded_qword(0, 6)

save a result, i.e. the lowest dword:
```
movd %xmm0, %eax      // eax = 6
```

Sample program is available.

Mask for zero/non-zero bytes

Sun, 09 Mar 2014 12:00:00 +0100

The description of Determine if a word has a zero byte from "Bit Twiddling Hacks" says about haszero(v): "the result is the high bits set where the bytes in v were zero".

Unfortunately this is not true. The high bits are also set for ones followed zeros, i.e. haszero(0xff010100) = 0x00808080. Of course the result is still valid (non-zero if there were any zero byte), but if we want to iterate over all zeros or find the last zero index, this could be a problem.

It's possible to create an exact mask:

uint32_t nonzeromask(const uint32_t v) {
        // MSB are set if any of 7 lowest bits are set
        const uint32_t nonzero_7bit = (v & 0x7f7f7f7f) + 0x7f7f7f7f;

        return (v | nonzero_7bit) & 0x80808080;
}

uint32_t zeromask(const uint32_t v) {
         // negate MSBs
         return (nonzeromask(v)) ^ 0x80808080;
}

Function nonzeromask requires four simple instructions, and zeromask one additional xor.

GCC --- asm goto

Sun, 09 Mar 2014 12:00:00 +0100

Starting from GCC 4.5 the asm statement has new form: asm goto. A programmer can use any label from a C/C++ program, however a output from this block is not allowed.

Using an asm block in an old form requires more instructions and an additional storage:

uint32_t bit_set;
asm (
       "bt %2, %%eax  \n"
       "setc %%al  \n"
       "movzx %%al, %%eax \n"
       : "=a" (bit_set)
       : "a" (number), "r" (bit)
       : "cc"
);

if (bit_set)
       goto has_bit;

Above code is compiled to:

80483f6:       0f a3 d8                bt     %ebx,%eax
80483f9:       0f 92 c0                setb   %al
80483fc:       0f b6 c0                movzbl %al,%eax
80483ff:       85 c0                   test   %eax,%eax
8048401:       74 16                   je     8048419

With asm goto the same task could be writting shorter and easier:

asm goto (
       "bt %1, %0 \n"
       "jc %l[has_bit] \n"

       : /* no output */
       : "r" (number), "r" (bit)
       : "cc"
       : has_bit // name of label
);

Complete demo is available. See also GCC documentation about Extended Asm.

Slow-paths in GNU libc strstr

Mon, 03 Mar 2014 12:00:00 +0100

I've observed that some patterns issued to strstr cause significant slowdown.

Sample program kill-strstr.c executes strstr(data, pattern), where data is a large string (16MB) filled with the character '?'. Patterns are read from the command line.

On my machine following times were recorded:

1. searching string 'johndoe'...
                time: 0.032
2. searching string '??????????????????a'...
                time: 0.050
3. searching string '??????????????????????????????a'...
                time: 0.049
4. searching string '???????????????????????????????a'...
                time: 0.274
5. searching string '??????????????????????????????a?'...
                time: 0.356
6. searching string '??????????????????????????????a??????????????????????????????'...
                time: 0.396

Slowdown is visible in case 4 (5 times slower than pattern 3). Pattern has 32 characters, and contains '?', except the last char.
Even bigger slowdown occurs in case 5 (7 times slower). This pattern also contains 32 chars, but the position of the single letter 'a' is last but one.
Similar slowdown occurs in case 6 (nearly 8 times slower). In this pattern single letter 'a' is surrounded by thirty '?'.

Penalties of errors in SSE floating point calculations

Sun, 26 Jan 2014 12:00:00 +0100

Introduction

SSE provides not widely known control register, called MXCSR. This register plays three roles:

Controls calculations:
1. flag flush to zero,
2. flag denormals are zeros,
3. rounding mode (not covered in this text).
Allow to mask/unmask floating-point exceptions.
Save information about floating-point errors — these flags are sticky, i.e. a programmer is responsible for clearing them.

Possible errors in SSE floating point calculations are:

division by zero,
underflow,
overflow,
operations on denormalized values,
invalid operations, like square root of negative number, division zero by zero.

x86 - ISA where 80% of instructions are unimportant

Wed, 01 Jan 2014 12:00:00 +0100

Detailed results

Whole table as txt file.

instruction	count	%
mov	5934098	37.63%
call	1414355	8.97%
lea	1071501	6.79%
movl	760677	4.82%
push	655921	4.16%
jmp	611540	3.88%
add	560517	3.55%
je	490250	3.11%
test	475899	3.02%
pop	441608	2.80%
sub	366228	2.32%
cmp	326379	2.07%
jne	264110	1.67%
nop	242356	1.54%
ret	238569	1.51%
xor	148194	0.94%
movzbl	122730	0.78%
and	88863	0.56%
xchg	66885	0.42%
cmpl	64907	0.41%
movzwl	64589	0.41%
movb	57247	0.36%
or	52138	0.33%
shl	50908	0.32%
cmpb	50152	0.32%
jle	41083	0.26%
leave	39923	0.25%
fldl	37428	0.24%
fstpl	37368	0.24%
shr	36503	0.23%
jbe	32866	0.21%
ja	32333	0.21%
sar	30917	0.20%
flds	29672	0.19%
subl	27636	0.18%
setne	27626	0.18%
testb	27420	0.17%
addl	25906	0.16%
imul	25569	0.16%
jg	24796	0.16%
fstp	24349	0.15%
fxch	23464	0.15%
js	21550	0.14%
fstps	21248	0.13%
sbb	16607	0.11%
inc	16200	0.10%
lock	16049	0.10%
jae	14825	0.09%
sahf	14765	0.09%
dec	14276	0.09%
fnstsw	14026	0.09%
sete	13902	0.09%
movw	13895	0.09%
adc	13640	0.09%
jb	12467	0.08%
jl	11700	0.07%
repz	11178	0.07%
fldcw	11110	0.07%
jge	11019	0.07%
movswl	10816	0.07%
fildl	8852	0.06%
cmpw	7601	0.05%
jns	7490	0.05%
fldz	7331	0.05%
fmul	7229	0.05%
out	7203	0.05%
not	7028	0.04%
movsbl	6720	0.04%
in	6503	0.04%
fld	6309	0.04%
faddp	6254	0.04%
fstl	5760	0.04%
fucom	5753	0.04%
neg	5725	0.04%
fucompp	5354	0.03%
rep	5059	0.03%
fmuls	5039	0.03%
pushl	4430	0.03%
jp	4424	0.03%
fnstcw	4400	0.03%
fld1	4176	0.03%
fmulp	4133	0.03%
orl	3927	0.02%
fadds	3789	0.02%
movq	3779	0.02%
fistpl	3709	0.02%
cltd	3597	0.02%
fmull	3313	0.02%
stos	3298	0.02%
lret	3183	0.02%
scas	3103	0.02%
lods	3066	0.02%
cwtl	3064	0.02%
fadd	2852	0.02%
fucomp	2678	0.02%
orb	2481	0.02%
fildll	2418	0.02%
andl	2379	0.02%
setb	2337	0.01%
andb	2263	0.01%
552 rows more...

I accidentally created an infinite loop

Mon, 30 Dec 2013 12:00:00 +0100

I needed to iterate through all values of 32-bit unsigned integer, so I wrote:

#include <stdint.h>
for (uint32_t i=0; i <= UINT32_MAX; i++) {
   // whatever
}

Is it ok? No, because the value of uint32_t will never exceed UINT32_MAX = 0xffffffff. Of course we can use larger types, like uint64_t, but on 32-bit machines this requires some additional instructions. For example gcc 4.7 compiled following code:

void loop1(void (*fun)()) {
        for (uint64_t i=0; i <= UINT32_MAX; i++) {
                fun();
        }
}

to:

00000000 :
   0:   57                      push   %edi
   1:   bf 01 00 00 00          mov    $0x1,%edi
   6:   56                      push   %esi
   7:   31 f6                   xor    %esi,%esi
   9:   53                      push   %ebx
   a:   8b 5c 24 10             mov    0x10(%esp),%ebx
   e:   66 90                   xchg   %ax,%ax
  10:   ff d3                   call   *%ebx
  12:   83 c6 ff                add    $0xffffffff,%esi
  15:   83 d7 ff                adc    $0xffffffff,%edi
  18:   89 f8                   mov    %edi,%eax
  1a:   09 f0                   or     %esi,%eax
  1c:   75 f2                   jne    10
  1e:   5b                      pop    %ebx
  1f:   5e                      pop    %esi
  20:   5f                      pop    %edi
  21:   c3                      ret

TBH, I have no idea why such a weird sequence has been generated (add, adc, or, jnz). The simplest and portable solution is to detect a wrap-around of 32-bit value after increment:

uint32_t i=0;
while (1) {
        // loop body

        i += 1;
        if (i == 0) // wrap-around
                break;
}

In an assembly code it's even simpler, because a CPU sets the carry flag:

         xor %eax, %eax
loop:
         ; loop body

         add $1, %eax
         jnc loop

Calculate floor value without FPU/SSE instruction

Sun, 29 Dec 2013 12:00:00 +0100

Presented algorithm works properly for any normalized floating point value, examples are given for double precision numbers (64-bit).

The layout of value 8192.625:

 S  exp + bias  fraction
┌─┬───────────┬────────────────────────────────────────────────────┐
│0│10000001100│0000000000000101000000000000000000000000000000000000│
└─┴───────────┴────────────────────────────────────────────────────┘
63 62       52 51                                                  0

The exponent value is 13, thus fraction bits spans range 0 .. 52 - 13:

┌─┬───────────┬────────────────────────────────────────────────────┐
│0│10000001100│0000000000000101000000000000000000000000000000000000│
└─┴───────────┴────────────────────────────────────────────────────┘
                           │                                      │
                           ╰──────╴ bits after decimal dot ╶──────╯

To calculate the floor of value, the bits after decimal dot have to be clear. This operation doesn't alter the exponent, so only single bit-and is required, and no extra calculations are needed.

To be precise, value of dot_position := 52 - exponent decides what have to be done:

If dot_position > 52 then the value is less than 1.0, i.e. floor(x) = 0.
If dot_position <= 0 then the value have no fraction part (it is larger than 2⁵²), i.e. floor(x) = x.
If 0 < dot_position <= 52 then the value have fraction part and bits after dot_position bits have to be cleared.

The number of operations:

Extracting exponent value requires: 1 shift, 1 bit-and, 1 subtract.
Determining which case have to be selected requires up to 4 comparisons.
When clearing bits is needed, then building bit-mask require: 1 shift, 1 subtract, 1 bit negation, 1 bit-and.

Sample program is available:

$ ./demo 123.75 0.012 120000000000000 0.99999999 99.999
floor(123.75000000) = 123.00000000
floor(0.01200000) = 0.00000000
floor(120000000000000.00000000) = 120000000000000.00000000
floor(0.99999999) = 0.00000000
floor(99.99900000) = 99.00000000

Convert float to int without FPU/SSE

Fri, 27 Dec 2013 12:00:00 +0100

This short article shows how a normalized floating point value could be safely converted to an integer value without assistance of FPU/SSE. Only basic bit and arithmetic operations are used. In the worst case following operations are performed:

6 comparisons,
2 subtracts,
1 and,
1 or,
2 shifts (one with variable, one with constant amount).

The floating point value is calculated as: − 1^sign ⋅ (1 + fraction) ⋅ 2^{exponent − bias}. The fraction part is in range [0, 1). For 32-bit values sign has 1 bit, exponent has 8 bits, fraction has 23 bits, and bias has value 127; exponent + bias is saved as an unsigned number.

The layout of binary word:

┌─┬────────┬───────────────────────┐
│S│exp+bias│        fraction       │
└─┴────────┴───────────────────────┘
31 30    23 22                     0

Let clear fields exponent + bias and sign and restore the implicit integer 1 at 24-th bit:

┌─┬────────┬───────────────────────┐
│0│00000001│XXXXXXXXXXXXXXXXXXXXXXX│
└─┴────────┴───────────────────────┘
31 30    23 22                     0

The value of such 32-bit word treated as an unsigned integer is (1 + fraction) ⋅ 2²³. To calculate the result this word have to be shifted left or right depending on value and sign of shift := exponent - 23; only few cases have to be considered:

If shift is negative, then the word must be shifted right. The number of significant bits is 24, so if shift < − 24 the result is always zero.
If shift is positive, then the word must be shifted left. Since destination is a 32-bit signed value, thus maximum shift is 31 - 24 = 7 bits --- when shift is greater than 7, then overflow will occur.
If − 24 < shift < 7 then the number could be safely shifted. When shift = 7, then result has exactly 31 significant bits, thus a range check is required: for positive numbers (sign = 0) maximum value is 2³¹ − 1 and for negative is 2³¹.

Sample program is available.

fopen a directory

Wed, 25 Dec 2013 12:00:00 +0100

It's not clear how function fopen applied to a directory should behave, manual pages don't say anything about this. So, our common sense fail — at least when use a standard library shipped with GCC, because fopen returns a valid handle.

Discussion on stackoverflow pointed that fseek or ftell would fail. But on my system it's not true, ftell(f, 0, SEEK_END) returns the size of an opened directory.

Only when we trying to read data using fread or fgetc the errno variable is set to EISDIR error code.

Here is output from simple test program:

$ ./a.out ~
testing '/home/wojtek'...
fopen: Success [errno=0]
fseek: Success [errno=0]
fseek result: 0
ftell: Success [errno=0]
ftell result: 24576
feof: Success [errno=0]
feof result: 0 (EOF=no)
fgetc: Is a directory [errno=21]
fgetc result: -1 (EOF=yes)
fread: Is a directory [errno=21]
fread result: 0

x86 extensions are useless

Thu, 12 Dec 2013 12:00:00 +0100

Intel announced new extension to SSE: instructions accelerating calculating hashes SHA-1 and SHA256. As everything else added recently to the x86 ISA, these new instructions address special cases of "something". The number of instructions, encoding modes, etc. is increasing, but do not help in general.

Let see what sha1msg1 xmm1, xmm2 does (type of arguments is packed dword):

result[0] := xmm1[0] xor xmm1[2]
result[1] := xmm1[1] xor xmm1[3]
result[2] := xmm1[2] xor xmm2[0]
result[3] := xmm1[3] xor xmm2[1]

The logical operation "xor" is hardcoded. Why can't we use "or", "and", "not and"? These operations are already present in ISA.
Indices to xmm1 and xmm2 are hardcoded too. The instruction pshufd accepts an immediate argument (1 byte) to select permutation, why sha1msg1 couldn't be feed with 2 bytes allowing a programmer to select any permutations of arguments?
Sources of operators are also hardcoded. Why not use another immediate (1 byte) to select sources, for example 00b = xmm1/xmm1, 01b = xmm1/xmm2, 10b = xmm2/xmm1, 11b = xmm2/xmm2.

Such generic instruction would be saved as generic_op xmm1, xmm2, imm_1, imm_2, imm_3 and execute following algorithm:

for i := 0 to 3 do
        arg1_indice := imm_1[2*i:2*i + 1]
        arg2_indice := imm_2[2*i:2*i + 1]

        if imm_3[2*i] = 1 then
                arg1 := xmm1
        else
                arg1 := xmm2
        end if

        if imm_3[2*i + 1] = 1 then
                arg2 := xmm2
        else
                arg2 := xmm1
        end if

        result[i] := arg1[arg1_indice] op arg2[arg2_indice]
end for

Then sha1msg1 is just a special case:

generic_xor xmm1, xmm2, 0b11100100, 0b01001110, 0b01010000

Maybe this example is "too generic", too complex, and would be hard to express in hardware. I just wanted to show that we will get shine new instructions useful in few cases. Compilers can vectorize loops and make use of SSE, but SHA is used in drivers, OS and is encapsulated in libraries — sha1msg1 and friends will never appear in ordinary programs.

Problems with PDO for PostgreSQL on 32-bit machines

Sat, 07 Dec 2013 12:00:00 +0100

The size of an integer in PHP depends on machine, it has 32 bits on 32-bit architectures, and 64 on 64-bit architectures.

On 32-bit machines PDO for PostgreSQL always convert bigint numbers returned by a server to string. Never casts to integer even if value of bigint would fit in 32-bit signed integer.

Type bigint is returned for example by COUNT and SUM functions.

On 64-bit machines there is no such problem because PHP integer is the same as bigint.

Encoding array of unsigned integers

Sat, 23 Nov 2013 12:00:00 +0100

Introduction

Russ Cox wrote very interesting article about algorithms behind service Google Code Search. In short: files are indexed with trigrams, the query string is split to trigrams and then an index is used to limit number of searched files.

Code search manages a reverse index — there is a file list, and for each trigram there is a list of file's id. I've interested how the list of file's id is stored on a disc. In the Russ implementation such list is built as follows:

ids are sorted in ascending order;
differences between current and previous id are encoded using variable length integers (varint).

The variable length integer is a byte-oriented encoding, similar to UTF-8, where each byte holds 7 bit of data and 1 bit (MSB) is used as the end-of-word marker. Thus smaller values require less bytes: integers less than 128 occupy 1 byte, less than 16384 occupy 2 bytes, and so on.

Important: lists of file's id are encoded independently, because of that it is not possible to use any common attributes to increase compression.

FBSTP --- the most complex instruction in x86 ISA

Thu, 07 Nov 2013 12:00:00 +0100

Sample sources

Sources are available at github.

Program fbst_tests.c converts a number to a string, remove leading zeros, and detects errors:

$ ./test 0 12 5671245 -143433 334535 4543985349054 999999999999999999999
printf => 0.000000
FBSTP  => 0
printf => 12.000000
FBSTP  => 12
printf => 5671245.000000
FBSTP  => 5671245
printf => -143433.000000
FBSTP  => -143433
printf => 334535.000000
FBSTP  => 334535
printf => 4543985349054.000000
FBSTP  => 4543985349054
printf => 10000000000000000000000.000000
FBSTP  => NaN/overflow

Program fbst_speed.c compares instruction FBSTP with simple implementation of itoa. There are no formatting, BCD to ASCII conversion, etc. Numbers from 1 to 10,000,000 are converted.

Results from quite old Pentium M:

FBSTP...
... 2.285 s
simple itoa...
... 0.589 s

and recent Core i7:

FBSTP...
... 2.165 s
simple itoa...
... 0.419 s

There is no difference! FBSTP is just 5% faster on Core i7.

Short story about PostgreSQL SUM function

Mon, 04 Nov 2013 12:00:00 +0100

Here is a simple PostgreSQL type:

CREATE TYPE foo_t AS (
        id    integer,
        total bigint
);

and a simple query wrapped in a stored procedure:

CREATE FUNCTION group_foo()
        RETURNS SETOF foo_t
        LANGUAGE "SQL"
AS $$
        SELECT id, SUM(some_column) FROM some_table GROUP BY id;
$$;

Now, we want to sum everything:

CREATE FUNCTION total_foo()
        RETURNS bigint -- same as foo_t.total
        LANGUAGE "SQL"
AS $$
        SELECT SUM(total) FROM group_foo();
$$;

And we have an error about types inconsistency!

This is caused by SUM function — in PostgreSQL there are many variants of this function, as the db engine supports function name overriding (sounds familiar for C++ guys). There are following variants in PostgreSQL 9.1:

$ \df sum
                                                 List of functions
   Schema   | Name | Result data type | Argument data types | Type
------------+------+------------------+---------------------+------
 pg_catalog | sum  | numeric          | bigint              | agg
 pg_catalog | sum  | double precision | double precision    | agg
 pg_catalog | sum  | bigint           | integer             | agg
 pg_catalog | sum  | interval         | interval            | agg
 pg_catalog | sum  | money            | money               | agg
 pg_catalog | sum  | numeric          | numeric             | agg
 pg_catalog | sum  | real             | real                | agg
 pg_catalog | sum  | bigint           | smallint            | agg

Smaller types are promoted: from integer we get bigint, from bigint we get numeric, and so on.

PostgreSQL --- faster reads from static tables

Sat, 02 Nov 2013 12:00:00 +0100

Introduction

"Static table" means that data is changed rarely, in my case it's a cache for some reporting system. Table has no any foreign keys, constraints etc. This cache is feed by night cron by a query looking like this:

TRUNCATE reports_cache;
INSERT INTO reports_cache
        (... list of columns ...)
        (SELECT ... FROM reports_cache_data(... some parameters ...));

reports_cache_data is a stored procedure that performs all boring transformations, contains many joins, filters out certain rows, etc. The cache table contains a lot of records.

The main column used by all report is a "creation date", range filtering on this column appear in virtually all queries. For example aggregating query looks like this:

SELECT
        idx1
        count(counter1) as quantity1_count
FROM
        reports_cache
WHERE
        date BETWEEN '2013-01-01' AND '2013-01-31';

Of course there is an index on date field.

PostgreSQL: printf in PL/pgSQL

Sun, 06 Oct 2013 12:00:00 +0100

PostgreSQL wiki has entry about sprintf — is is quite simple approach (and isn't marked as immutable). The main drawback is iterating over all chars of a format string. Here is a version that use strpos to locate % in the format string, and it's faster around 2 times:

CREATE OR REPLACE FUNCTION printf2(fmt text, variadic args anyarray) RETURNS text
LANGUAGE plpgsql IMMUTABLE AS $$
   DECLARE
      argcnt  int  := 1;
      head    text := ”;     -- result
      tail    text := fmt;    -- unprocessed part
      k       int;
   BEGIN
      LOOP
         k := strpos(tail, '%');
         IF k = 0 THEN
            -- no more '%'
            head := head || tail;
            EXIT;
         ELSE
            IF substring(tail, k+1, 1) = '%' THEN
               -- escape sequence '%%'
               head := head || substring(tail, 1, k);
               tail := substring(tail, k+2);
            ELSE
               -- insert argument
               head := head || substring(tail, 1, k-1) || COALESCE(args[argcnt]::text, ”);
               tail := substring(tail, k+1);
               argcnt := argcnt + 1;
            END IF;
         END IF;
      END LOOP;

      RETURN head;
END;
$$;

SSE: trie lookup speedup

Mon, 30 Sep 2013 12:00:00 +0100

Introduction

Trie is a multiway tree where each edge is labelled by a letter; such trees are used as dictionaries. Lookup takes O(k) time, where k is a string length.

Trie node is defined in C:

typedef struct TrieNode {
    bool        eow;        // end of word marker
    int         n;          // children count
    char*       letter;     // list of edge labels
    TrieNode**  children;   // pointers to children nodes
}

Lookup procedure is simple.

bool trie_lookup(TrieNode* root, const char* word, const int size) {
    TrieNode* node = root;

    for (int i=0; i < size; i++) {
        // go through edge labelled by i-th letter
        node = trie_next(node, word[i]);
        if (node == NULL) {
            return false;
        }
    }

    // we visited 'size' nodes, check if last
    // visited node is located at end-of-word

    return node ? node->eow : false;
}

Function trie_next returns a child node labelled with given letter.

TrieNode trie_next(TrieNode node, char letter);

The implementation of this procedure determines overall searching and inserting time.

Detecting intersection of convex polygons in 2D

Sun, 15 Sep 2013 12:00:00 +0100

Introduction

Detecting intersection of convex polygons is a common problem in a wide range of problems. The method of separated axis theorem (SAT) is widely used, and considered as the easiest and the fastest.

This article presents a quite naive algorithm, that in terms of processing polygon vertices is better than SAT — in the worst case it requires fewer additions & multiplications. However, in practice SAT is faster.

I believe that the approach was already discussed, but quick search didn't return anything.

PHP quirk

Sun, 01 Sep 2013 12:00:00 +0100

PHP is very funny language. Here we are a simple class, without a constructor:

class Foo
{
}

$foo = new Foo($random, $names, $are, $not, $detected);
echo "ok!\n";

One can assume that interpreter will detect undeclared variables, but as their names state this doesn't happen (PHP versions 5.3..5.5):

$ php foo1.php
ok!

When the class Foo have the constructor:

class Foo
{
        public function __construct() { }
}

$foo = new Foo($random, $names, $are, $not, $detected);

everything works as expected:

$ php foo2.php
PHP Notice:  Undefined variable: random in /home/wojtek/foo2.php on line 10
PHP Notice:  Undefined variable: names in /home/wojtek/foo2.php on line 10
PHP Notice:  Undefined variable: are in /home/wojtek/foo2.php on line 10
PHP Notice:  Undefined variable: not in /home/wojtek/foo2.php on line 10
PHP Notice:  Undefined variable: detected in /home/wojtek/foo2.php on line 10

Average of two unsigned integers

Mon, 02 Jul 2012 12:00:00 +0100

Sample program

Following python script verifies both methods (for 8-bit integers)

bits = 8    # unsigned width
mask = (1 << bits) - 1
MSB  = (1 << (bits - 1))

def base(a, b):
    return (a + b)/2


def safe1(a, b):
    LSB_carry = (a & b & 1)
    return (a >> 1) + (b >> 1) + LSB_carry


def safe2(a, b):
    sum = (a + b) & mask
    if sum < a:
        return (sum >> 1) | MSB
    else:
        return sum >> 1


def safe3(a, b):
    sum = (a + b) & mask
    MSB_carry = int(sum < a)

    return (sum >> 1) | (MSB_carry << (bits - 1))


def main():
    n = 1 << bits
    for a in xrange(n):
        for b in xrange(n):
            ref = base(a, b)
            r1  = safe1(a, b)
            r2  = safe2(a, b)
            r3  = safe3(a, b)

            assert ref == r1
            assert ref == r2
            assert ref == r3

if __name__ == '__main__':
    main()

Speeding up LIKE '%text%' queries (at least in PostgeSQL)

Fri, 25 May 2012 12:00:00 +0100

Introduction

PostgreSQL executing queries with construct like '%text%' can't use any system index (I guess none of existing SQL servers have such feature).

But it's possible to speed up such queries with an own index using n-grams. N-gram is a subword consists N successive characters. For example all 3-grams (trigrams) for word "bicycle" are: "bic", "icy", "cyc", "cle". Likewise all 4-grams for this word are: "bicy", "icyc", "ycle".

This article shows two approaches:

An n-gram index as a regular table — may be applicable for engines other than PostgreSQL;
Full Search Text indexes GIN and GIST existing in PostgreSQL since 8.4. Note that both types of indexes has different properties in terms of update and read timings.

SSE: conversion integers to decimal representation

Fri, 21 Oct 2011 12:00:00 +0100

Introduction

With SSE2 instructions it's possible to convert up to four numbers in range 0..9999_9999 and get 32 decimal digits results. This texts describe code for two numbers (suitable for 64-bit conversions) and for one number (suitable for 32-bit conversions).

The outline of algorithm 1 has been posted by Piotr Wyderski on the usenet group pl.comp.lang.c, I merely implemented it. The main idea is to perform in parallel divisions & modulo by 10⁸ (for 64-bit numbers), then 10⁴, 10² and finally 10¹.

I've developed the algorithm 2, converting just a single 8-digit number. First division & modulo by 10⁸ is performed, and an input vector is formed [abcd, abcd, abcd, abcd, efgh, efgh, efgh, efgh]. Then this vector is divided by vector [10³, 10², 10¹, 10⁰, 10³, 10², 10¹, 10⁰] yielding vector [a, ab, abc, abcd, e, ef, efg, efgh]. Obtaining an 8 digits result from this vector requires some shuffling, multiply by 10 and a subtract.

Traversing DAGs

Mon, 11 Apr 2011 12:00:00 +0100

If a DAG has got one component, then the simplest traversing method is depth-first-search, which could be easily implemented recursively (using an implicit stack).

struct DAGNode {
        // user data
        bool    visited;        // after construction = 0
}

void DFS_aux(DAGNode* node, const bool val) {
        if (node->visited != val) {
                // visit node

                node->visited = val;
                for (n in node.connected)
                        DFS_aux(n, val)
        }
}

void DFS(DAGNode node) {
        static val = true;

        DFS_aux(node, val);
        val = not val;
}

On every call of DFS() the variable val is switched, and visited member is marked alternately with true or false.

There is just one problem — what if a traversing method stop execution before visiting all nodes? Of course in such situation we have to visit the DAG twice: on first pass reset (possibly many times) visited member to false, and then visit once each node.

But usually bool have at least 8 bits, so numbers could be used instead of boolean values 0 or 1. On each call of DFS() a reference number is incremented, thanks to that even if previous call stopped in the middle, the procedure will work correctly.

The only moment when visited markers have to be cleared is wrapping a reference numbers to zero. This happen every 256 calls if 8-bit values used; for wider counters (16, 32 bits) max value is greater.

void DFS(DAGNode node) {
        static unsigned int val = 1;
        if (val == 0) {
                // set visited member of all nodes to 0
                val += 1;
        }

        DFS_aux(node, val);
        val += 1;
}

DAWG as dictionary? Yes!

Sat, 09 Apr 2011 12:00:00 +0100

If you read the Wikipedia entry about DAWG, then you find following sentence:

Because the terminal nodes of a DAWG can be reached by multiple paths, a DAWG is not suitable for storing auxiliary information relating to each path, e.g. a word's frequency in the English language. A trie would be more useful in such a case.

This isn't true!

There is a quite simple algorithm, that allow to perform two-way minimal perfect hashing (MPH), i.e. convert any path representing a word to a unique number, or back — a number to a path (word). Values lie in the range 1 .. n, where n is the number of distinct words saved in a DAWG.

The algorithm is described in Applications of Finite Automata Representing Large Vocabularies, by Claudio Lucchiesi and Tomasz Kowaltowski (preprint is freely available somewhere online).

The main part of the algorithm is assigning to each node the number of reachable words from a node; this can be easily done in one pass. Then these numbers are used to perform perfect hashing. Hashing algorithm is fast and simple, translation from pseudocode presented in the paper is straightforward.

Algorithm requires additional memory for numbers in each node and a table of size n to implement dictionary lookups.

I've updated pyDAWG to support MPH.

Python: C extensions --- sequence-like object

Fri, 08 Apr 2011 12:00:00 +0100

If a class has to support standard len() function or operator in, then must be a sequence-like. This requires a variable of type PySequenceMethods, that store addresses of proper functions. Finally the address of this structure have to be assigned to tp_as_sequence member of the main PyTypeObject variable.

Here is a sample code:

static PySequenceMethods class_seq;

static PyTypeObject class_type_dsc = {
        ...
};

ssize_t
classmeth_len(PyObject* self) {
        if (not error)
                return sequence_size;
        else
                return -1;
}

int
classmeth_contains(PyObject* self, PyObject* value) {
        if (not error) {
                if (value in self)
                        return 1;
                else
                        return 0;
        }
        else
                return -1;
}


PyMODINIT_FUNC
PyInit_module() {
        class_seq.sq_length   = classmeth_len;
        class_seq.sq_contains = classmeth_contains;

        class_type_dsc.tp_as_sequence = &class_seq;

        ...
}

Efficient trie representation

Sat, 26 Mar 2011 12:00:00 +0100

Introduction

Each node of a tree store two kinds of data: user data and trie-related data. Let say an alphabet has at most 256 letters, thus a letter could be saved on a byte.

During maintaining tree structure following issues appear:

Amount of memory required to store nodes and edges.
Internal memory fragmentation in underlying dynamic memory allocation routines (malloc/free), that makes real size of tree larger. If small object are allocated/reallocated fragmentation could be significant.
Dynamic allocation/reallocation scatters data on a heap, making cache misses visible. Using arrays may improve memory locality.
Data alignment in nowadays CPUs is important, making reading and writing memory faster. Structure of node could be packed to fill the smallest possible memory at cost of speed.
Time required to retrieve a child node depending on a letter (edge label):
- const (arrays),
- logarithmic (arrays) or,
- linear (arrays, lists).

In experiments Polish, English, French and German words lists were used. User data has got two pointers, i.e. additional 8 bytes per node.

list	words	trie nodes
english	138 622	312 855
german	162 032	610 470
french	629 420	1 297 080
polish	3 588 729	5 933 658

The sample program were linked against GNU libc, and procedure malloc_stats from malloc.h was used to obtain statistics about the real memory usage.

Python: test if object is iterable

Sat, 26 Feb 2011 12:00:00 +0100

def isiterable(obj)
        try:
                iter(obj)
                return True
        except TypeError:
                return False

Traversing tree without stack

Thu, 17 Feb 2011 12:00:00 +0100

Introduction

Obvious traversal algorithms require O(logn) memory, i.e. an explicit or an implicit stack or a queue.

An iterative algorithm described here performs depth-first-search and requires O(1) memory. In technical terms a reference (or a pointer) to one tree node is needed. This node, called p, is a node processed in the previous step of the algorithm.

Following properties of a tree node x are needed:

it is possible to check if a node y is a child (x.is_child(y));
it is possible to check if a node y is a parent (y = x.parent());
it is possible to get the first child node (x.first_child());
it is possible to get the next sibling child node if another child y is given (x.next_sibling(y));

For binary trees all of these functions are quite simple. The main disadvantage is necessary to remember the parent of each node.

Branchless set mask if value greater or how to print hex values

Wed, 09 Jun 2010 12:00:00 +0100

Suppose we need to get a mask when a nonnegative argument is greater then some constant value; in other words, we want to evaluate following expression:

if x > const_n then
   mask := 0xffffffff;
else
   mask := 0x00000000;

Portable branchless solution:

choose a magic number M := (1 << (k-1)) - 1 - n, where k is a bit position, for example 31 if we operate on 32-bit words
calculate R := x + M
k-th bit of R is set if x > n
fill mask with this bit - see note Fill word with selected bit

The key to understand this trick is binary form of M: 0111..1111zzzz, where z is 0 or 1 depending on n value. When x is greater then n, then x + M has form 1000..000zzzz, because the carry bit propagates through series of ones to the k-th position of the result.

Real world example — branchless converting hex digit to ASCII (M=0x7ffffff6 for k=31 and n=9).

; input:    eax - hex digit
; output:   eax - ASCII letter (0-9, A-F or a-f)
; destroys: ebx

        andl 0xf, %eax
        leal 0x7ffffff6(%eax), %ebx     ; MSB(ebx)=1 when eax >= 10
        sarl $31, %ebx                  ; ebx - mask
        andl  $7, %ebx                  ; ebx = 7 when eax >= 10 (for A-F letters)
        ;andl $39, %ebx                 ; ebx = 39 when eax >= 10 (for a-f letters)
        leal '0'(%eax, %ebx), %eax      ; eax = '0' + eax + ebx => ASCII letter

It is also possible to convert 4 hex digits in parallel using similar algorithm, but the input data have to be correctly prepared. Moreover generating mask requires 3 instructions and one extra register (in a scalar version just one arithmetic shift). I guess it wont be fast on x86, maybe this approach would be good for a SIMD code, where similar code transforms more bytes at once.

; input: eax - four hex digits in form [0a0b0c0d]
; output: eax - four ascii letters
; destroys: ebx, ecx

        leal 0x76767676(%eax), %ebx        ; MSB of each byte is set when corresponding eax byte is >= 10
                                           ; (here: 0x7f - 9 = 0x76)
        andl $0x80808080, %ebx
        movl %ebx, %ecx
        shrl    $7, %ebx
        subl %ebx, %ecx                    ; ecx - byte-wise mask
        ;andl $0x07070707, %ecx            ; for ASCII letters A-F
        andl $0x27272727, %ecx             ; for ASCII letters a-f
        leal 0x30303030(%eax, %ecx), %eax  ; ecx - four ascii letters

See also: SSSE3: printing hex values (weird use of PSHUFB instruction)

Speedup reversing table of bytes

Sat, 01 May 2010 12:00:00 +0100

Tests

Two scenarios of test were considered:

The table size is hardware friendly, i.e. is multiply of implementation base step; also address of table is aligned:
- all procedures use MOVUPS,
- all procedures use MOVAPS.
The size is not hardware friendly and address is not aligned.

Procedures were tested on following computers:

recent Core 2 Due E8200,
quite old Pentium M (instruction PSHUFB not available).

Quick results discussion:

As always speedup depends on the table size — for larger tables speedup is also larger. Max speedup:
- Core 2:
  - 15.5 times — PSHUFB unrolled & MOVAPS
  - 3.5 times — PSHUFB & MOVUPS
- Pentium M:
  - 4 times — BSWAP unrolled
Unaligned memory access kills performance — results clearly shows this behaviour.

Results for Core 2

Results

Results for Pentium M

Results

Determining if an integer is a power of 2

Sun, 11 Apr 2010 12:00:00 +0100

Method from Bit Twiddling Hacks: (x != 0) && (x & (x-1) == 0). GCC compiles this to following code:

; input/ouput: eax
; destroys: ebx

        test    %eax,  %eax     ; x == 0?
        jz      1f

        leal -1(%eax), %ebx     ; ebx := x-1
        test    %eax,  %ebx     ; ZF  := (eax & ebx == 0)

        setz     %al
        movzx    %al, %eax       ; eax := ZF
1:

We can use also BSF and BSR instructions, which determine position of first and last bit=1, respectively. If a number is power of 2, then just one bit is set, and thus these positions are equal. BSx sets also ZF flag if input value is zero.

; input/output: eax
; destroys: ebx, edx

        bsf     %eax, %ebx      ; ebx := LSB's position if eax != 0, ZF = 1 if eax = 0
        jz      1f
        bsr     %eax, %edx      ; edx := MSB's position

        cmp     %ebx, %edx      ; ZF  := (ebx = edx)

        setz    %al
        movzx   %al, %eax       ; eax := ZF
1:

Brenchless conditional exchange

Thu, 08 Apr 2010 12:00:00 +0100

Suppose we have to exchange (or just move) two registers A and B:

C := A xor B
C := 0 if condition is not true
A := A xor C
B := B xor C

If C is 0, then A and B left unchanged, else A and B are swapped. If only a conditional move from B to A is needed, then step 4th have to be skipped.

Here is a sample x86 code, where condition is value of CF:

sbb edx, edx ; part of step 2. - edx = 0xffffff if CF=1, 0x000000 otherwise
mov ecx, eax
xor ecx, ebx ; step 1
and ecx, edx ; completed step 2. - now C is 0 or (A xor B)
xor eax, ecx ; step 3
xor ebx, ecx ; step 4

Branchless moves are possible in Pentium Pro and higher with instructions cmovcc.

STL: map with string as key --- access speedup

Sat, 03 Apr 2010 12:00:00 +0100

The idea is quite simple: we do not have a single stl::map<string, something>, but a vector of maps, indexed with O(1) time — each map stores keys sharing certain properties. Drawback: additional memory.

I've tested following grouping schemes:

the length of string,
the first letter of string (one level trie),
both length and the first letter.

Third is the fastest — around 60% faster then plain std::map from GCC (red-black tree).

Tests: my program read plain text (I've used The Illiad from http://gutenberg.org), text is split into words (~190000) and then each words is inserted into a dictionary (~28000 distinct words); then the same words are searched in dictionaries. Table below summarizes results on my computer (gcc 4.3.4 from Cygwin).

data struct	running time [ms]			speedup [%]
data struct	min	avg	max	min	avg	max
inserting
std::map	269	287	355	100	100	100
first char	218	241	395	81	84	111
length	218	240	345	81	84	97
len./char	165	172	207	61	60	58
searching
std::map	295	322	483	100	100	100
first char	243	263	460	82	82	95
length	238	248	292	80	77	60
len./char	184	190	241	62	60	50

Download test program.

Fill word with selected bit

Thu, 01 Apr 2010 12:00:00 +0100

The most general algorithm

mask bit:
```
[10111010] => [00010000]
```
clone word:
```
a=[00010000], b=[00010000]
```
shift bit in first word to MSB, and to LSB in second word:
```
a=[10000000], b=[00000001]
```
subtract c = a - b:
```
c=[01111111]
```
add missing MSB c = c OR a:
```
c=[11111111]
```

uint32_t fill1(uint32_t a, int bit) {
        uint32_t b;

        b = a = a & (1 << bit);

        a <<= 31 - bit;
        b >>= bit;

        return (a - b) | a;
}

Branchless signum

Thu, 01 Apr 2010 12:00:00 +0100

Problem: calculate value of sign(x):

-1 when x < 0
0 when x = 0,
+1 when x > 0.

My solution do not involve any hardware specific things like ALU flags nor special instructions — just plain AND, OR, shifts.

; input: eax = X

movl %eax, %ebx
sarl $31, %eax  // eax = -1 if X less then zero, 0 otherwise

andl $0x7fffffff, %ebx
addl $0x7fffffff, %ebx // MSB is set if any lower bits were set
shrl $31, $ebx  // eax = +1 if X greater then zero, 0 otherwise

orl %ebx, %eax  // eax = result

C99 implementation:

int32_t sign(int32_t x) {
        int32_t y;
        y = (x & 0x7fffffff) + 0x7fffffff;
        return (x >> 31) | ((uint32_t)y >> 31);
}

Transpose bits in byte using SIMD instructions

Wed, 31 Mar 2010 12:00:00 +0100

Method presented here allows to get any bit permutation, transposition is just one of possible operations. Lookup-based approach would be faster, but algorithm is worth to (re)show.

Algorithm outline for 8-byte vector (with SSE instruction it is possible to get 2 operations in parallel):

fill vector with given byte:

[11010001|11010001|11010001|11010001|11010001|11010001|11010001|11010001]
    ▲        ▲        ▲        ▲        ▲        ▲        ▲        ▲
    │        │        │        │        │        │        │        │
[11010001] ╶─┴────────┴────────┴────────┴────────┴────────┴────────┘

leave one bit per byte:

[10000000|01000000|00000000|00010000|00000000|00000000|00000000|00000001]

perform desired transposition ("move" bits around):

[00000001|00000010|00000000|00001000|00000000|00000000|00000000|10000000]

perform horizontal OR of all bytes:
```
[10001011]
```

Here is my old MMX code (polish text); below SSE/SSE5 implementation details.

Ad 1. Series of punpcklbw/punpcklwb/shufps or pshufb if CPU supports SSSE3.

# 1.1
movd       %eax, %xmm0
shufps    $0x00, %xmm0, %xmm0
punpcklbw %xmm0, %xmm0
punpcklwd %xmm0, %xmm0

# 1.2
pxor      %xmm1, %xmm1
movd       %eax, %xmm0
pshufb    %xmm1, %xmm0

Ad 2. Simple pand with mask packed_qword(0x8040201008040201).

pand  MASK1, %xmm0

Ad 3. If plain SSE instructions are supported this step requires some work. First, each bit is populated to fill the whole byte (using pcmpeq — we get negated result), then mask bits on desired positons.

SSE5 has powerful instruction protb that can do perform rotation of each byte with independent amount — so in this case just one instruction is needed.

# SSE
pcmpeq  %xmm1, %xmm0
pandn   MASK2, %xmm0    # pandn - to negate

# SSE5
protb    ROT, %xmm0, %xmm0

Ad 4. Since bits are placed on distinct positions, we can use instruction psadbw, that calculate horizontal sums of bytewide differences from two registers (separately for low and high registers halves). If one register is full of zeros, we get sum of bytes from other register.

psadbw  %xmm1, %xmm0
movd    %xmm0, %eax

Depending on instruction set, three (SSE) or two (SSE5) additional tables are needed.

PostgreSQL: get selected rows with given order

Tue, 30 Mar 2010 12:00:00 +0100

Suppose that a database stores some kind of a dictionary and an user picks some items, but wants to keep the order. For example the dictionary has entries with id=0..10, and the user picked 9, 2, 4 and 0. This simple query does the job:

foo = SELECT (ARRAY[9,2,4,0])[i] AS index, i AS ord FROM generate_series(1, 4) AS i
SELECT * FROM dictionary INNER JOIN (foo) ON dictionary.id=foo.index ORDER BY foo.ord

Join locate databases

Wed, 03 Dec 2008 12:00:00 +0100

man locatedb says: "Databases can not be concatenated together, even if the first (dummy) entry is trimmed from all but the first database. This is because the offset-differential count in the first entry of the second and following databases will be wrong".

It's true if we follow man authors — but concatenation is possible without reencoding any database.

For details about the compression scheme algorithm please refer to Wikipedia, the file format is described in man locatedb. In short: compression is based on common prefix elimination in a sequence of strings — when a string share prefix with the previous string, we store pair (length of prefix, rest of string). For example if previous string is "aaabbb" and current is "aaabcd", then output is (4, "cd"), where 4 is length of common prefix: "aaab". Locate files also store differences between prefixes lengths; for example (4, "..."), (5, "..."), (2, "...") is encoded as (4, "..."), (5-4=1, "..."), (2-5=-3, "...") — this is the reason why we can't simply join database files.

However joining locate files isn't very complicated and, as I previously stated, do not require reencoding databases. We have to set diff value for the first entry of an appended file to negative value of the length of common prefix for the last entry of first file.

For example when the first file contains three entries (0, "..."), (10, "..."), (-2, "..."), then last length is 0+10-2 = 8. The second file contains (0, "..."), (5, "..."). After join: (0, "..."), (10, "..."), (-2, "..."), (-8, "..."), (5, "...").

Some time ago I wrote python utility/library, and now extended it to perform this task. Implementation details:

To obtain the length L of common prefix of the last entry, the first database is decoded in a dry-mode (no results are saved).
Then the first file is simply copied to an output file.
Before copy the second database skip first dummy-entry (diff=0, string="LOCATE02") and skip diff=0 of second entry — this need simple file seek. Then save diff=-L, and finally copy rest of the second database file.

I've tested joined database with native Linux locate (under Cygwin) and didn't notice any problems.

SSE4.1: PHMINPOSUW --- insertion sort

Sun, 03 Aug 2008 12:00:00 +0100

Unusual application of PHMINPOSUW instruction as key part of insertion sort for 8 element tables. I guess it won't find any practical usage.

Implementation:

typedef uint16_t table[8];

table max[8] = {
    {0xffff, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000},
    {0x0000, 0xffff, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000},
    {0x0000, 0x0000, 0xffff, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000},
    {0x0000, 0x0000, 0x0000, 0xffff, 0x0000, 0x0000, 0x0000, 0x0000},
    {0x0000, 0x0000, 0x0000, 0x0000, 0xffff, 0x0000, 0x0000, 0x0000},
    {0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0xffff, 0x0000, 0x0000},
    {0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0xffff, 0x0000},
    {0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0x0000, 0xffff}
};

void sse4_sort(table T) {
    uint32_t dummy;

    __asm__ volatile (
    "       movdqu (%%eax), %%xmm0          \n"
    "       xor %%ecx, %%ecx                \n"     // i = 0
    "1:                                     \n"
    "       phminposuw %%xmm0, %%xmm1       \n"     // find min, and its index j
    "       movd %%xmm1, %%edx              \n"
    "       movw   %%dx, (%%eax, %%ecx, 2)  \n"     // save min at i-th position
    "                                       \n"
    "       shrl   $16, %%edx               \n"
    "       shll    $4, %%edx               \n"
    "                                       \n"
    "       por  max(%%edx), %%xmm0         \n"     // set max at pisition j
    "                                       \n"
    "       addl    $1, %%ecx               \n"     // i += 1
    "       cmp     $8, %%ecx               \n"
    "       jl      1b                      \n"

    :
    : "a" (T)
    : "ecx", "edx"
    );
}

SSSE3: PMADDUBSW and image crossfading

Sat, 21 Jun 2008 12:00:00 +0100

Introduction

Image crossfading is a kind of alpha blending where a final pixel is the result of linear interpolation of pixels from two images:

result_pixel = pixel1 * alpha + pixel2 * (1 - alpha)

where alpha lie in range [0, 1]. Of course when operating on "pixels" color components are considered; components are unsigned bytes.

SSSE3 introduced instruction PMADDUBSW. This instruction multiply a destination vector of unsigned bytes by a source vector of signed bytes — the result is a vector of signed words. Then adjacent words are added with signed saturation (the same operation as PHADDSW).

This is exactly what crossafading needs.

The obvious drawback is that instruction operates on signed values. Because alpha must be positive, this reduces resolution of alpha from 8 to 7 bits. (was: Because multiplication results are signed and then added, the sum must not be greater than 32767 — this requirement reduces resolution by another bit. Finally alpha must lie in range [0..63].) Dmitry Petrov pointed out that alpha can be a 7-bit value, as such value never cause an overflow. Let's assume that both pixel1 and pixel2 have maximum value, and check if following inequality is true:

(1) 255 * alpha + 255 * (127 - alpha) < 2^15 - 1
(2)                         255 * 127 < 2^15 - 1
(3)                             32385 < 32767

Obviously the inequality is true.

SSE: conversion uint32 to float

Wed, 18 Jun 2008 12:00:00 +0100

There is no such instruction — CVTDQ2PS converts signed 32-bit ints. Solution: first zero the MSB, such number is never negative in U2, so mentioned instruction could be used. Then add 2³² if the MSB was set.

float    CONST[4]     SIMD_ALIGN = packed_float((float)((uint32_t)(1 << 31))); /* 2^31 */
uint32_t MASK_0_30[4] SIMD_ALIGN = packed_dword(0x7fffffff);
uint32_t MASK_31[4]   SIMD_ALIGN = packed_dword(0x80000000);

void convert_uint32_float(uint32_t in[4], float out[4]) {
    __asm__ volatile (
    "movdqu   (%%eax), %%xmm0  \n"
    "movdqa    %%xmm0, %%xmm1  \n"

    "pand   MASK_0_30, %%xmm0  \n" // xmm0 - mask MSB bit - never less then zero in U2
    "cvtdq2ps  %%xmm0, %%xmm0  \n" // convert this value to float

    "psrad        $32, %%xmm1  \n" // populate MSB in higher word (enough to mask CONST)
    "pand       CONST, %%xmm1  \n" // xmm1 = MSB set ? float(2^31) : float(0)

    "addps     %%xmm1, %%xmm0  \n" // add 2^31 if MSB set

    "movdqu    %%xmm0, (%%ebx) \n"

    : /* no output */
    : "a" (in),
      "b" (out)
    );
}

See a sample implementation.

Floating point tricks

Sun, 15 Jun 2008 12:00:00 +0100

Converting float to int

Few years ago I've developed a method that do not need any floating-point operations — description is written in Polish, but sample code should be easy to understand. In short words mantissa is completed with the implicit bit 23 (or 52) and treated as a natural number. Then this number is shifted left or right to place the dot position at 0 — the shift amount depends on the exponent value.

Another method uses floating point operations and is limited to positive number less than 2²³ (float) (and 2⁵² for doubles).

When value 2²³ is added to another float, then just 23 most significant bits are stored — the fraction bits are shifted out.

Let see an example, number 7.25 (111.01) has following floating point representation:

┌─┬────────┬───────────────────────┐
│0│10000001│11010000000000000000000│
└─┴────────┴───────────────────────┘
 S exp+127    normalized mantissa

After adding 2²³:

┌─┬────────┬───────────────────────┐
│0│10010110│00000000000000000000111│
└─┴────────┴───────────────────────┘
 S exp+127    normalized mantissa

Mantissa field treated as natural number contains an integer part of number.

Because addition is used, then the result is rounded or truncated, depending on the current FPU's rounding settings. When bare bit shift is used instead of addition (as in the method mentioned earlier), then the number is always truncated.

Note: this method could be used to get fixed point, just smaller value is needed: 2^{23 − fractionbits}, but this also limit the maximum value of float.

Implementation from sample program float2int.c:

void convert_simple() {
        double C = (1ll << 52);
        union {
                double  val;
                int64_t bin;
        } tmp;

        int i;
        for (i=0; i < SIZE; i++) {
                tmp.val  = in[i] + C;
                tmp.bin  = tmp.bin & 0x000fffffffffffffll;
                out_2[i] = tmp.bin;
        }
}

However this method is slower than ordinal FPU instructions, i.e.:

fldl    (%eax)
fistpl  (%ebx)  (or fisttpl (%ebx) on CPU with SSSE3)

RDTSC on Core2

Sun, 08 Jun 2008 12:00:00 +0100

RDTSC is incremented with bus-clock cycles, and then multiplied by core-clock/bus-clock ratio. From programmer view, RDTSC counter is incremented by value greater then 1, for example on C2D E8200 it is 8.

Latency of RDTSC in Pentium4 is about 60-120 cycles, on AMD CPU around 6 cycles.

PABSQ --- absolute value of two singed 64-bit numbers

Sun, 08 Jun 2008 12:00:00 +0100

Branch-less x86 code:

movl  %eax, %ebx
sarl   $32, %ebx        ; fill ebx with sign bit
xorl  %ebx, %eax        ; negate eax (if negative)
subl  %ebx, %eax        ; increment eax by 1 (if negative)

SSE2:

pshufd $0b11110101, %xmm0, %xmm1        ; populate dwords 3 and 1
psrad   $32, %xmm1      ; fill quad words with sign bit
pxor  %xmm1, %xmm0      ; negate (if negative)
psubq %xmm1, %xmm0      ; increment (if negative)

GCC asm constraints

Sat, 07 Jun 2008 12:00:00 +0100

Read-write variables

asm(
        "..."
        : "+a" (var)
);

SSSE3/SSE4: alpha blending --- operator over

Tue, 03 Jun 2008 12:00:00 +0100

Introduction

Alpha blending refers to many different operations. This note describes results for the over operator that works on RGBA pixels with premultiplied alpha.

Basic formula:

background = (alpha * foreground) + background

where alpha in range [0 .. 255], and + denotes add with saturation.

The reference implementation coded in C:

Rf =  foreground & 0xff
Gf = (foreground >>  8) & 0xff
Bf = (foreground >> 16) & 0xff
Af = (foreground >> 24) & 0xff

Rb =  background & 0xff
Gb = (background >>  8) & 0xff
Bb = (background >> 16) & 0xff

R = (Rf * Af)/256 + Rb
G = (Gf * Af)/256 + Gb
B = (Bf * Af)/256 + Bb

if (R > 255) R = 255
if (G > 255) G = 255
if (B > 255) B = 255

background = R | (G << 8) | (B << 16)

Note: dividing by 256 never bring component value 255 — to obtain correct range some additional operations are needed. Probably no one notice differences.

SSE4: grater/less or equal relations for unsigned bytes/words

Mon, 02 Jun 2008 12:00:00 +0100

Proof

In the proof we consider three cases: x < y, x = y and x > y.

The second and the last column (i.e. left and right side of equivalence) are the same in both cases. QED

Greater or equal

case	x <= y	min(x, y)	min(x, y) = x
x < y	true	x	true
x = y	true	x	true
x > y	false	y	false

Less or equal

case	x >= y	max(x, y)	max(x, y) = x
x < y	false	y	false
x = y	true	x	true
x > y	true	x	true

16bpp/15bpp to 32bpp pixel conversions --- different methods

Sun, 01 Jun 2008 12:00:00 +0100

Introduction

Basically this kind of conversion needs following steps:

extract components R, G and B (using bitwise and)
extend words from 6 or 5 bits to 8 bits (shift left)
place components at desired places in a 32-bit word (shift, bitwise or)

R = (pixel16 and 0x001f) shl 3
G = (pixel16 and 0x07e0) shr 5
B = (pixel16 and 0xf800) shr 11

pixel32 = R or (G shl 8) or (B shl 16)

Since there aren't many pixels (32 or 64 thousand) lookup tables can be used. First approach is to use one big table indexed by pixels treated as natural numbers: this table has size 65536 * 4 bytes = 262144 bytes. Just one memory access is needed to get 32bpp pixel, however the table size is big, and even if it fits in a L2 cache, then the memory latency kill performance.

pixel32 = LUT[pixel16]

Another approach needs two tables indexed by the lower and the higher byte of a pixel, the final pixel is result of bitwise or. These tables has size 2 * 256 * 4 bytes = 2048 bytes — perfectly fit in a L1 cache.

pixel32 = LUT_hi[pixel16 shr 8] or LUT_lo[pixel16 and 0xff]

SSE: modify 32bpp images with lookup tables

Sun, 01 Jun 2008 12:00:00 +0100

x86 code

The x86 code is a base for further improvements. If pixel is loaded into an x86 register, following code can be used to extract all RGBA components:

movl  (%%esi), %%eax    ; eax - pixel

movzbl  %%al, %%ebx     ; R
movzbl  %%ah, %%ecx     ; G
shrl     $16, %%eax
movzbl  %%al, %%edx     ; B
movzbl  %%ah, %%eax     ; A

movl    LUT_R(,%%ebx,4), %%ebx
orl     LUT_G(,%%ecx,4), %%ebx
orl     LUT_A(,%%edx,4), %%ebx
orl     LUT_B(,%%eax,4), %%ebx ; ebx - transformed_pixel

movl    %%ebx, (%%edi)

Code that works with RGB pixels is of course shorter:

movl  (%%esi), %%eax    ; eax - pixel

movzbl  %%al, %%ebx     ; R
movzbl  %%ah, %%ecx     ; G
shrl     $16, %%eax
movzbl  %%al, %%edx     ; B

movl    LUT_R(,%%ebx,4), %%ebx
orl     LUT_G(,%%ecx,4), %%ebx
orl     LUT_B(,%%eax,4), %%ebx ; ebx - transformed_pixel

movl    %%ebx, (%%edi)

SSE4 string search --- modification of Karp-Rabin algorithm

Tue, 27 May 2008 12:00:00 +0100

Introduction

String search is a common task in text processing. There are many algorithms that try to minimize number of exact comparing substrings.

One of them is Karp-Rabin algorithm — char-wise comparison is performed only when values of hash function calculated for a part of text and a substring are equal.

SSE4 introduced complex instruction MPSADBW which calculate eight Manhattan distances (L1) between given 4-byte vector and 8 subsequent vectors; if the distance is zero, then vectors are equal.

The idea of modification is to use equality of 4-byte substring's prefix instead of hash values equality. MPSADBW is fast, it has latency 4 cycles and throughput 2 cycles. Even if latency is not compensated, overall performance is very promising — 0.5 cycle per one 4-byte vectors comparison.

Unfortunately there are three disadvantages:

Searching a substring shorter than 4 chars need some additional work.
A hash is calculated for whole substring, MPSADBW consider just 4-byte prefix, thus the number of false-negative alarms could be greater.
At least the length of a substring must be known. In the sample application text length is also given — this make program shorter and faster.

SSSE3: fast popcount

Sat, 24 May 2008 12:00:00 +0100

Introduction

Population count is a procedure of counting number of ones in a bit string. Intel introduced instruction popcnt with SSE4.2 instruction set. The instruction operates on 32 or 64-bit words.

However SSSE3 has powerful instruction PSHUFB. This instruction can be used to perform a parallel 16-way lookup; LUT has 16 entries and is stored in an XMM register, indexes are 4 lower bits of each byte stored in another XMM register.

SSSE3: printing hex values

Tue, 29 Apr 2008 12:00:00 +0100

SIMD algorithm

Instruction PSHUFB does parallel lookup from 16-byte array stored in an XMM register — this is exactly what bin to hex conversion needs.

Code snippet showing the idea:

movdqa    (%eax), %xmm0 ; xmm0 = {0xba, 0xdc, 0xaf, 0xe8, ...}
movdqa     %xmm0, %xmm1 ; xmm1 -- bits 4..7 shifted 4 positions right
psrlw         $4, %xmm1 ; xmm1 = {0xad, 0xca, 0xfe, 0x80, ...}
punpcklbw  %xmm0, %xmm1 ; xmm0 = {0xba, 0xad, 0xdc, 0xca, 0xaf, 0xfe, 0xe8, 0x80, ...}
                        ; MASK = packed_byte(0x0f)
pand        MASK, %xmm1 ; xmm0 = {0xb0, 0xa0, 0xd0, 0xc0, 0xa0, 0xf0, 0xe0, 0x80, ...}
                        ;      -- bits 0..3
movdqa HEXDIGITS, %xmm0 ; HEXDIGITS = {'0', '1', '2', '3', ..., 'a', 'b', 'c', 'd', 'e', 'f'}
pshufb     %xmm1, %xmm0 ; xmm0 = {'b', 'a', 'd', 'c', 'a', 'f', 'e', '8', ...}