• The situation with loops in Datalove
• A simple design for for loops
• Getting at that loop induction variable
• A protocol for iterating over collections
• What worked and what remains to be solved

The situation with loops in Datalove

Here's how ord.contains worked before:

fun contains<T>(ref self: [T], ref elem: T): bool where { T is ord }
  var found = false
  let n = list_len(ref self)
  var i: index = 0
  loop while i .< n
    if equal_to_at(ref self, i, ref elem) |same|
      if same
        set found = true
        break
      end if
    end if
    set i = icall add_wrapping_index(i, 1)
  end loop
  ret found
end fun

fun equal_to_at<T>(
  ref self: [T], i: index, ref elem: T
): ?bool where { T is ord }
  ret some equal(ref self[i]?, ref elem)
end fun

(For readers new to Datalove: .< is less-than, and icall calls a compiler intrinsic.)

This code has many problems, but here I care about one: self[i]?, which does a bounds check every iteration. (equal_to_at exists only to make room for the early return that ? does.)

Indexing gets much worse for sets and maps. Here's set.is_subset_of:

fun is_subset_of<T>(ref self: #{T}, ref other: #{T}): bool where { T is ord }
  var all_present = true
  let n = len(ref self)
  var i: index = 0
  loop while i .< n
    if get(ref self, i) |elem|
      if contains(ref other, ref elem)
      else
        set all_present = false
      end if
    end if
    set i = icall add_wrapping_index(i, 1)
  end loop
  ret all_present
end fun

get(ref self, i) returns the element at position i in sort order. A set is a B+tree, so it can't jump straight to position i. Each call starts at the leftmost leaf, walks the chain of leaves past i elements, and clones the element it lands on. That's far more than a bounds check and some pointer math. Reaching element i costs O(i), so a loop over the whole set costs O(n²). Maps have the same problem with map.key_at and map.value_at. These algorithmically abhorrent APIs can't be retired until there's another way to iterate.

Language-level iteration, like Python's for x in xs, gets rid of both the bounds checks and the quadratic walks. The simple design is obvious, so I'll just do it now!

There are some constraints. Datalove has a linear type system. Values are neither garbage-collected nor reference-counted. They are always inline value types. So copying can be expensive, but iteration should be cheap. Datalove also has no first-class references. The surface language can't take a pointer into a collection and treat it as a value, to move between bindings or return from a function. That will limit what we can express. But working within limits is fun, so let's have fun.

A simple design for for loops

I want to write contains like this:

fun contains<T>(ref self: [T], ref elem: T): bool where { T is ord }
  var found = false
  for item in ref self
    if equal(ref item, ref elem)
      set found = true
      break
    end if
  end for
  ret found
end fun

item is a reference (ref) binding, a pointer straight into the container. So item can't be moved, and self can't be mutably referenced or mutated by index. This differs from match and destructuring if, which move their operand into the new bindings.

I don't plan to add by-value iteration soon. Still, the loop header should probably say ref somewhere. Datalove generally requires the word ref everywhere there's a reference binding. Here I put it next to the collection, not the binding. As we'll see, a loop header can have two bindings, and I don't want to think yet about one being by-value and the other by-ref. I may have to revisit this.

Lists aren't the only collections that need iteration. There are also maps, sets, tensors, and tables.

Maps and sets are trees kept in total order, so for walks them in that order. A set binds a reference to each element. A map binds references to the key and the value.

fun restock(ref stock: %{string = u32}, low: u32): [string]
  var short: [string] = []
  for name, count in ref stock
    if count .< low
      call list.push(mut short, name@)
    end if
  end for
  ret short
end fun

name@ clones the key out of the map; name itself only points into it.

The bindings are not a tuple: a (K, V) would be a new value built from the two, not a place in the map. Datalove has no first-class references to put in a tuple, so building one would mean cloning both elements. As with lists, the key binding is immutable. The value could someday be mut to allow in-place mutation.

Tensors and tables each add their own complications, but they mostly extend the same design.

Getting at that loop induction variable

Many loops in the system library use the induction variable for more than indexing the collection. ord.index_of needs to keep the index and return it:

fun index_of<T>(ref self: [T], ref elem: T): ?index where { T is ord }
  var at: ?index = none
  let n = list_len(ref self)
  var i: index = 0
  loop while i .< n
    if equal(ref self[i]?, ref elem)
      set at = some i
      break
    end if
    set i = icall add_wrapping_index(i, 1)
  end loop
  ret at
end fun

So iterating over the collection elements isn't enough. Without the index, users would keep their own counter in sync by hand, which is the code I'm trying to get rid of. The loop needs to provide the index too. Sets and maps don't support this form, since they aren't indexed by number.

Go adds a second name to the loop header for the index, as in for i, x := range xs. Zig goes further: for (xs, ys, 0..) |x, y, i| walks any number of slices and ranges in lockstep. I'll do the Go thing for now! (Though I already have Zig syntax elsewhere and it would make sense here as well).

fun index_of<T>(ref self: [T], ref elem: T): ?index where { T is ord }
  var at: ?index = none
  for i, item in ref self
    if equal(ref item, ref elem)
      set at = some i
      break
    end if
  end for
  ret at
end fun

This smells a bit to me, like a big special case. I'm used to Rust and its many iterator adapters, which do this and much more. (What about reverse iteration? Seems important...) I don't know if Datalove will grow an iterator protocol like Rust's. I don't know if it will ever get first-class references to store in an iterator struct. I don't intend Datalove's language integration to be very expressive. So for now I'll accept this compromise to make for useful in the most common case.

A protocol for iterating over collections

Datalove can't express an iterator abstraction in the language. So the runtime gets an iterator protocol, and that is the abstraction. The runtime defines the binary layout of Datalove's types. It also defines functions that operate on those types. It is called by compiler-emitted code and by native code linked to Datalove packages. So iteration needs a clear ABI. In practice most backends won't call the runtime for every step. Calling a native function per element would be slow when the compiler can just bump a pointer. But spelling the API out in the runtime makes the contract clear. And the reference IR interpreter does make those calls, which gives the fast backends something to compare against.

Array-like collections (lists and rank-1 tensors) and tree-like collections (sets and maps) iterate in two fundamentally different ways. Array-like collections bump a pointer through their buffer by a fixed stride. Tree-like collections follow pointers between heap-allocated nodes. Rank-2+ tensors walk row by row, with a third cursor type I won't cover here.

Two cursor types represent these two styles:

#[repr(C)]
pub struct StrideCursor {
    pub elem: *const u8,     // The current element.
    pub stride: usize,       // Bytes from one element to the next.
    pub index: Index,        // The current element's position; len is done.
    pub len: Index,          // How many elements the walk visits.
}

#[repr(C)]
pub struct TreeCursor {
    pub leaf: *const u8,       // The current leaf; null is done.
    pub elem: *const u8,       // The current key; a set's element.
    pub value: *const u8,      // A map's current value; null for a set.
    pub elem_stride: usize,    // Bytes from one key to the next.
    pub value_stride: usize,   // Bytes from one value to the next.
    pub slot: u32,             // The current key's slot within the leaf.
    pub leaf_len: u32,         // How many keys the current leaf holds.
    pub next_leaf_offset: u32, // Where a leaf keeps its next pointer.
    pub keys_offset: u32,      // Where a leaf's keys start.
    pub values_offset: u32,    // Where a leaf's values start.
}

The tree cursor carries a lot more than its position. Starting a walk reads the collection and its type descriptor, and puts everything the rest of the walk needs into the cursor: the strides, and where a leaf keeps its keys, its values, and its pointer to the next leaf. After that, advancing reads only the cursor.

These are part of a public ABI that the compiler knows directly, so it's convenient to share them rather than give each collection its own type. But each collection still needs its own functions in the runtime API:

// Stride cursors
fn dtlv_rti_list_cursor_begin_local(
    rt: LocalRtHandle,
    list_ref: *const u8,
    list_tydesc: *const TyDesc,
    cursor_out: *mut StrideCursor,
) -> RtStatus;
fn dtlv_rti_tensor_cursor_begin_local(
    rt: LocalRtHandle,
    tensor_ref: *const u8,
    tensor_tydesc: *const TyDesc,
    cursor_out: *mut StrideCursor,
) -> RtStatus;
fn dtlv_rti_stride_cursor_advance_local(
    rt: LocalRtHandle,
    cursor_mut: *mut StrideCursor,
) -> RtStatus;

// Tree cursors
fn dtlv_rti_btreeset_cursor_begin_local(
    rt: LocalRtHandle,
    set_ref: *const u8,
    set_tydesc: *const TyDesc,
    cursor_out: *mut TreeCursor,
) -> RtStatus;
fn dtlv_rti_btreemap_cursor_begin_local(
    rt: LocalRtHandle,
    map_ref: *const u8,
    map_tydesc: *const TyDesc,
    cursor_out: *mut TreeCursor,
) -> RtStatus;
fn dtlv_rti_tree_cursor_advance_local(
    rt: LocalRtHandle,
    cursor_mut: *mut TreeCursor,
) -> RtStatus;

Much of this detail comes from Datalove's runtime API conventions. Every call takes a LocalRtHandle and returns an RtStatus. Results go in out-pointers. Some functions are specific to one collection, and some are shared by every collection with that cursor type. Using a cursor goes like this:

• Call the collection's begin function.
• Check whether iteration is done. A StrideCursor is done when index equals len. A TreeCursor is done when leaf is null.
• Read the element pointer off the cursor, and for a map the value pointer.
• For an indexed for i, elem, read index off the StrideCursor. The loop needs no counter of its own.
• Call an advance function to move forward.

The cursors own nothing, so they need no destructor. They are just raw pointers into the collection, which is why the loop body can't mutate self.

I love this for the compiler. Every backend knows exactly what a cursor looks like, so each can open-code the hot parts by doing exactly what the runtime's advance does. The done check, the element read, and every advance are just loads and adds against the struct, plus a pointer chase when a tree cursor reaches the end of a leaf. Only begin is a call, once per loop. For hypothetical users, though, it's tough luck. They can't write an iterator, and only the collections the compiler knows about can be walked.

What worked and what remains to be solved

With the cursor protocol in place, I converted 25 loop while loops to for loops. Fifteen were in the system library, seven in the "store demo" I use to prove features and performance, and three in the benchmarks. All 25 explicit induction variables are gone.

The demo loops were a special kind of ugly. With no way to walk a map, they built a list of its keys with map.keys, then did a full lookup per key to get the value. Now they just bind both:

for category, r in ref revenue
  let c = map.get_or(ref cost, ref category, 0)
  let u = map.get_or(ref units, ref category, 0)
  call list.push(mut rows, (r, category@, c, u))
end for

Whether for is fast depends on that open-coding. This benchmark sums a list of 100,000 u32s 200 times and a set of 5,000 elements 20 times. These are hyperfine means for a release build, including process startup and building the collection.

The middle row is why the backends open-code the cursors. The interpreter always calls the runtime, so its two for rows are the same run. Under the JIT, calling the runtime for every advance made the loop 1.7x slower than open-coding it, even with the done check and the element read already inline. Open-coded, a list's for beats its index loop under the interpreter and AOT, and ties it under the JIT. The index loop was already just a bounds check and an add. The set is the biggest win: linear instead of quadratic, 10x faster on the interpreter and 39x under AOT, at only 5,000 elements. The old set loop runs about the same on every engine because nearly all its time is spent inside the runtime's get.

Not every loop converted. Eight functions in the system library still count by hand.

Some iterate only a sub-collection, where the for loop doesn't quite fit the problem. Some functions jump around, like the merge in sorted and binary_search. No cursor helps there. zip walks two lists in lockstep, which would need two cursors in one loop header, as Zig allows.

And some come down to first-class references. is_sorted and deduped compare each element to the one before it. The obvious way to write that holds a reference to the previous element across iterations.

fun is_sorted<T>(ref self: [T]): bool where { T is ord }
  var ordered = true
  let n = list_len(ref self)
  var i: index = 1
  loop while i .< n
    let i_prev = icall sub_wrapping_index(i, 1)
    if greater_at(ref self, i_prev, i) |out_of_order|
      if out_of_order
        set ordered = false
        break
      end if
    end if
    set i = icall add_wrapping_index(i, 1)
  end loop
  ret ordered
end fun

Datalove can't do that, so these functions keep an index and read both elements back out by position. least and greatest did convert, but they have the same problem in another shape. They can't hold on to the best element so far, so they keep its index and re-index the list on every comparison:

fun least<T>(ref self: [T]): ?T where { T is ord }
  var best: index = 0
  for i, item in ref self
    if less(ref item, ref self[best]?)
      set best = i
    end if
  end for
  ret some self[best]?@
end fun

So the special-cased for that counts the index doubles as a workaround for missing references. A pleasant consolation.

The human docs consist of:

• Datalove Literals - an informal spec of the data model and declarative language.
• Datalove Functions Primer - a short and kinda complete description of the functional-imperative language.
• Datalove Principles - guiding principles for the language.

Besides the docs, this checkpoint was primarily about establishing a baseline language design and compiler architecture that can uphold those principles and serve as a foundation for my own personal language-design goals.

I'm feeling happy about it!

Some of the things I like

The hard distinction between Datalove Literals and Datalove Functions. Even if the declarative language isn't useful as a general-purpose serialization format (and I don't think it is), I continue to find value in the declarative language. I think it helps me clarify my thinking about the type system, imposing hard rules about the syntactic representation of values and synthesizability of types; and I think it will help learners understand the functions language by first understanding the declarative language.

The acyclic module system requires all dependencies to be declared explicitly in a way that is easy to scan, and fully resolves quickly and early.

The compiler has relatively few passes: lex -> parse -> name resolution -> typecheck -> ownership -> IR lowering -> codegen. So far they are relatively simple and fast.

Many backends are implemented: the reference IR walker, bytecode, JIT via Cranelift, AOT via Cranelift, AOT via C. The C backend is low-quality and exists to maintain a proof that it's possible. There is no wasm backend, but the compiler builds for wasm, I have done compile-to-wasm experiments, and I am confident the model is fully compatible.

The performance of the compiler, bytecode interpreter, and JIT are all satisfactory and give me confidence in the language model.

The language fully supports compile-time function evaluation, and the compiler is structured to phase it cleanly.

The syntax has no obvious ambiguities and doesn't ever rely on lookahead to disambiguate potential conflicts. It is a bit unwieldy and verbose in places. For example, all statements lead with a reserved word. I do like a simple old-school statement language.

The brace-matched lexing. This is the first thing I wrote (myself!) for Datalove. The way it allows line-oriented statement parsing without statement terminators, but still allows statements to break across lines in natural ways, feels good to me. I'm sure cases will come up where line-breaking is awkward, but so far I'm happy with it. I'm even fine that forcing < and > to be matched sacrifices them as comparison operators, which become .< and .>.

// No statement terminators anywhere.
fun foo<
  X, Y, Z
>(
  arg1: X, arg2: Y, arg3: Z
): {
  field1: X,
  field2: Y,
  field3: Z,
}
  let packed = (
    arg1, arg2, arg3
  )
  debuglog packed
  let (a, b, c) = packed
  ret {
    field1 = a,
    field2 = b,
    field3 = c,
  }
end fun

The analysis and execution model of chained script units is mostly the same as for functions and function frames, and seems like it's going to work well to fulfill my goals of creating a reactive and deterministic interactive scripting environment. Hard won, but nascent. Lots more to discover here.

The runtime architecture and type representations. Also early code I wrote by hand. The runtime is well-factored, with a clear firewall between type representations, the function interface, and the implementation itself. It doesn't use any global state or rely on platform-specific linker magic and should be compatible with all reasonable target platforms. Type representations are generally simple, what one would expect, and suitable for forming a supportable public ABI.

Some things I don't love

The heavy reliance on sigils for constructors of all the built-in types. This is mostly to keep them unambiguous and to keep the declarative language concise. There are more built-in types to come - am I going to keep finding new sigils for them?

I'm happy to have established very strict numeric semantics as a baseline, but it is not very fun to do math in Datalove. That's not great for a language explicitly intended for interactive use. I expect to have to come up with a more ergonomic experience that involves subtyping and/or lossless coercions. Any concessions here will be done as a mode that can be activated or deactivated.

The @ (adapt) operator. So far it does just two things: clone, and perform lossless numeric conversions. As a magic multi-purpose coercion tool, having just two tasks isn't pulling its weight. And it's a syntactic eyesore.

Type-erased generics and their implementation. I think monomorphization is not viable to achieve the compile-time performance I want, but I am wary of the runtime overhead the current implementation adds to all operations, the complexity of optimizing that, and hard-to-understand performance cliffs. I also find the current internal rules about how type descriptors are passed complex.

Generics also have only the most primitive form of trait bounds. Enough to implement a few standard library functions. Lots more to do here.

For bytecode performance I am comparing to Python. For JIT to Julia. Both have dynamic typesystems, where Datalove is static, though Python's bytecode interpreter is amazingly performant for happy-path synthetic benchmarks, and Julia of course is 100% jitted via LLVM and is very fast. Julia unfortunately has slow startup time though. I am aiming for fast startup time and fast performance. So far Datalove's startup and compilation time is fast, and runtime performance is in the ballpark, enough to give me confidence about the basic data and runtime model (at least without considering generic functions). Sometimes performance is still slower than my expectations. I won't throw out any numbers at this time since they are not rigorous and the implementation and language model are changing rapidly.

Today I was looking at (having an LLM look at, under my close expert supervision) the performance of the sum benchmark, the Datalove implementation being as follows.

fun sum_to(limit: u32): !int
  var total: int = 0
  var i: u32 = 1
  loop while i <= limit
    set total = total + i@
    set i = i +! 1
  end loop
  ret ok total
end fun

let result = sum_to(30000000)
if result |value|
  debuglog(value)
else |err|
  debuglog(false)
end if

This is a bigint benchmark that simply sums the numbers from 1 to 30 million, so once the interpreter is reasonably tuned generally it becomes primarily a test of bigint efficiency. Bigints are implemented in the runtime in Rust, where performance is about representation (are all numbers on the heap? are small numbers packed into the stack?) and algorithmic efficiency. But because bigints are non-copyable types whose digits live on the heap (Datalove has no small-number optimization; every nonzero bigint owns a heap buffer), the basic semantics of math ops and their interaction with deep-cloning has a significant impact on bigint performance.

I generally prefer to implement the simplest and clearest surface semantics reasonable to accomplish a task, and only expand complexity as I better understand the need. So Datalove only supports a few built-in math operations today: binary addition, subtraction, multiplication, division, and unary negation (+, -, *, /, unary -). Though because one of Datalove's principles insists on strict numeric correctness the exact formulation of these is quirky.

So on the sum benchmark, after squeezing out all the basic performance blunders in the allocator and the bigint implementation I was finding the bytecode performance still significantly behind Python's bytecode interpreter, on the order of 2x.

The primary culprit was excessive allocation of bigints. Let's see our sum_to function:

fun sum_to(limit: u32): !int
  var total: int = 0
  var i: u32 = 1
  loop while i <= limit
    set total = total + i@
    set i = i +! 1
  end loop
  ret ok total
end fun

That set total = total + i@ is the whole story. Binary ops borrow their operands so they don't themselves require a clone, but they do produce a fresh value, so a bigint + always performs an allocation (furthermore, because both sides of addition must be the same type, we have to widen the u32 counter to int, another allocation). It seems like we could instead do that operation in place directly into total without reallocating.

To address the allocating math operator that immediately assigns to one of its operands here we could do a fairly simple peephole optimization on set total = total + <something>, the IR for which looks like

v3 = widen s1
v4 = add s0, v3
drop v3
drop.tracked s0
store.move.tracked s0, v4

Eminently pattern-matchable. But one of Datalove's principles is that nothing happens that was not written. I much prefer the language to have surface constructs that map to the required underlying performance mechanisms. That both makes the language's performance characteristics clear, and lets the compiler avoid accumulating passes that slowly eat at compile times. And the surface operation for this is obvious: compound assignment.

set total += i@

This common operation, assign total + i@ to total, implies exactly the optimization needed, and lowers to a single IR instruction that updates total where it lies:

v3 = widen s1
add.assign s0, v3
drop v3

Aside: even though optimizing bigint math is our motivator, having compound assignment should be a win for all types for the bytecode interpreter too, since it can fuse the math and the assignment into a single opcode. Further aside: I'll probably end up doing the peephole optimization anyway...

In-place bigint math

This optimization only matters if we can actually do the bigint math without allocating a new buffer. Can we?

For addition and subtraction, mostly yes. A Datalove int is a pointer to a buffer of 32-bit limbs, a signed limb count, and the buffer's capacity. A sum needs at most one limb more than its longer operand, for the carry, so addition always allocates that extra limb, and when the carry doesn't use it, it is left spare. A running total like total almost always fits in the buffer it already has, so set total += i@ just adds the limbs of i@ into it: no allocation, nothing freed. Only when the sum outgrows the buffer is a new one allocated, just as the binary + would have. Subtraction is an addition of the negated operand, and when the signs make it a true subtraction the result is no longer than the longer operand, so it fits unless the right-hand side has more limbs than the place has room for. The right-hand side may even be the place itself, set a += a, since each limb of the operand is read before the same limb of the place is written.

For multiplication and division, no. Each limb of a product depends on many limbs of both operands, which are read all the way to the end, so the product is built in a separate buffer that then replaces the place's. Division is the same. *= and /?= are still a single instruction, but they allocate just as the binary forms do.

Compound assignment semantics

The semantics of compound assignment ops are exactly as if they had been written as an assignment of the binary operator, so the following are observably the same in all respects, including error handling.

var a: int = 10_000_000_000
let b: int = 20_000_000_000
set a = a + b
debuglog (a, b)

var a: int = 10_000_000_000
let b: int = 20_000_000_000
set a += b
debuglog (a, b)

And fixed ints with checked ops:

var a: u8 = 10
let b: u8 = 20
set a +!= b
debuglog (a, b)

b is evaluated, a is read, then assigned a plus the right-hand side value. If an error occurs, the code returns early and a is unchanged. a can be both immutably and mutably borrowed on the right-hand side.

var a: int = 10_000_000_000
let b: int = 20_000_000_000

fun double(ref n: int): int
  ret n + n
end fun

set a += b + double(ref a) // Immutable borrow and read.
debuglog (a, b)

var a: int = 10_000_000_000
let b: int = 20_000_000_000

fun sneaky_edit(mut a: int): int
  set a = 1
  ret 5
end fun

set a += b + sneaky_edit(mut a) // Mutable borrow and write.
debuglog (a, b)

Because the right-hand side is evaluated first, the second example adds to the 1 that sneaky_edit wrote, not to the original a.

Adding compound assignment operators, lowered through to compound IR and bytecode ops, immediately reduced time spent on the sum benchmark by about 45% on every engine (IR, bytecode, JIT, AOT), mostly by reducing bigint reallocation in the loop.

But not completely eliminating it: i@ still widens the counter into a freshly allocated int every iteration. More on that later.

Compound assignment to index projections

Writing this explanation and doing the implementation (having an LLM do the implementation...) brought yet more dogfooding wins: the actual existing implementation did not have a sensible evaluation order for the left-hand and right-hand side of an assignment. In most prior cases this didn't matter since they were simple without potential side-effects. But that wasn't true for assignment to index projections.

Compound assignment works on any place set can write, including elements reached through ? and ! index steps:

set counts[w]! += 1
set grid[r]?[c]? *= 2
set rows[i]!.total +!= amount

The place is evaluated once, so each key is computed and each lookup done a single time, and the update happens in the element where it lies. A bare map index, set m[k] += 1, is rejected: a set on a bare index means upsert, and it isn't obvious that's desirable nor what upserting means for every compound op (start from zero? is that useful for *=?).

The old order reached the place first, then evaluated the right-hand side. Reaching an element means checking its index and taking a reference to it, so a right-hand side that grew the collection reallocated it out from under that reference:

set xs[0]?.n += grow(mut xs) // Wrote through a freed buffer.

Plain set xs[0]?.n = grow(mut xs) had the same use-after-free; compound assignment just made it common. Now every set, plain or compound, evaluates in one order:

1. the right-hand side,
2. the index keys, left to right,
3. the lookups that reach the place, which run none of the program's code,
4. the write, or the compound op.

All of the program's own code runs before any reference into the place exists, so the example above writes into the grown list, and an index the right-hand side pushes into range is in range when it's checked.

Some consequences: the right-hand side's effects happen even if a lookup then fails; when both could fail, the right-hand side's error wins; and what the right-hand side moves is gone by the time the place is reached, so set m[k]? = k is a use-after-move and needs k@.

Datalove's current math ops

For reference, here is the full set of arithmetic operators as of today, and their compound forms. Each compound op takes exactly the types its binary form does.

Bigints can't overflow, so they get the bare +, - and *, but division can still divide by zero, so it is checked. Floats follow IEEE 754 and get the bare operators throughout. Fixed-width integers get only checked operators. The ? forms return none from the enclosing function on overflow or division by zero, and the ! forms return an error, so they require the function to return an option or a result, respectively. A compound op that fails leaves its place unchanged.

Wrapping and saturating arithmetic is available as library functions, like u8.add_wrapping and u8.add_saturating.

Future work

Several other near-term language additions were suggested by this round of profiling. As mentioned previously, math binops require both operands to have the same type, and when the type is a bigint, that means they both need to be on the heap.

fun sum_to(limit: u32): !int
  var total: int = 0
  var i: u32 = 1
  loop while i <= limit
    set total += i@
    set i +!= 1
  end loop
  ret ok total
end fun

Often one side of these ops doesn't need to be large: the i in the above is a u32 but must be widened to int, an expensive thing to do in a loop. To make that cheap I could give bigints a small-size optimization, but I'm disinclined to add the representational complexity and additional branching, and as always I prefer that performance not rely on optimizations that aren't implied by the surface language. I'm instead thinking of having binops support limited mixed-type operations, where the two sides can have different types, where one losslessly converts to the other (I suspect Datalove will also eventually get auto-widening of numerics in general, at least as a mode that is active for interactive/script use).

Secondly, this round of work revealed that the standard library widely uses a less-than-ideal pattern for loop counters:

fun repeated<T>(ref elem: T, n: index): [T]
  var built: [T] = []
  var i: index = : index / 0
  loop while i .< n
    call push(mut built, elem@)
    set i = icall add_wrapping_index(i, : index / 1)
  end loop
  ret built
end fun

This maintains an index-type loop induction variable i and uses wrapping addition to increment it with the add_wrapping_index intrinsic. The main problem here is the wrapping math to avoid handling overflow. It does this to avoid the +? operator, a checked operator that requires the containing function to return an option type. It's provably not possible for the index to overflow here, but needing to use explicit wrapping for loop iteration leaves lots of room for accidental errors, the reader needing to think hard about the meaning of every instance it occurs.

While I expect to add built-in saturating and wrapping math ops, +| and +% a la Zig, the solution here is a looping construct that iterates collections. So that's probably coming pretty soon.

Writing these has shaken out many bugs at all levels of the language and compiler. Write some examples; run them through the compiler; fix the bugs. Pretty happy with that.

I have a runtime with an interface that is accessible to the compiler, and we have intrinsics via the icall keyword, but Datalove functions can't themselves call native code.

Typically at this stage one might introduce a simple mechanism that allows std to declare the existance of foreign functions, have the compiler always link in the runtime so std can call those functions. The runtime functions needed by std just exist in the ether.

Then down the road we need to provide a robust mechanism for user-level packages other than std. I don't want std to be special, to whatever extent that is possible, so I'm just going to design the general mechanism and have std use it from the start.

I call that mechanism native riders, each package able to declare a Rust crate that rides along with it wherever it goes.

The model

A rider is a Rust cdylib/staticlib crate that belongs to a datalove package. One crate per package, for simplicity. It has two parts: an interface file (rider.dli) declaring native function signatures in datalove syntax, and a Rust crate implementing them.

// sys/std/rider.dli
native fun string_len(ref self: string): index
native fun string_split(ref self: string, ref delimiter: string): [string]
native fun string_parse_int(ref self: string): ?int

Modules opt in with require rider and then import native functions like they'd import from any other module:

require rider std
import std.string_split

fun split(ref self: string, ref delimiter: string): [string]
  ret string_split(self, delimiter)
end fun

require in Datalove is the syntax the compiler pre-scans to determine inter-module dependencies, distinct from import, which has only local effect. In Datalove all inputs are known upfront, including the module graph and all native interfaces.

We support only Rust for native code -- we can mostly count on being able to find and compile Rust crates, and datalove is written in Rust anyway.

Datalove code isn't otherwise exposed to the nativeness of native functions. After the require and import, they just look like functions from a module.

The calling convention

Everything is passed by pointer for simplicity. Easy to expand the ABI later. Every native function receives a runtime handle, and for each declared argument, a pointer and a type descriptor. Return values go through an out-parameter.

#[no_mangle]
pub extern "C-unwind" fn dlr_std__string_split(
    rt: *mut u8,
    s_ptr: *const u8, _s_td: *const u8,
    delim_ptr: *const u8, _delim_td: *const u8,
    out: *mut u8, out_td: *const u8,
) -> u8 {
    // ...
}

The type descriptors on every argument will matter once we have generics: they carry element types, field layouts, everything needed for polymorphic dispatch. For now they're mostly there for consistency with the existing runtime ABI.

The symbol naming convention is dlr_{rider}__{function}. The compiler generates matching declarations.

Execution backends

The interpreter loads the rider's shared library via dlopen, looks up each dlr_* symbol, and registers them in a native function table. When it hits a call to a native code unit it marshals arguments to the C ABI and calls through the function pointer.

The JIT registers native symbols with the Cranelift JIT engine. Each gets a local trampoline using call_indirect with the absolute address baked in — relocations are already greater than the 2GB limit imposed by Cranelift so can't jump directly.

The AOT compiler treats native functions as imported symbols with Linkage::Import. The rider's static library gets linked alongside the runtime at the final link step.

Future work

So that's all implemented enough for std to work, and now std contains a string module with all the basic APIs you might expect.

Most other standard library functions need generics so that's the next big task.

fun scale(const factor: int, x: int): int
    ret x * factor
end fun

const TWO: int = 2
const TEN: int = 10

let a = scale(TWO, 5)   // specialized for factor=2
let b = scale(TEN, 5)   // specialized for factor=10

The compiler evaluates const arguments at compile time, generates specialized function variants, and rewrites call sites to target the correct specialization.

Specialization plus const evaluation work together naturally.

fun scale(const factor: int, x: int): int
    const calculated_factor = calculate(factor)    // compile-time evaluated
    ret x * calculatod_factor
end fun

This is mostly just architectural work to make sure it all fits together. I'm hopeful this simple specialization will naturally extend to generic type parameters.

I'm experimenting with a compromise monomorphization strategy that relies on whole-program compilation to see all instantiations: instead of generating a function for every instantiation (or every set with the same "shape"), I am generating one function that branches on a single discriminator that represents the set of instantiations. Curious if it is easier on codegen and linking than typical monomorphization, but not super hopeful.

For now const arguments must be named const bindings; they don't support abitrary const expressions. Implementation simplicity. Could be expanded in the future, but value implementation simplicity.

Our CTFE and const specialiation strategy right now is friendly to the compiler pipeline, no ouruboroses here. We rely on two properties: all const statements can be evaluated as if they are let statements; all functions with const parameters can be evaluated as if they were non-const. This is super sweet because we can turn on-off const evaluation and specialization and do differential testing against both modes (we rely very heavily on differential testing); and also we can structure const evalutation and specialization as "add-on" passes, relatively decoupled from the rest of the pipeline.

The lowering pipeline: first we lower with const statements lowering to let statements, and with calls to specializable functions as ComptimeCall IR instructions (which the interpreter and cranelift can lower exactly the same as regular calls if needed), recording the sites of both; then we evaluate all const statements and patch the IR, then specialize the functions and patch the IR. Downside of patching instead of re-lowering is that we must run a dead-code elimination pass afterwards.

This basic design is Zig-inspired, but I'm not decided yet if it will extend to full generics or not — I will probably experiment with Zig's first-class, kinda-dependent types, but think I prefer more traditionally-constrained type parameters for ease of reasoning.

Now most bindings and all temporaries are tracked precisely and the IR describes exactly when to drop them.

All let statements are precise. All var statements are dynamically tracked.

out parameters use dynamic tracking to avoid dropping before initialization.

More analysis could improve precise tracking, at the cost of more analysis. This is a nice and simple baseline for correctness and efficiency.

Because functions are pure, deterministic with no side effects, and we have a working interpreter, and no pointers, it's almost trivial to evaluate functions at compile time.

Getting this feature in early to inform the architucture. Initial feature const-evals all const statements, in scripts, script functions, modules, and module functions. After that works fully I'll try to add const parameter specialization.

The simple specialization that requires will hopefully inform full generics.