Even though Bitcoin is technically Turing complete, in practice implementing a non-trivial computation on Bitcoin is borderline impossible.¹ Only after Ethereum was introduced, smart contracts entered common parlance. But even recent blockchains offer execution environments much more constrained than those available on desktop computers, servers or even home appliances such as routers or NASes.

Things are no different on Solana. I’ve previously discussed its transaction size limit, but there’s more: Despite using the ELF file format, Solana Virtual Machine (SVM) does not support mutable global state.

Meanwhile, to connect Solana and Composable Foundation’s Picasso network, I was working on an on-chain light client based on the tendermint crate. The crate supports custom signature verification implementations via a Verifier trait, but it does not offer any way to pass state to such implementation. This became a problem since I needed to pass the address of the signatures account to the Ed25519 verification code.²

This article describes how I overcame this issue with a custom allocator and how the allocator can be used in other projects. The custom allocator implements other features which may be useful in Solana programs, so it may be useful even for projects that don’t need access to mutable global state.

The problem

The issue is easy to reproduce since all that’s needed is a static variable that allows internal mutability. OnceLock is a type that fits the bill.

fn process_instruction(
    _: &Pubkey,
    _: &[AccountInfo],
    _: &[u8],
) -> Result<(), ProgramError> {
    use std::sync::OnceLock;
    #[no_mangle]
    static VALUE: OnceLock<u32> = OnceLock::new();
    VALUE.set(42).map_err(ProgramError::Custom)
}

Compilation of the above Solana program succeeds with no issues, but trying to deploy it fails with an ELF error as seen below:

$ cargo build-sbf
⋮
$ solana program deploy target/deploy/global_mut_test.so
Error: ELF error: ELF error: Found writable section (.bss.VALUE) in ELF, read-write data not supported

The Solana CLI tool makes it quite clear what the issue is. Even though, without the #[no_mangle] annotation, the error message is less informative: ‘Section or symbol name .bss._ZN15global is longer than 16 bytes’ (with _ZN15global depending on the smart contract’s and global variable’s name and type).

The idea

Global (or static) variables are declared outside any function, making them accessible throughout the program. Their memory address is fixed and any part of the code can reference them without needing their address explicitly passed. From the perspective of programming languages like C or Rust, this address remains unchanged after compilation.

The memory layout of Solana programs is fixed, with the heap always starting at address 0x3 0000 0000 (and a convenient HEAP_START_ADDRESS constant holds that value). Since this address is known by the program, it should theoretically be possible to place a global variable there. This can be tested using the following simple Solana program:

fn process_instruction(
    _: &Pubkey,
    _: &[AccountInfo],
    _: &[u8],
) -> Result<(), ProgramError> {
    let global = unsafe {
        &mut *(HEAP_START_ADDRESS as *mut usize)
    };
    *global = 42;
    msg!("Testing global: {}", *global);
    Ok(())
}

If the global variable worked as intended, calling this smart contract would produce a log message displaying its value, i.e. 42. However, when the program is executed, the following logs appear:

Program … invoke [1]
Program log: Error: memory allocation failed, out of memory
Program … consumed 230 of 200000 compute units
Program … failed: SBF program panicked

The failure is because the heap is managed by contract’s allocator and by modifying the start of the heap, the program overwrites information about available free space that the allocator maintains.³ The msg! macro attempts to allocate a temporary buffer but fails because the allocator’s state has been corrupted.

The solution

The solution is to replace the allocator with one that understands the need for mutable global state. Solana programs can declare custom global allocators, and using the one defined in the solana-allocator repository is sufficient. To do this, first add the dependency to Cargo.toml and enable the custom-heap feature:⁴

[dependencies.bytemuck]
version = "*"
optional = true

[dependencies.solana-allocator]
git = "https://github.com/mina86/solana-allocator"
optional = true

[features]
default = ["custom-heap"]
custom-heap = ["dep:bytemuck", "dep:solana-allocator"]

All mutable global variables used by the program must be collected into a single structure. This structure is then declared as global state managed by the allocator. The crate provides the custom_global macro, which automates this process:

// This does three things:
// 1. Defines a Global struct with specified fields.
// 2. Declares the custom global allocator with Global
//    object as global state.
// 3. Defines a global function which returns shared
//    reference to this Global object.
#[cfg(feature = "custom-heap")]
solana_allocator::custom_global!(struct Global {
    counter: core::cell::Cell<usize>
});

fn process_instruction(
    _: &Pubkey,
    _: &[AccountInfo],
    _: &[u8],
) -> Result<(), ProgramError> {
    // Call to global function can be used to
    // get a shared reference to the global state.
    let counter = &global().counter;
    counter.set(42);
    msg!("Testing global: {}", counter.get());
    Ok(())
}

Caveats

The approach has a few caveats and limitations. Since it’s not supported by the language, static declarations won’t work for mutable state, meaning all global variables must be declared in a single location. Additionally, any third-party crates that declare mutable global state will need to be patched. For example, I encountered one such case where the solution was simply to remove an unused dependency.

Similarly, this approach cannot be used inside library crates. For instance, if a Solana program can be compiled as a library (say by enabling a cpi feature), the library code cannot use any mutable static state.

Furthermore, the approach doesn’t work on non-Solana builds. Conditional compilation may be necessary to either remove code that accesses global state or modify it so that the global state is accessed differently depending on the target platform.

Another caveat is that all the variables are initialised to the all-bits-zero value. This means that the types of the variables must implement the bytemuck::Zeroable trait. Non-zeroable types can be handled by wrapping them in a MaybeUninit.

Lastly, due to Solana’s technical limitations (more about this below), the allocator effectively over-commits memory. Allocations never fail, even if they exceed the available heap space. The allocation failure error is deferred until the client tries to use the memory.

Additional features

On the flip side, the custom allocator has features not present in Solana’s default allocator. By default, Solana programs have access to 32 KiB of heap space, which can be increased on a per-transaction basis. However, the default allocator doesn’t use this additional space. The custom allocator works with arbitrarily large heaps but at the cost of over-committing memory. It’s extremely convoluted in Solana to get the size of the heap, so the allocator assumes there’s always free space (that is, it over-commits memory), deferring failures to when the client first tries to use the memory.

Secondly, unlike Solana’s default allocator, which doesn’t free memory, the custom allocator opportunistically frees memory. If an object is allocated and then deallocated (without any allocations in-between), its memory will be freed and reused. Furthermore, depending on the alignment and size of allocations, if objects are freed starting from the last one that was allocated, multiple objects may be freed at once. This is especially helpful for code that uses temporary buffers (e.g. msg macro or Anchor events).

To use the allocator without the global state feature, you can use the solana_allocator::custom_heap macro.