Atomic Store: Crash-Safe, Multi-File Commits in a Single C Library

5 min read

At Hamkee, we build high-performance systems where data integrity is a non-negotiable requirement. A fundamental challenge in Unix and Linux environments is the nature of file system atomicity. While the POSIX rename() system call provides a robust guarantee for atomically updating a single file, this guarantee does not extend to operations involving multiple related files. A crash or power loss during a multi-file update can lead to a corrupted or inconsistent state.

To solve this critical problem, we developed Atomic Store, a minimal, dependency-free C library designed to provide true atomic commits for multiple logical files.

This library encapsulates our philosophy of building simple, robust tools that solve complex, real-world problems. It is designed for scenarios where a full database like SQLite is overkill, but the risk of data corruption is unacceptable.

The Multi-File Consistency Gap

In many applications, system state is distributed across several configuration or data files that must be updated in concert. Consider a typical application managing user data, world state, and inventory:

  • player.dat
  • world.dat
  • inventory.dat

A standard approach to saving progress would involve writing to each file sequentially:

  1. Write new content to player.dat
  2. Write new content to world.dat
  3. Write new content to inventory.dat

The critical vulnerability lies between these steps. If the system crashes after player.dat is updated but before inventory.dat is written, the files are now out of sync. The player’s state might reflect a level-up, but their inventory remains unchanged, leading to an inconsistent and potentially unrecoverable state.

This is the multi-file consistency gap, a problem that standard POSIX I/O does not solve.

Our Solution: A Transactional, Append-Only Store

We designed Atomic Store to close this gap by abstracting file management into a transactional, crash-safe paradigm. Instead of directly manipulating individual files on the filesystem, Atomic Store treats a collection of logical files as a single unit, committing them together within a snapshot.

The core mechanism is an append-only binary file—the store.

How it works

  • Transaction Begin
    An application signals its intent to update by beginning a transaction. This transaction is built in-memory, based on the last known-good state.

  • Staging Changes
    The application adds, updates, or removes logical files within the transaction. These operations modify the in-memory snapshot, not the store file on disk.

  • Atomic Commit
    When the application commits the transaction, Atomic Store serializes the entire snapshot—including all file paths, content, and metadata—into a single binary blob.

  • Checksum and Write
    A checksum is calculated for the entire snapshot to ensure its integrity. The snapshot blob, followed by a footer containing the checksum and its size, is appended to the end of the store file.

  • Durability
    A call to fsync() (or an equivalent) explicitly flushes all buffered data to physical storage, ensuring the write survives power loss.

Upon opening the store, the library reads the file from the end, searching for the last valid footer. It uses the footer to locate the snapshot, verifies its checksum, and loads it as the definitive state. Any partial or corrupt data from an interrupted write is ignored.

The result: the application always starts from a consistent state.

Architecture and API Design

The simplicity of the concept is reflected in its API, exposed via atomic_store.h. We focused on a minimal and intuitive interface that encourages correct usage.

Core API functions

  • aw_store_open(path, create_if_missing)
    Opens the store file and recovers the last valid snapshot.

  • aw_tx_begin(store)
    Creates an in-memory transaction used as a staging area.

  • aw_tx_add_buffer(tx, logical_path, data, len)
    Adds or replaces a logical file in the transaction.

  • aw_tx_commit(tx)
    Executes the atomic commit, performing serialization, write, and fsync().

  • aw_store_get(store, logical_path, &data, &len)
    Retrieves a logical file from the active snapshot.

This design enforces clear transactional boundaries and the all-or-nothing principle. The append-only format avoids complex filesystem manipulations, making the system easier to reason about, audit, and verify.

A Practical Use Case: High-Integrity IoT Device Configuration

Consider an embedded Linux system, such as a fleet of IoT devices deployed in the field. These devices often rely on multiple configuration files:

  • network.conf — network endpoints, credentials, protocols
  • sensors.json — sensor calibration, sample rates, enabled features
  • firmware.info — firmware metadata and update policies

These files are frequently updated remotely.

The risk

A remote update script might overwrite network.conf and then sensors.json. If a power flicker or network interruption occurs between these writes, the device is left in a split-brain state: the new network configuration points to a server expecting a data format the old sensor configuration cannot produce.

The device may become inoperable, requiring costly manual intervention.

The Atomic Store solution

By integrating Atomic Store, the update agent performs the operation safely:

// Simplified C example for an IoT update agent
void apply_remote_update(aw_store_t *store,
                         const char *new_net_conf,
                         const char *new_sens_conf) {

    // Begin a new transaction based on the current stable state
    aw_tx_t *tx = aw_tx_begin(store);
    if (!tx) {
        // Handle error: could not start transaction
        return;
    }

    // Stage both configuration changes within the transaction
    aw_tx_add_buffer(tx, "network.conf", new_net_conf, strlen(new_net_conf));
    aw_tx_add_buffer(tx, "sensors.json", new_sens_conf, strlen(new_sens_conf));

    // Commit both changes in a single, atomic, durable operation
    if (aw_tx_commit(tx) != 0) {
        // Handle error: the commit failed, but the old state remains intact
    }
}

With this approach, either both files are updated together, or neither is. The on-disk state is never left partially modified, transforming a high-risk operation into a predictable and resilient one.

Conclusion

Atomic Store provides a targeted, robust, and auditable solution to the problem of multi-file atomicity in environments where it is not natively guaranteed. By abstracting file operations into a transactional, append-only log, it eliminates the risk of data corruption from partial writes and ensures consistent application state across crashes. Atomic Store was developed by the engineering team at Hamkee, where we specialize in high-performance Unix/Linux solutions. This project is a direct result of our experience building systems where data integrity is not a feature, but a fundamental requirement. We invite you to explore the repository Atomic Store, examine the implementation, and consider Atomic Store for your next project where crash-safe state management is paramount.