Storage¶

Added in version 0.5.

Changed in version 0.6:

The harp_apps.storage application was previously named harp_apps.sqlalchemy_storage.
The blob storage system was added, and now supports both SQL or Redis backends.
the storage.type was removed, because there was only one valid value.

The harp_apps.storage application implements various storage systems for your proxies to use.

There are two different storage kinds:

Transaction/Core storage: Currently SQL-only, the core storage will handle persisting the transactions and messages, along with a few other utilities like users, tags, flags … The core storage implements the IStorage interface.
Blobs storage: The blobs storage is a separate storage system that stores the binary content of the messages. It is used to store the headers and the body of the requests and responses. The blobs storage implements the IBlobStorage interface. The blob storage is content-addressable, so that multiple indentical blobs are stored only once.

By default, blob storage will use the same SQL database as the core storage, but you can configure it to use a different underlying system (for now, the only alternative is Redis, but the simple interface allows for easy addition of new backends for blobs).

Warning

Recording here is best-effort. Transactions passing through HARP are recorded, including requests carrying Cache-Control: no-store, but under load the storage worker sheds message detail first and then whole transactions rather than slowing traffic down. Do not treat this store as a complete account of what crossed the proxy. See What HARP retains, and what a caller can do about it for what is measured, and #945 for the measurements.

In 0.10 a caller’s Cache-Control: no-store binds the cache and does not affect what is recorded here. Whether a caller may suppress payload recording is decided in #927 and lands in 0.11, so do not build on the current behaviour in either direction.

Choosing what is not recorded is currently the operator’s decision, made with the rules application and the Transaction Markers below.

Core storage¶

The core storage leverages SQLAlchemy to store the transactions in a SQL database.

We officially support the following SQL databases (although any async SQLAlchemy-supported database should work, they are not included in the test suite and may bring unexpected behaviours):

SQLite (config-less default, do not use for production), using aiosqlite driver.
PostgreSQL (our favorite choice for production), using asyncpg driver.
MySQL / MariaDB, using aiomysql or asyncmy driver.

Configuration¶

Storages are configured under the storage key of the configuration, and the settings implementation is backed by StorageSettings.

The main configuration option you’re interested in is storage.url, which configure the target SQL server.

PostgreSQL example (asyncpg driver):

storage:
  url: postgresql+asyncpg://user:password@localhost:5432/harp

MySQL/MariaDB example (aiomysql driver):

storage:
  url: mysql+aiomysql://user:password@localhost:3306/harp

MySQL/MariaDB example (asyncmy driver):

storage:
  url: mysql+asyncmy://user:password@localhost:3306/harp

SQLite example (aiosqlite driver):

SQLite is the default storage engine because requires no configuration. Do not use it in production. It is slow and has concurrency issues. It is only suitable (and handy) for development and testing.

storage:
  url: sqlite+aiosqlite:///harp.db

Blob storage¶

Without configuration, the blob storage will use the same SQL database as the core storage. For intensive production use, we suggest you switch the underlying implementation to a Redis-based one. The test suite only runs against the official redis docker image, but we’ll add tested support for alternatives, like KeyDB or Valkey in the future.

Configuration¶

To setup a redis-based blob storage, you can add the following to your configuration:

Redis example:

storage:
  blobs:
    type: redis
  redis:
    url: redis://localhost:6379/0 # this is the default, only set to override

Migrations¶

By default, migrations are run every time you start HARP, to ensure the database schema is always up-to-date. On that default, upgrading HARP and starting it is enough, and there is nothing to run by hand.

You can disable this behaviour by setting storage.migrate to false in your configuration. Applying migrations then becomes yours to do, with harp-proxy db:migrate up head, before starting the new version. A schema older than the running code is not a degraded mode: 0.10 widened blobs.id for cache keys, and against a database still holding the narrow column, any response the cache would store fails with a 500 rather than falling back to a cache miss.

Migrations are implemented using Alembic.

Resetting¶

If you want to reset the storage content, you can run the harp-proxy db:reset command. It will drop all tables, then run the migrations again (if enabled), so that you get an empty and up-to-date database schema.

You must provide your configuration files and configuration arguments to this command, like you’d do with the harp-proxy start command, so that it points to the right storage.

harp-proxy db:reset -f /path/to/config.yml

or…

harp-proxy db:reset --set storage.url=...

Logging¶

To instruct SQLAlchemy to log all queries, set LOGGING_SQLALCHEMY=INFO in your env.

LOGGING_SQLALCHEMY=INFO harp-proxy start ...

It works with all HARP commands, not just start.