Storehouse.Archive (Storehouse v0.2.0)

Copy Markdown View Source

An archive of files, stored ones and the app's own, made as a stream: an export of an account's files, a folder to download, a backup. Nothing is written to the disk and no more than a chunk is in memory, however large the archive.

entries = [
  {"README.txt", "Your photographs, as of 23 October 2026.\n"}
  | for(photo <- photos, do: {"photos/" <> photo.name, {:stored, photo.key}})
]

# Its size before it is made: "3.2 GB, 1,204 photographs".
{:ok, bytes} = Storehouse.Archive.size(entries)

# Into the store, out of the request, then the app's function with it.
Storehouse.Archive.store_later(entries, "exports/42.zip",
  expires_in: 86_400,
  then: {MyApp.Exports, :ready, [account.id]}
)

Entries

{name, content}, the name a path in the archive, the content one of:

  • {:stored, key} - a stored file, read a chunk at a time from :adapter, by the size the store gives.
  • a binary.
  • {:call, {module, function, args}} - a function that gives the content, a binary or an enumerable of iodata, called as the archive is made: what the app makes for it (a JSON of its records), and what a job can keep (store_later/3).
  • {:stream, enumerable} - iodata, as the archive is made.

A name is a relative path of UTF-8, with no .., no backslash, and no leading /: the archive never puts a file outside the folder it is opened in. Two entries never have one name. A stored file must be the size it was when the archive began: one that changed stops it.

Formats

:format chooses it, else config :storehouse, :archive_format:

Compression

:compression is :auto (the default): each file that shrinks is compressed (text, CSV, JSON, SVG, documents), and each that does not is stored as it is (photographs, video, audio, other archives, PDF). Both give the same bytes back, compression loses nothing: a photograph only takes ten to thirty times longer to compress, to save 1 to 3 %. :compress compresses every file; :none, none.

Splitting

max_bytes: makes several archives, each a whole one under the size, that each open alone (export-1.zip, export-2.zip, …): for a mailbox's limit, a FAT32 drive's 4 GB. A file larger than the size alone gives {:error, {:too_large, name}}.

Out of the request

Archiving 2 GB takes as long as reading 2 GB from the store and writing 2 GB back: a minute or two, not a request's time. store_later/3 makes it in a job of the app's runner (Storehouse.Runner: Oban's, kept through deploys and tried again), then calls the app's function with it.

Telemetry

store/3 is a span, [:storehouse, :archive, ...], with the :key, the :format, the number of :entries, and the :bytes made.

Summary

Types

entry()

A file in the archive: its name, and its content.

Functions

size(entries, opts \\ [])

The archive's size, before it is made, with the stored files' sizes asked of the store (not read)

store(entries, key, opts \\ [])

The archive, written into the store under key as it is made (Storehouse.Adapter.stream_open/2: S3's multipart upload, a file on the disk): a Storehouse.File, with its size and checksum. With :max_bytes, a list of them, the parts' keys key's with -1, -2, … before the extension.

store_later(entries, key, opts)

Makes the archive (store/3) out of the request, in a job of the app's runner (Storehouse.Runner), then calls the app's :then with {:ok, file} (or the list of parts): to email its link.

stream(entries, opts \\ [])

The archive as a stream of iodata, made as it is taken. Raises Storehouse.ReadError when the store fails halfway, and an ArgumentError when a file is not the size it was.

Types

entry()

@type entry() ::
  {String.t(),
   binary()
   | {:stored, Storehouse.Adapter.key()}
   | {:call, mfa()}
   | {:stream, Enumerable.t()}}

A file in the archive: its name, and its content.

Functions

size(entries, opts \\ [])

@spec size([entry()], keyword()) ::
  {:ok, non_neg_integer()} | {:about, non_neg_integer()} | {:error, term()}

The archive's size, before it is made, with the stored files' sizes asked of the store (not read):

  • {:ok, bytes} - exact, when nothing is compressed.
  • {:about, bytes} - with files compressed: the most it can be, its files' sizes as they are (a zip's or a tar's no larger).
  • {:error, :unknown_size} - with an entry of unknown size (a stream, a function).

Takes :adapter, :format, :compression and :password.

store(entries, key, opts \\ [])

@spec store([entry()], Storehouse.Adapter.key(), keyword()) ::
  {:ok, Storehouse.File.t() | [Storehouse.File.t()]} | {:error, term()}

The archive, written into the store under key as it is made (Storehouse.Adapter.stream_open/2: S3's multipart upload, a file on the disk): a Storehouse.File, with its size and checksum. With :max_bytes, a list of them, the parts' keys key's with -1, -2, … before the extension.

Takes the options of stream/2, and:

  • :expires_in - seconds after which it is deleted, under an expiring key (Storehouse.expiring_key/2): an export whose link lasts a day.
  • :max_bytes - the largest each archive is (see Splitting).

A store's error, or a file not the size it was, gives {:error, reason}, and nothing is left in the store.

store_later(entries, key, opts)

@spec store_later([entry()], Storehouse.Adapter.key(), keyword()) ::
  :ok | {:error, term()}

Makes the archive (store/3) out of the request, in a job of the app's runner (Storehouse.Runner), then calls the app's :then with {:ok, file} (or the list of parts): to email its link.

Storehouse.Archive.store_later(entries, "exports/42.zip",
  expires_in: 86_400,
  then: {MyApp.Exports, :ready, [account.id]}
)

A job keeps what it is given, so its entries are what can be kept: {:stored, key}, {:call, mfa}, and binaries of up to 1 MB; not {:stream, enumerable}, which {:call, mfa} gives instead. A :password is never kept: give {:call, mfa}, a function that gives it in the job. A store that fails makes the job try again.

Takes the options of store/3, :then (required), and :runner.

stream(entries, opts \\ [])

@spec stream([entry()], keyword()) :: {:ok, Enumerable.t()} | {:error, term()}

The archive as a stream of iodata, made as it is taken. Raises Storehouse.ReadError when the store fails halfway, and an ArgumentError when a file is not the size it was.

Takes :adapter, :format, :compression, :password (with a format that encrypts), :chunk_bytes (as for Storehouse.stream/2) and :modified_at, the time each file is dated (the default is now).