@vltpkg/cache-unzip

This is a script that can be run as a detached background process to un-gzip any cached response bodies in the vlt cache, and to explode cached tarballs into the global store.

Usage · Global Store · Why Do This?

Usage

Whenever you get a cache entry with a gzipped body, tell this module about it.

import { register } from '@vltpkg/cache-unzip'
import { Cache } from '@vltpkg/cache'

const cache = new Cache({ path: cachePath })

// later...

const response = get_response_cache_entry_somehow()
cache.set(myKey, response.encode())

// unzip it after this process is done
if (response.isGzip) {
  register(cachePath, myKey)
}

On process exit, these registered keys will be passed as arguments to a detached deref'ed vlt-cache-unzip process. So, the main program exits normally, but the child process ignores the SIGHUP and keeps going until it's done. The next time that cache entry is read, it won't have to be unzipped. It exits 1 only on no path, a corrupt gzip body or a failed explode.

Global Store

Pass the global store root as a third argument to register(), and optionally the tarball's integrity, for entries that may only be cached under it:

register(cachePath, myKey, storeRoot, integrity)

When VLT_STORE_LINKER is auto, hardlink or copy, the child also explodes each tarball entry with a sha512 integrity header into <storeRoot>/<integrity-hex>/, with its sidecar index next to it at <integrity-hex>.json. Unset or unpack: nothing is written there.

• Existing entries are skipped, unless their sidecar is missing or invalid: then they are redone. Bad tarballs are skipped too.
• Entries are built in <storeRoot>/.tmp/ and renamed into place, sidecar first. If another process wins the rename, its entry is kept. Leftovers older than one hour are removed.
• Explode runs first. Entries it writes stay gzipped: installs link them from the store. The rest are un-gzipped: store off, already in the store, not a tarball, no sha512 integrity, failed. So a gzipped entry registered again by an install that read it instead of linking it (say, after switching to store-linker=unpack) is un-gzipped.
• VLT_CACHE_UNZIP=0 never un-gzips. VLT_CACHE_UNZIP=1 un-gzips every entry before exploding.
• VLT_CACHE_EXPLODE_CONCURRENCY sets how many entries are read at once (default 1).
• NODE_DEBUG=vlt prints a summary to stderr, seen only when the child is run by hand (vlt ignores its output): keys read, exploded and un-gzipped, and with the store on, entries written, skipped (already there), ignored (missing or not a tarball), failed, bytes, ms.

With the global store on, the child runs at the lowest CPU priority.

Why Do This

Because it's faster to not have to decompress the same content more times than necessary.