MagickCache: an Efficient Image Cache

The Magick cache is a work in progress. Do not use the cache in production services until version is at least 1.0.0. It is currently 0.9.2. The MagickCache requires ImageMagick version 7.1.0-0 or above.

The MagickCache provides secure methods and tools to cache images, image sequences, video, audio or metadata in a local folder. Any content is memory-mapped for efficient retrieval. Additional efficiencies are possible by retrieving a portion of an image. Content can persist or you can assign a time-to-live (TTL) to automatically expire content when the TTL is exceeded. MagickCache supports virtually unlimited content upwards of billions of images making it suitable as a web image service.

The MagickCache works in concert with ImageMagick. Download the MagickCache and install. You will now want to create the cache and populate it with images, video, and associated metadata.

Create a MagickCache

You will require a place to store and retrieve your content. Let's create a cache on our local filesystem:

    $ magick-cache -passkey passkey.txt create /opt/magick-cache

Where `passkey.txt` contains your cache passkey. Don't lose your passkey. Without it, you will be unable to identify, expire, or delete content in your cache.

You only need to create a MagickCache once. You can, however, create more than one MagickCache with different paths.

Once the MagickCache is created, you will want to populate the cache with content that includes images, video, audio, or metadata.

Put content in the MagickCache

Let's add a movie cast image to our newly created cache:

    $ magick-cache put /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson 20210508-rebecca-ferguson.jpg

Note, the image identifier is an IRI composed of project/type/resource-path. In this example, the project is movies, type is image, and the resource path is mission-impossible/cast/rebecca-ferguson. The path uniquely identifies a cache resource. Two different images cannot be stored with the same resource path. Instead use something like mission-impossible/cast/20210508-rebecca-ferguson-1 and mission-impossible/cast/20210508-rebecca-ferguson-2.

Now, let's set a resource passkey and the time to live to 2 days. Anytime after the second day, the image is automatically deleted with the expire function. To get, expire, or delete the image, you will need to use the same resource passkey:

    $ magick-cache -passkey passkey.txt -ttl "2 days" put /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson 20210508-rebecca-ferguson.jpg

Where passkey.txt contains your resource passkey. Don't lose your passkey. Without it, you will be unable to get, identify, expire or delete resources you own.

The resource passkey ensures only you and the cache owner can access your image. To prevent the cache owner from viewing its content, scramble it with:

    $ magick-cache -passkey passkey.txt -passphrase passphrase.txt -ttl "2 days" put /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson 20210508-rebecca-ferguson.jpg

Note, blobs and metadata are stored in the cache in plaintext. To prevent snooping, scramble the content before you store it in the cache.

Get content from the MagickCache

Eventually you will want retrieve your content, let's get our original cast image from the cache:

    $ magick-cache -passkey passkey.txt get /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.png

Notice the original image was put in the cache in the JPEG format. Here we conveniently convert it to the PNG format as we extract the image.

The -extract option is useful when retrieving an image. To extract a portion of the image, specify tile width, height, and offset:

    $ magick-cache -passkey passkey.txt -extract 100x100+0+0 get /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.png

To resize instead, do not specify the offset:

    $ magick-cache -passkey passkey.txt -extract 100x100 get /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.png

If your image is scrambled, provide the passphrase to descramble it:

    $ magick-cache -passkey passkey.txt -passphrase passphrase.txt get /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson rebecca-ferguson.png

Delete content from the MagickCache

We can explicitly delete content:

    $ magick-cache -passkey passkey.txt delete /opt/magick-cache movies/images/mission-impossible/cast/rebecca-ferguson

or we can delete all cast images that have expired (exceeded their respective time to live), try this comand:

    $ magick-cache -passkey passkey.txt expire /opt/magick-cache movies/images/mission-impossible/cast

Identify the MagickCache content

Perhaps you want to identify all the content you own:

    $ magick-cache -passkey passkey.txt identify /opt/magick-cache movies/images/mission-impossible/cast
movies/images/mission-impossible/cast/rebecca-ferguson[1368x912] 406B 1:0:0:0 2021-05-30T17:41:42Z
identified 1 resources

Each entry includes the IRI, image dimensions for images, the content extent in bytes, time to live, whether the resource is expired (denoted with a `*`), and the creation date.

Others can store content in the cache along side your content. However, their content is unavailable to you. You cannot get, identify, expire or delete content that you do not own as determined by your secret passkey.

The MagickCache owner can get, identify, expire, or delete all the content, including content you own, with this command, for example:

    $ magick-cache -passkey passkey.txt identify /opt/magick-cache /

Note, expired reesources are annotated with an asterisks.

MagickCache is not just for Images

In addition to a type of image, you can store the image content in its original form, video, or audio as content type of blob or metadata with a content type of meta:

    $ magick-cache -passkey passkey.txt put /opt/magick-cache movies/blob/mission-impossible/cast/rebecca-ferguson 20210508-rebecca-ferguson.mp4

or

    $ magick-cache -passkey passkey.txt put /opt/magick-cache movies/meta/mission-impossible/cast/rebecca-ferguson 20210508-rebecca-ferguson.txt

Images must be in a format that ImageMagick understands. Metadata must be text. Blobs can be any content including images, video, audio, or binary files.

Delete a MagickCache

The MagickCache owner can completely delete all the content within a cache:

    $ magick-cache -passkey passkey.txt delete /opt/magick-cache /

Be careful, after this command, your cached content is irrevocably lost.

Security

MagickCache security is not crytographically strong. Instead it generates a unique hash for each resource ensuring the resource ID cannot be discovered. A resource is accessible to both the user of the cache and the cache owner provided they can present their respective passkeys. They are also accessible to anyone with sufficient privileges to access the MagickCache disk path.