image#

The compressed::image<T> is a container of multiple Channel Struct, it maps these channels by channel name and can additionally store arbitrary image metadata. It also supports reading directly into the compressed::image<T> for very memory efficient (and fast) file loading (with file writes TBD).

Reading Images#

The compressed::image<T> class (or compressed.Image) provides a convenient read function which not only abstracts the file reads of multiple formats, but also reads the data in chunks before compressing leading to much greater performance and memory efficiency than doing this yourself. For details visit the benchmarks section.

Reading files this way additionally allows for automatic translation of the image metadata into the metadata field of the compressed image.

These read functions provide a variety of overloads for reading subimages, only specific channels as well as supporting scanline and tiled image formats.

c++

compressed::image<T> image = compressed::image<T>::read("path/to/my/file");

python

import compressed_image as compressed

image = compressed.Image.read(np.uint8, "path/to/my/file")

We can also do more advanced things such as specifying the subimage, channels and compression settings:

c++

// Note: this will convert to the specified dtype on-read, so it doesn't even have to be the dtype of the image!
compressed::image<T> image = compressed::image<T>::read(
                                "path/to/my/file" /* path */,
                                0 /* subimage */,
                                compressed::enums::codec::blosclz /* compression codec */,
                                5 /* compression level */
                            );

python

import compressed_image as compressed


image = compressed.Image.read(
            dtype = compressed.Image.dtype_from_file("path/to/my/file"),
            filepath = "path/to/my/file",
            subimage = 0,
            channel_names = ["R", "G", "B", "A"],
            compression_codec = compressed.enums.Codec.blosclz,
            compression_level = 5
        )

Using postprocesses#

Often times it is helpful to apply a postprocess to the image data directly on read. For example for color space conversions or to e.g. calculate statistics such as average, psnr etc. Since the image data is read first, and then compressed in chunks, we provide a callback to the read functions that operates on one of these read chunks before it is compressed. This allows for very efficient modification on read as it doesn’t have the overhead of having to de- and re-compress the image data at a later point.

Note

At the moment only the C++ api supports postprocess callbacks, although this is intended to come in a future release.

std::filesystem::path filepath = "image.exr";

    // Read an image file and apply a post-process which adds 1 to the pixel value for all RGB channels (0, 1, 2).
    auto postprocess = [](size_t channel_idx, std::span<T> chunk)
            {
                    if (channel_idx > 2)
                    {
                            return;
                    }

                    std::for_each(std::execution::par_unseq, chunk.begin(), chunk.end(), [](T& value)
                    {
                            value += 1;
                    }
            };

    auto img = compressed::image::read<uint8_t>(
            filepath,
            std::forward(postprocess),
            0, // subimage
            compressed::enums::codec::lz4, // compression_code
            5 // compression_level
    );

A note on color spaces#

The compressed::image<T> struct (and the compressed::channel<T> struct) have no notion of color spaces, color management etc. This is by design as it is meant to simply be a wrapper around a compressed buffer with image-specific utilities, but not a full on image processing library.

If you wish to encode color information on the compressed::image<T> you should do this via the metadata.

For example:

c++

compressed::image<T> image = ...;
auto& metadata = image.metadata();

// Set the source and destination space metadata, this is not managed by the compressed-image API, it also has no
// special meaning and any metadata is ignored entirely during parsing. This would just be for your convenience
metadata["OCIO_source_space"] = "linear";
metadata["OCIO_destination_space"] = "sRGB";

python

import compressed_image as compressed


image: compressed.Image = ...

// Set the source and destination space metadata, this is not managed by the compressed-image API, it also has no
// special meaning and any metadata is ignored entirely during parsing. This would just be for your convenience
image.set_metadata(
    {
        "OCIO_source_space": "linear",
        "OCIO_destination_space": "sRGB",
    }
)

Note

For performing color space transformations on read, it is recommended to do this a postprocess during the read function as that is the most efficient way of doing this (rather than having to decompress the data again). See Using postprocesses for more!

Image Struct#

c++

template<typename T> struct image : public std::ranges::view_interface<image<T>>#

Compressed Image representation with easy access to different channels. Internally functions very similar to an NDArray with the important distinction that the number of dimensions is fixed to be 3-Dimensional (width, height, channels). They are laid out in scanline order with each channel being its own distinct object which may have any size.

The image is stored in a non-resizable fashion so whatever the resolution was going into it, is what the image will be. To rescale or refit the image a new image has to be constructed.

The data is compressed in memory and we store it as part of a blosc2 super-chunk which is essentially a 3d array of super-chunk -> chunk -> block. Where having the block size fit into L1 cache and the Chunk size into L3 cache is desirable as each block can be handled by a single cpu core while the chunk fits well within shared L3 memory.

Public Types

using value_type = T #

Public Functions

image() = default#

image(image&&) = default#

image &operator=(image&&) = default#

image(const image&) = delete#

image &operator=(const image&) = delete#

~image() = default#

inline image(std::vector<std::span<const T>> channels, size_t width, size_t height, std::vector<std::string> channel_names = {}, enums::codec compression_codec = enums::codec::lz4, size_t compression_level = 9, size_t block_size = s_default_blocksize, size_t chunk_size = s_default_chunksize)#

Constructs an image object with the specified channels, dimensions, and optional compression parameters.

This constructor creates an image from a given set of channels. The channel names can optionally be specified. The image is then compressed using the provided codec and compression level.

Example:

std::vector<std::span<const uint8_t>> channels = ...;
compressed::image<uint8_t> my_image(channels, 1920, 1080, {"r", "g", "b"}, codec::lz4, 5);

Parameters:

channels – A vector of spans containing the image channels (each channel is a 2D array of pixel data). on construction these will be compressed thus the data can be safely freed after this function.
width – The width of the image in pixels.
height – The height of the image in pixels.
channel_names – (Optional) A list of channel names, must match the number of channels provided. If omitted or incorrect, channel names are ignored.
compression_codec – (Optional) The codec used for compression, default is codec::lz4.
compression_level – (Optional) The compression level, default is 9.
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Throws:

std::runtime_error – if a channel fails to be inserted.

inline image(std::vector<std::vector<T>> channels, size_t width, size_t height, std::vector<std::string> channel_names = {}, enums::codec compression_codec = enums::codec::lz4, size_t compression_level = 9, size_t block_size = s_default_blocksize, size_t chunk_size = s_default_chunksize)#

Constructs an image object with the specified channels, dimensions, and optional compression parameters.

This constructor creates an image from a given set of channels. The channel names can optionally be specified. The image is then compressed using the provided codec and compression level.

Example:

std::vector<std::vector<uint8_t>> channels = ...;
compressed::image<uint8_t> my_image(channels, 1920, 1080, {"r", "g", "b"}, codec::lz4, 5);

Parameters:

channels – A vector of vectors containing the image channels (each channel is a 2D array of pixel data).
width – The width of the image in pixels.
height – The height of the image in pixels.
channel_names – (Optional) A list of channel names, must match the number of channels provided. If omitted or incorrect, channel names are ignored.
compression_codec – (Optional) The codec used for compression, default is codec::lz4.
compression_level – (Optional) The compression level, default is 9.
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Throws:

std::runtime_error – if a channel fails to be inserted.

inline image(std::vector<compressed::channel<T>> channels, size_t width, size_t height, std::vector<std::string> channel_names = {})#

Constructs an image object with the specified channels and dimensions, optionally passing channelnames.

This constructor creates an image from a given set of channels. The channel names can optionally be specified. The passed channels should already be compressed::channel instances.

Parameters:

channels – A vector of compressed::channel instances to initialize the image with.
width – The width of the image in pixels.
height – The height of the image in pixels.
channel_names – (Optional) A list of channel names, must match the number of channels provided. If omitted or incorrect, channel names are ignored.

inline void add_channel(compressed::channel<T> _channel, std::optional<std::string> name = std::nullopt)#

Adds a compressed channel to the image.

This method moves the provided channel into the image’s internal storage, adding it to the list of channels.

Example:

compressed::channel<uint8_t, BlockSize, ChunkSize> channel = ...;
my_image.add_channel(std::move(channel));

Parameters:

_channel – The channel to be added to the image.
name – (Optional) Channel name of the channel to be inserted. If no channel names are set this argument is ignored.

inline void add_channel(std::span<const T> data, size_t width, size_t height, std::optional<std::string> name = std::nullopt, enums::codec compression_codec = enums::codec::lz4, uint8_t compression_level = 5)#

Adds a channel to the image.

This method moves the provided channel into the image’s internal storage, compressing it and adding it to the list of channels.

Example:

std::span<constT> channel = ...;
my_image.add_channel(channel, 1920, 1080, "red"));

Parameters:

data – The channel to be added to the image.
width – The width of the channel
height – The height of the channel
name – (Optional) Channel name of the channel to be inserted. If no channel names are set this argument is ignored.
compression_codec – (Optional) Compression codec to apply to the channel, every channel is allowed to have a different one.
compression_level – (Optional) Compression level, defaults to 5.

inline void remove_channel(size_t index)#

Remove a channel by its index.

Parameters:: index – The index of the channel to remove.
Throws:: std::out_of_range – if the index is out of bounds.

inline void remove_channel(const std::string_view name)#

Remove a channel by its name.

Parameters:: name – The name of the channel to remove.
Throws:: std::out_of_range – if the channel name is invalid.

inline compressed::channel<T> extract_channel(size_t index)#

Extracts a channel by its index.

Remove the channel from the image and gives you full control over the channel. Also erases its channel name.

Parameters:: index – The index of the channel to retrieve.
Throws:: std::out_of_range – if the index is out of bounds.
Returns:: The channel object.

inline compressed::channel<T> extract_channel(const std::string_view name)#

Extracts a channel by its name.

Remove the channel from the image and gives you full control over the channel. Also erases its channel name.

Parameters:: name – The name of the channel to retrieve.
Throws:: std::out_of_range – if the channel name is invalid.
Returns:: The channel object.

inline void print_statistics()#

Prints statistical information about the image file structure.

This function outputs various details about the compressed image, including dimensions, number of channels, compression ratio, and metadata.

Example output:

Statistics for image buffer:
 Width:             1024
 Height:            768
 Channels:          3
 Channelnames:      [R, G, B]
 --------------
 Compressed Size:   123456 bytes
 Uncompressed Size: 3145728 bytes
 Compression ratio: 25.5x
 Num Chunks:        512
 Metadata:
 {
    "author": "User",
    "timestamp": "2024-03-15"
 }

inline double compression_ratio() const noexcept#: Return the compression ratio over all channels.

inline auto begin() noexcept#

inline auto begin() const noexcept#

inline auto end() noexcept#

inline auto end() const noexcept#

inline compressed::channel<T> &channel(size_t index)#

Retrieves a reference to a channel by its index.

Parameters:: index – The index of the channel to retrieve.
Throws:: std::out_of_range – if the index is out of bounds.
Returns:: A reference to the requested channel.

inline compressed::channel<T> &channel(const std::string_view name)#

Retrieves a reference to a channel by its name.

Parameters:: name – The name of the channel to retrieve.
Throws:: std::out_of_range – if the channel name is invalid.
Returns:: A reference to the requested channel.

template<typename... Args> inline requires (std::conjunction_v< std::is_constructible< std::string, Args >... >) auto channels(Args... channel_names)

Retrieves references to multiple channels by name and returns them as a tuple.

Can be used with structured bindings to quickly get the specified channels from an image. These are returned as references (but don’t have to be bound as such)

Example:

compressed::image my_image = ...;
auto [r, g, b] = my_image.channels("r", "g", "b");

Template Parameters:: Args – Variadic template arguments, each convertible to std::string.
Parameters:: channel_names – The names of the channels to retrieve.
Returns:: A tuple containing references to the requested channels.

template<typename... Args> inline requires (std::conjunction_v< std::is_convertible< size_t, Args >... >) auto channels(Args... channel_indices)

Retrieves references to multiple channels by index and returns them as a tuple.

Can be used with structured bindings to quickly get the specified channels from an image. These are returned as references (but don’t have to be bound as such)

Example:

compressed::image my_image = ...;
auto [r, g, b] = my_image.channels(0, 1, 2);

Template Parameters:: Args – Variadic template arguments, each convertible to size_t.
Parameters:: channel_indices – The indices of the channels to get
Returns:: A tuple containing references to the requested channels.

inline std::vector<compressed::channel<T>&> channels(std::vector<size_t> channel_indices)#

Retrieves references to multiple channels their indices and returns them in a vector.

Parameters:: channel_indices – A vector of channel indices.
Throws:: std::out_of_range – if any channel indec is invalid.
Returns:: A vector containing references to the requested channels.

inline std::vector<compressed::channel<T>&> channels(std::vector<std::string> channel_names)#

Retrieves references to multiple channels by name and returns them in a vector.

Parameters:: channel_names – A vector of channel names.
Throws:: std::out_of_range – if any channel name is invalid.
Returns:: A vector containing references to the requested channels.

inline std::vector<compressed::channel<T>> &channels()#

Retrieves references to all of the channels in the image

Returns:: A vector containing references to the all the channels.

inline const std::vector<compressed::channel<T>> &channels() const#

Retrieves const references to all of the channels in the image

Returns:: A vector containing references to the all the channels.

inline std::vector<std::vector<T>> get_decompressed() const#

Decompress all of the channels and return them in planar fashion.

Each channel’s decompressed data is stored as a separate vector.

Returns:: A vector of decompressed channel data, where each inner vector corresponds to a channel.

inline size_t get_channel_offset(const std::string_view channelname) const#

Retrieve the logical index of the given channel.

This function searches for the specified channel name in the list of available channels. If the channel is not found, it throws a std::invalid_argument.

Parameters:: channelname – The name of the channel to search for.
Throws:: std::invalid_argument – if the channel is not available.
Returns:: The index of the channel if found.

inline size_t width() const noexcept#: Width of the Image.

inline size_t height() const noexcept#: Height of the image.

inline size_t num_channels() const noexcept#: Total number of channels in the image.

inline std::vector<std::string> channelnames() const noexcept#: Names of the channels stored on the image, are stored in the same order as the logical indices. So if the channelnames are { “B”, “G”, “R” } accessing channel “R” would be index 2.

inline void channelnames(std::vector<std::string> _channelnames)#: Set the channelnames according to their logical indices,.

inline void metadata(const json_ordered &_metadata) noexcept#: Arbitrary user metadata, not authored or managed by image class, it’s up to the caller to handle what goes in and comes out.

inline json_ordered &metadata() noexcept#: Arbitrary user metadata, not authored or managed by the image class, it’s up to the caller to handle what goes in and comes out.

inline const json_ordered &metadata() const noexcept#: Arbitrary user metadata, not authored or managed by image class, it’s up to the caller to handle what goes in and comes out.

inline void update_nthreads(size_t nthreads)#: Update the number of threads used internally by c-blosc2 for compression and decompression. This is automatically set when iterating through the images with compressed::for_each for example by specifying the compression codec.

inline size_t chunk_size() const#

Get the chunk size used for compression, this is the same across all channels.

Throws:: std::runtime_error – If the channels of the image do not all share the same chunk size as this is currently unsupported.
Returns:: The chunk size in bytes.

inline size_t block_size() const#

Public Static Functions

static inline image read(std::filesystem::path filepath, int subimage = 0, enums::codec compression_codec = enums::codec::lz4, size_t compression_level = 9, size_t block_size = s_default_blocksize, size_t chunk_size = s_default_chunksize)#

Reads a compressed image from a file using OpenImageIO and compresses it during reading.

Requires CompressedImage to have been compiled with OpenImageIO support.

This function reads an image file in chunks and compresses it on the fly leading to much lower memory usage at near-identical performance to raw OpenImageIO reads. On an image that is well compressible this can easily achieve a compression ratio of 5-10x.

The type does not have to match that of the underlying image as OpenImageIO will take care of converting the files into the specified format. It is perfectly valid to read a floating point image as e.g. uint16_t etc.

Example:

std::filesystem::path filepath = "image.exr";
auto img = compressed::image::read<uint8_t>(filepath, 0, compressed::enums::codec::lz4, 5);

Parameters:

filepath – The file path of the image to read.
subimage – The subimage to extract the channels from (default: 0). Only relevant for multi-part images.
compression_codec – The compression codec to use (default: LZ4).
compression_level – The compression level (default: 9).
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Returns:

A compressed image instance.

template<typename PostProcess> static inline requires std::invocable< std::remove_reference_t< PostProcess >, size_t, std::span< T > > image read (std::filesystem::path filepath, PostProcess &&postprocess, int subimage=0, enums::codec compression_codec=enums::codec::lz4, size_t compression_level=9, size_t block_size=s_default_blocksize, size_t chunk_size=s_default_chunksize)

Reads a compressed image from a file using OpenImageIO and compresses it during reading.

Requires CompressedImage to have been compiled with OpenImageIO support.

This overload allows you to specify a custom invocable function which is executed after a chunk has been read and before it is compressed. If you have some common operations like color management or a filter which you wish to apply this would go in here. Specifying these right away in the read is much more efficient than iterating over the image again later and applying these.

The function passed should have no notion of coordinates or similar, it should simply assume to receive a block of data (that is part of an image) as well as the channel index we are currently operating over.

Example:

std::filesystem::path filepath = "image.exr";

// Read an image file and apply a post-process which adds 1 to the pixel value for all RGB channels (0, 1, 2).

auto postprocess = [](size_t channel_idx, std::span<T> chunk)
    {
        if (channel_idx > 2)
        {
            return;
        }

        std::for_each(std::execution::par_unseq, chunk.begin(), chunk.end(), [](T& value)
        {
            value += 1;
        }
    };

auto img = compressed::image::read<uint8_t>(
    filepath, 
    std::forward(postprocess),
    0, // subimage
    compressed::enums::codec::lz4, // compression_code
    5 // compression_level
);

Parameters:

filepath – The file path of the image to read.
postprocess – A postprocessing function to run after read but before re-compression. This function should take a size_t and a std::span<T> where the size_t is the channel index we are currently iterating over (e.g. 3 for the alpha channel) and the std::span<T> is a chunk within that channel, where this chunk is and what coordinates it represents is not passed along.
subimage – The subimage to extract the channels from (default: 0). Only relevant for multi-part images.
compression_codec – The compression codec to use (default: LZ4).
compression_level – The compression level (default: 9).
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Returns:

A compressed image instance.

static inline image read(std::unique_ptr<OIIO::ImageInput> input_ptr, std::vector<int> channelindices, int subimage = 0, enums::codec compression_codec = enums::codec::lz4, size_t compression_level = 9, size_t block_size = s_default_blocksize, size_t chunk_size = s_default_chunksize)#

Reads a compressed image from a file using OpenImageIO and compresses it during reading.

Requires CompressedImage to have been compiled with OpenImageIO support.

This overload allows you to only extract the channels specified which is useful if you have e.g. a multilayer file but only wish to extract the RGBA components.

We will internally take care of optimizing the calls to the OpenImageIO API for maximum read throughput.

Example:

std::filesystem::path filepath = "image.exr";

auto input_ptr = OIIO::ImageInput::open(filepath);
if (!input_ptr)
{
    throw std::runtime_error(std::format("file {} does not exist on disk", filepath.string()));
}

auto img = compressed::image::read<uint8_t>(input_ptr, {0, 1, 2, 3});

Parameters:

input_ptr – The opened OIIO input pointer.
channelindices – The channels you wish to extract. These may be specified in any order. We throw a std::out_of_range if one of the passed channels does not exist. It is perfectly valid to e.g. call this with {3, 1, 2} when the underlying channel structure may be RGBA. Sorting these back into their underlying channel structure is done on read.
subimage – The subimage to extract the channels from (default: 0). Only relevant for multi-part images.
compression_codec – The compression codec to use (default: LZ4).
compression_level – The compression level (default: 9).
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Returns:

A compressed image instance.

template<typename PostProcess> static inline requires std::invocable< std::remove_reference_t< PostProcess >, size_t, std::span< T > > image read (std::unique_ptr< OIIO::ImageInput > input_ptr, PostProcess &&postprocess, std::vector< int > channelindices, int subimage=0, enums::codec compression_codec=enums::codec::lz4, size_t compression_level=9, size_t block_size=s_default_blocksize, size_t chunk_size=s_default_chunksize)

Reads a compressed image from a file using OpenImageIO and compresses it during reading.

Requires CompressedImage to have been compiled with OpenImageIO support.

This overload allows you to only extract the channels specified which is useful if you have e.g. a multilayer file but only wish to extract the RGBA components.

We will internally take care of optimizing the calls to the OpenImageIO API for maximum read throughput.

This function allows you to specify a custom invocable function which is executed after a chunk has been read and before it is compressed. If you have some common operations like color management or a filter which you wish to apply this would go in here. Specifying these right away in the read is much more efficient than iterating over the image again later and applying these.

Example:

std::filesystem::path filepath = "image.exr";

auto input_ptr = OIIO::ImageInput::open(filepath);
if (!input_ptr)
{
    throw std::runtime_error(std::format("file {} does not exist on disk", filepath.string()));
}

auto postprocess = [](size_t channel_idx, std::span<T> chunk)
    {
        if (channel_idx > 2)
        {
            return;
        }
    
        std::for_each(std::execution::par_unseq, chunk.begin(), chunk.end(), [](T& value)
        {
            value += 1;
        }
    };

// Read an image file and apply a post-process which adds 1 to the pixel value for all RGB channels (0, 1, 2).
auto img = compressed::image::read<uint8_t>(
    std::move(input_ptr), 
    std::forward(postprocess),
    { 0, 1, 2, 3}, // only read the RGBA channels
    0, // subimage
    compressed::enums::codec::lz4, 
    5
);

Parameters:

input_ptr – The opened OIIO input pointer.
postprocess – A postprocessing function to run after read but before re-compression. This function should take a size_t and a std::span<T> where the size_t is the channel index we are currently iterating over (e.g. 3 for the alpha channel) and the std::span<T> is a chunk within that channel, where this chunk is and what coordinates it represents is not passed along.
channelindices – The channels you wish to extract. These may be specified in any order. We throw a std::out_of_range if one of the passed channels does not exist. It is perfectly valid to e.g. call this with {3, 1, 2} when the underlying channel structure may be RGBA. Sorting these back into their underlying channel structure is done on read.
subimage – The subimage to extract the channels from (default: 0). Only relevant for multi-part images.
compression_codec – The compression codec to use (default: LZ4).
compression_level – The compression level (default: 9).
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Returns:

A compressed image instance.

static inline image read(std::unique_ptr<OIIO::ImageInput> input_ptr, std::vector<std::string> channelnames, int subimage = 0, enums::codec compression_codec = enums::codec::lz4, size_t compression_level = 9, size_t block_size = s_default_blocksize, size_t chunk_size = s_default_chunksize)#

Reads a compressed image from a file using OpenImageIO and compresses it during reading.

Requires CompressedImage to have been compiled with OpenImageIO support.

This overload allows you to only extract the channels specified which is useful if you have e.g. a multilayer file but only wish to extract the RGBA components.

We will internally take care of optimizing the calls to the OpenImageIO API for maximum read throughput.

Example:

std::filesystem::path filepath = "image.exr";

auto input_ptr = OIIO::ImageInput::open(filepath);
if (!input_ptr)
{
    throw std::runtime_error(std::format("file {} does not exist on disk", filepath.string()));
}

auto img = compressed::image::read<uint8_t>(std::move(input_ptr), {"R", "G", "B", "A"});

Parameters:

input_ptr – The opened OIIO input pointer.
channelnames – The channels you wish to extract. These may be specified in any order. We throw a std::out_of_range if one of the passed channels does not exist. It is perfectly valid to e.g. call this with {“G”, “R”, “A”} when the underlying channel structure may be RGBA. Sorting these back into their underlying channel structure is done on read.
subimage – The subimage to extract the channels from (default: 0). Only relevant for multi-part images.
compression_codec – The compression codec to use (default: LZ4).
compression_level – The compression level (default: 9).
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Returns:

A compressed image instance.

template<typename PostProcess> static inline requires std::invocable< std::remove_reference_t< PostProcess >, size_t, std::span< T > > image read (std::unique_ptr< OIIO::ImageInput > input_ptr, PostProcess &&postprocess, std::vector< std::string > channelnames, int subimage=0, enums::codec compression_codec=enums::codec::lz4, size_t compression_level=9, size_t block_size=s_default_blocksize, size_t chunk_size=s_default_chunksize)

Reads a compressed image from a file using OpenImageIO and compresses it during reading.

Requires CompressedImage to have been compiled with OpenImageIO support.

This overload allows you to only extract the channels specified which is useful if you have e.g. a multilayer file but only wish to extract the RGBA components.

We will internally take care of optimizing the calls to the OpenImageIO API for maximum read throughput.

Example:

std::filesystem::path filepath = "image.exr";

auto input_ptr = OIIO::ImageInput::open(filepath);
if (!input_ptr)
{
    throw std::runtime_error(std::format("file {} does not exist on disk", filepath.string()));
}

auto postprocess = [](size_t channel_idx, std::span<T> chunk)
    {
        if (channel_idx > 2)
        {
            return;
        }

        std::for_each(std::execution::par_unseq, chunk.begin(), chunk.end(), [](T& value)
        {
            value += 1;
        }
    };

// Read an image file and apply a post-process which adds 1 to the pixel value for all RGB channels (0, 1, 2).
auto img = compressed::image::read<uint8_t>(
    std::move(input_ptr), 
    std::forward(postprocess),
    { 0, 1, 2, 3}, // only read the RGBA channels
    0, // subimage
    compressed::enums::codec::lz4, 
    5
);

Parameters:

input_ptr – The opened OIIO input pointer.
postprocess – A postprocessing function to run after read but before re-compression. This function should take a size_t and a std::span<T> where the size_t is the channel index we are currently iterating over (e.g. 3 for the alpha channel) and the std::span<T> is a chunk within that channel, where this chunk is and what coordinates it represents is not passed along.
channelnames – The channels you wish to extract. These may be specified in any order. We throw a std::out_of_range if one of the passed channels does not exist. It is perfectly valid to e.g. call this with {“G”, “R”, “A”} when the underlying channel structure may be RGBA. Sorting these back into their underlying channel structure is done on read.
subimage – The subimage to extract the channels from (default: 0). Only relevant for multi-part images.
compression_codec – The compression codec to use (default: LZ4).
compression_level – The compression level (default: 9).
block_size – The size of the blocks stored inside the chunks, defaults to 32KB which is enough to comfortably fit into the L1 cache of most modern CPUs. If you know your cpu can handle larger blocks feel free to up this number.
chunk_size –
The size of each individual chunk, defaults to 4MB which is enough to hold a 2048x2048 channel. This should be tweaked to be no larger than the size of the usual images you are expecting

to compress for optimal performance but this could be upped which might give better compression ratios. Must be a multiple of sizeof(T).

Returns:

A compressed image instance.

static inline json_ordered read_oiio_metadata(const OIIO::ImageSpec &spec)#

Read the metadata from the openimageio pointer into a json representation.

Parameters:: input_ptr – The input file to query
Returns:: The metadata encoded as json. This does not recursively parse jsons!

static inline json_ordered read_oiio_metadata(std::filesystem::path filepath)#

Read the metadata from the file into a json representation.

Parameters:: input_ptr – The input file to query
Throws:: std::invalid_argument – if the file does not exist on disk.
Returns:: The metadata encoded as json. This does not recursively parse jsons!

python

class compressed_image.Image#

A dynamically-typed compressed image composed of multiple channels, supporting interaction with numpy arrays and efficient memory/storage via lazy compression.

Supports the following np.dtypes as fill values:

np.float16
np.float32
np.uint8
np.int8
np.uint16
np.int16
np.uint32
np.int32

The channels are stored as compressed buffers allowing for efficient traversal and decompression.

__init__(self: compressed_image.lib64.compressed_image.Image, dtype: object, channels: collections.abc.Sequence[numpy.ndarray], width: typing.SupportsInt, height: typing.SupportsInt, channel_names: collections.abc.Sequence[str] = [], compression_codec: compressed_image.lib64.compressed_image.Codec = <Codec.lz4: 1>, compression_level: typing.SupportsInt = 9, block_size: typing.SupportsInt = 32768, chunk_size: typing.SupportsInt = 4194304) → None#

Construct a compressed image from a list of numpy arrays, converting the arrays into compressed_image.Channel instances. The compresion settings can be controlled with the compression_codec, compression_level, block_size, chunk_size parameters.

Parameters:

dtype – The dtype of the image data.
channels – List of 2D numpy arrays representing image channels. These should have the shape (height, width) storing the image data in scanline order.
width – Width of the image.
height – Height of the image.
channel_names – Optional list of channel names to assign.
compression_codec – Compression codec, defaults to lz4 which is a good compromise between compression ratio and speed usually achieving between 5-10x compression ratios.
compression_level – Compression level, defaults to 9 for lz4 and typically ranges between 0-9. For lz4 9 is usually a good value as performance doesn’t suffer much from having this at its highest.
block_size – The block size used internally in the 3d containers, defaults to 32KB which, from testing, appears to be a good all-purpose value for both performance and compression ratio. This is the size of a single run of data to be compressed and should roughly fit into the L1 cache of your CPU for best performance. The blocks are not transparent to the user of the API, the lowest level that can be accessed is chunks.
chunk_size –
Chunk size used internally (next level after blocks). Defaults to 4 * 1024 * 1024.

This value should be:
- Small enough to allow efficient partial extraction (e.g., for region-of-interest).
- Large enough to saturate available cores when divided by block_size.
These defaults are tuned for good performance on a wide range of systems.

add_channel(self: compressed_image.lib64.compressed_image.Image, data: numpy.ndarray, width: typing.SupportsInt, height: typing.SupportsInt, name: str | None = None, compression_codec: compressed_image.lib64.compressed_image.Codec = <Codec.lz4: 1>, compression_level: typing.SupportsInt = 9, block_size: typing.SupportsInt = 32768, chunk_size: typing.SupportsInt = 4194304) → None#

Add a channel to the image from the given uncompressed data. Compresses it and stores it in self.channels.

Parameters:

data – The image data to add as the channel, this needs to both have the same dtype as self.dtype as well as having to have the shape of (height, width).
width – The width of the channel, must be the same as the rest of the image.
height – The height of the channel, must be the same as the rest of the image.
compression_codec – Compression codec, defaults to lz4 which is a good compromise between compression ratio and speed usually achieving between 5-10x compression ratios.
compression_level – Compression level, defaults to 9 for lz4 and typically ranges between 0-9. For lz4 9 is usually a good value as performance doesn’t suffer much from having this at its highest.
block_size – The block size used internally in the 3d containers, defaults to 32KB which, from testing, appears to be a good all-purpose value for both performance and compression ratio. This is the size of a single run of data to be compressed and should roughly fit into the L1 cache of your CPU for best performance. The blocks are not transparent to the user of the API, the lowest level that can be accessed is chunks.
chunk_size –
Chunk size used internally (next level after blocks). Defaults to 4 * 1024 * 1024.

This value should be:
- Small enough to allow efficient partial extraction (e.g., for region-of-interest).
- Large enough to saturate available cores when divided by block_size.
These defaults are tuned for good performance on a wide range of systems.

block_size(self: compressed_image.lib64.compressed_image.Image) → int#: Get the block size used for compression, this is the same across all channels.

channel(*args, **kwargs)#

Overloaded function.

channel(self: compressed_image.lib64.compressed_image.Image, index: typing.SupportsInt) -> compressed_image.lib64.compressed_image.Channel

Retrieve a channel by its logical index

Parameters:: index – The index of the channel, must be valid
Returns:: The channel, this may be modified in-place and will be updated on the containing image.

channel(self: compressed_image.lib64.compressed_image.Image, name: str) -> compressed_image.lib64.compressed_image.Channel

Retrieve a channel by its name

Parameters:: name – The name of the channel, must be valid
Returns:: The channel, this may be modified in-place and will be updated on the containing image.

channels(self: compressed_image.lib64.compressed_image.Image) → list[compressed_image.lib64.compressed_image.Channel]#: Retrieve a reference to all the channels of the image, the compressed_image.Channel instances may be modified directly but modifying the list will not update the channels of the image. To add or remove a channel please call add_channel or remove_channel.

chunk_size(self: compressed_image.lib64.compressed_image.Image) → int#: Get the chunk size used for compression, this is the same across all channels.

compression_ratio(self: compressed_image.lib64.compressed_image.Image) → float#

Returns:: The compression ratio of the data.

static dtype_from_file(filepath: str) → numpy.dtype#

Extract the dtype from a given image file without having to read the whole image. This is the recommended method for extracting the files’ dtype from an image.

Returns:: The dtype of the image at the given filepath.

static dtypes_from_file(filepath: str) → list[numpy.dtype]#

Extract all the dtypes from a given image file without having to read the whole image. This is the recommended method for extracting the files’ dtype from an image. This will either extract a list of all dtypes of the image with each entry corresponding to the a channel or a list of a single value with the dtype for the whole image if the image does not have per-channel dtypes.

So in an image with these channels

[“R”, “G”, “B”, “A”, “Z”]

it might be like this:

[np.float16, np.float16, np.float16, np.float16, np.float32]

Returns:: The dtypes of the image at the given filepath.

get_channel_index(self: compressed_image.lib64.compressed_image.Image, channelname: str) → int#: Return the index of a channel by its name, raising a ValueError if the name is not valid.

get_channel_names(self: compressed_image.lib64.compressed_image.Image) → list[str]#: Retrieve the channel names, these aren’t guaranteed to be populated if e.g. the image doesn’t have any channel names assigned to them on creation. This is always guaranteed however to be either empty or the size of self.num_channels

get_decompressed(self: compressed_image.lib64.compressed_image.Image) → list[numpy.ndarray]#: Retrieve all the channels as a list of numpy.ndarrays. These are guaranteed to be in the same order as the underlying channels so it is safe to access these by their logical index.

get_metadata(self: compressed_image.lib64.compressed_image.Image) → json#: Retrieve the metadata stored on the image. If the image was created via one of the read() methods this will be populated with the image metadata which will store additional information.

property height#

Returns:: The image height. All channels in the image are guaranteed to be of this height.

property num_channels#

Returns:: Number of channels. Equivalent to calling len(image)

print_statistics(self: compressed_image.lib64.compressed_image.Image) → None#

Prints some general image statistics such as compression ratio, channel names, bytesize etc.

Example output:

Statistics for image buffer: Width: 1024 Height: 768 Channels: 3 Channelnames: [R, G, B] ————– Compressed Size: 123456 bytes Uncompressed Size: 3145728 bytes Compression ratio: 25.5x Num Chunks: 512 Metadata: { “author”: “User”, “timestamp”: “2024-03-15” }

static read(*args, **kwargs)#

Overloaded function.

read(dtype: object, filepath: str, subimage: typing.SupportsInt, compression_codec: compressed_image.lib64.compressed_image.Codec = <Codec.lz4: 1>, compression_level: typing.SupportsInt = 9, block_size: typing.SupportsInt = 32768, chunk_size: typing.SupportsInt = 4194304) -> compressed_image.lib64.compressed_image.Image

Reads the specified image from disk, converting into the passed dtype and compressing on the fly. This method is much more memory efficient and faster than doing this step yourself (reading and then passing read data to compressed_image.Image) as it reads in chunks and compressed on the fly.

All the channels at the given subimage are read, if multiple subimages are to be read it is recommended to split this into several compressed_image.Image for the above mentioned reasons.

This method automatically populates the metadata that can be accessed through self.get_metadata() with the image metadata.

Parameters:

dtype – The data type to read as, this doesn’t have to correspond to the data type of the image as we will convert to the data on read. If you wish to find out the data type of an image without having to read it you can use the Image.dtype_from_file method.
filepath – The path to the image file, this must be in a format supported by OpenImageIO.
subimage – The subimage within the image to read. Only relevant for a couple of formats such as tiff or exr. If you need to find out how to read the subimage information from an image please refer to the docs for OpneImageIO.
compression_codec – Compression codec, defaults to lz4 which is a good compromise between compression ratio and speed usually achieving between 5-10x compression ratios.
compression_level – Compression level, defaults to 9 for lz4 and typically ranges between 0-9. For lz4 9 is usually a good value as performance doesn’t suffer much from having this at its highest.
block_size – The block size used internally in the 3d containers, defaults to 32KB which, from testing, appears to be a good all-purpose value for both performance and compression ratio. This is the size of a single run of data to be compressed and should roughly fit into the L1 cache of your CPU for best performance. The blocks are not transparent to the user of the API, the lowest level that can be accessed is chunks.
chunk_size –
Chunk size used internally (next level after blocks). Defaults to 4 * 1024 * 1024.

This value should be:
- Small enough to allow efficient partial extraction (e.g., for region-of-interest).
- Large enough to saturate available cores when divided by block_size.
These defaults are tuned for good performance on a wide range of systems.

read(dtype: object, filepath: str, subimage: typing.SupportsInt, channel_indices: collections.abc.Sequence[typing.SupportsInt], compression_codec: compressed_image.lib64.compressed_image.Codec = <Codec.lz4: 1>, compression_level: typing.SupportsInt = 9, block_size: typing.SupportsInt = 32768, chunk_size: typing.SupportsInt = 4194304) -> compressed_image.lib64.compressed_image.Image

The channels specified by channel_indices are read from subimage, if multiple subimages are to be read it is recommended to split this into several compressed_image.Image for the above mentioned reasons.

This method automatically populates the metadata that can be accessed through self.get_metadata() with the image metadata.

Parameters:

dtype – The data type to read as, this doesn’t have to correspond to the data type of the image as we will convert to the data on read. If you wish to find out the data type of an image without having to read it you can use the Image.dtype_from_file method.
filepath – The path to the image file, this must be in a format supported by OpenImageIO.
subimage – The subimage within the image to read. Only relevant for a couple of formats such as tiff or exr. If you need to find out how to read the subimage information from an image please refer to the docs for OpneImageIO.
channel_indices – The indices of the channels to read. These must be valid for the passed image and subimage as otherwise we will throw an exception.
compression_codec – Compression codec, defaults to lz4 which is a good compromise between compression ratio and speed usually achieving between 5-10x compression ratios.
compression_level – Compression level, defaults to 9 for lz4 and typically ranges between 0-9. For lz4 9 is usually a good value as performance doesn’t suffer much from having this at its highest.
block_size – The block size used internally in the 3d containers, defaults to 32KB which, from testing, appears to be a good all-purpose value for both performance and compression ratio. This is the size of a single run of data to be compressed and should roughly fit into the L1 cache of your CPU for best performance. The blocks are not transparent to the user of the API, the lowest level that can be accessed is chunks.
chunk_size –
Chunk size used internally (next level after blocks). Defaults to 4 * 1024 * 1024.

This value should be:
- Small enough to allow efficient partial extraction (e.g., for region-of-interest).
- Large enough to saturate available cores when divided by block_size.
These defaults are tuned for good performance on a wide range of systems.

read(dtype: object, filepath: str, subimage: typing.SupportsInt, channel_names: collections.abc.Sequence[str], compression_codec: compressed_image.lib64.compressed_image.Codec = <Codec.lz4: 1>, compression_level: typing.SupportsInt = 9, block_size: typing.SupportsInt = 32768, chunk_size: typing.SupportsInt = 4194304) -> compressed_image.lib64.compressed_image.Image

The channels specified by channel_names are read from subimage, if multiple subimages are to be read it is recommended to split this into several compressed_image.Image for the above mentioned reasons.

This method automatically populates the metadata that can be accessed through self.get_metadata() with the image metadata.

Parameters:

dtype – The data type to read as, this doesn’t have to correspond to the data type of the image as we will convert to the data on read. If you wish to find out the data type of an image without having to read it you can use the Image.dtype_from_file method.
filepath – The path to the image file, this must be in a format supported by OpenImageIO.
subimage – The subimage within the image to read. Only relevant for a couple of formats such as tiff or exr. If you need to find out how to read the subimage information from an image please refer to the docs for OpneImageIO.
channel_names – The names of the channels to read. These must be valid for the passed image and subimage as otherwise we will throw an exception.
compression_codec – Compression codec, defaults to lz4 which is a good compromise between compression ratio and speed usually achieving between 5-10x compression ratios.
compression_level – Compression level, defaults to 9 for lz4 and typically ranges between 0-9. For lz4 9 is usually a good value as performance doesn’t suffer much from having this at its highest.
block_size – The block size used internally in the 3d containers, defaults to 32KB which, from testing, appears to be a good all-purpose value for both performance and compression ratio. This is the size of a single run of data to be compressed and should roughly fit into the L1 cache of your CPU for best performance. The blocks are not transparent to the user of the API, the lowest level that can be accessed is chunks.
chunk_size –
Chunk size used internally (next level after blocks). Defaults to 4 * 1024 * 1024.

This value should be:
- Small enough to allow efficient partial extraction (e.g., for region-of-interest).
- Large enough to saturate available cores when divided by block_size.
These defaults are tuned for good performance on a wide range of systems.

remove_channel(self: compressed_image.lib64.compressed_image.Image, name_or_index: SupportsInt | str) → None#: Remove the channel by name or index.

set_channel_names(self: compressed_image.lib64.compressed_image.Image, arg0: collections.abc.Sequence[str]) → None#: Set the channel names, the names of these may be whatever you wish the to be, the only restriction is that the length of this list must be == len(channel).

set_metadata(self: compressed_image.lib64.compressed_image.Image, metadata: json) → None#

Set the metadata on the image, this may be any arbitrary dict and it is entirely up to you to manage this metadata/interpret it. This may for example encode color space information.

Parameters:: metadata – The metadata to set on the image.

property shape#

Returns:: Image shape in format (nchannels, height, width)

update_nthreads(self: compressed_image.lib64.compressed_image.Image, nthreads: SupportsInt) → None#: Update the number of threads used internally for compression/decompression. By default this will use all available system threads.

property width#

Returns:: The image width. All channels in the image are guaranteed to be of this width.