When working with images as a software engineer, color may seem a straightforward concept. After decoding an image you typically end up with an array of pixel values, each a tuple of three integer components. These components generally are “RGB” and so the value (255, 0, 0) must denote red, (0, 255, 0) is green, etc. Great, you think, and move on. However, beyond this simple idea lies a fair amount of complexity and subtle errors are easy to introduce.
Let’s consider a small example for illustrative purposes. Suppose we have a background image on which we want to display a logo. Let’s load the values of the image, choose a nice color for the logo, and blend so we get smooth edges. You can view the example in Shadertoy. Looks like it did what we want, but upon closer inspection it’s a bit off. The edges of our logo have a pink-ish halo, instead of the deep purple of our logo. Something went wrong in the blending function and clearly things aren’t as simple as initially assumed. In this post, we’ll have a look at the underlying mechanisms of digital color, what actually goes wrong in the example, and how to fix it.
Defining color
To understand the problem we are observing, we will first have a closer look at the underlying mechanisms of color. After decoding an image, we end up with an array of pixels. Each pixel is a tuple of values. A color encoding is the digital encoding of a color space and defines how the tuples map to a coordinate in a color space and how varying those values moves the coordinate throughout the space. The color space is a geometric representation of colors in space and defines how those coordinates unambiguously map into actual colors. Note that there are multiple, conflicting definitions of these terms, we use the ones as defined by [1] and [2]. As we’ll see later, the color encoding of our example is standard RGB (sRGB) with the associated sRGB color space. This defines each color so that it can be produced identically, whether that is on a digital display or printed on paper. To continue with our example, RGB color spaces are typically defined using three components:
- The primaries: the absolute coordinates or chromaticities of the primary colors red, green, and blue. These together define the gamut, which is the range of colors that can be represented.
- The white point: the chromaticity of white. It can be loosely thought of as being the “color temperature” of white.
- The transfer function: how linear light intensity is encoded into non-linear numeric RGB values.
The gamut and white point are mostly straightforward and carve out a subset from all visible colors. They are defined as coordinates in an absolute space. A commonly used color space for describing these values is the CIE XYZ space, which represent the full range of human-visible color. Usually color spaces avoid capturing the full range, as it limits the granularity at which colors can be represented given fixed-precision data values. Furthermore, many devices are limited in the amount of colors that can be represented. As an example, the commonly used sRGB is smaller compared to Adobe RGB or Display P3.
The transfer function is slightly more involved. The problem it aims to address is that color is a continuous space which must be represented by a finite number of values. It so happens that human eyes are better at distinguishing differences in low brightness values than differences in high brightness values. For instance, the difference between 10% brightness and 20% brightness appears to be larger than the difference between 80% and 90%. To better utilize the limited number of values, low brightness colors are represented with a fine quantization and high brightness colors are represented with a coarse quantization. The transfer function is the non-linear function that maps color values based on the brightness. As an example the sRGB transfer function is shown below, which is sometimes approximated by (as “having a gamma” of 2.2).
Finding the color encoding
It may not be trivial to find the color encoding of your image data. It usually depends on the image format. For instance, JPEG is usually stored as YCbCr using the ITU-R BT.601 matrix to convert to RGB values. These RGB values are then usually interpreted to be sRGB. However, there may be application-specific metadata attached to signify another interpretation. PNG has multiple standardized ways to express the color encoding in metadata chunks, though none may be present. A standardized way to specify color encoding is via ICC profiles, and it is possible to add such profiles both to JPEG and PNG. Modern formats like JPEG XL take a different approach: the color encoding is part of the data, not the metadata. Therefore a valid JPEG XL file will have an unambiguous color interpretation.
Once you know what file format you are working with, finding the color encoding becomes a matter of reading the specification. In the case of older image formats, you may need to also read third-party specification documents. A great tool to help with this is ExifTool. Contrary to what the name suggests, this tool helps with metadata processing not only of Exif metadata, but just about any metadata that an image may contain.
The below example shows color encoding metadata in action. The left image is compressed with ImageMagick, instructed to produce a JPEG in the ProPhoto color encoding. ProPhoto is an encoding with a very wide gamut. The right image shows what happens if the JPEG metadata is removed. The decoder will fall back to decoding with the BT.601 matrix and interpreting the resulting RGB as sRGB, causing the wrong colors to be displayed.
Note that this website uses a content delivery network (CDN) to optimize network traffic when sending the images to the browser. The CDN happens to normalize the images to WebP, converting the actual color values instead of preserving metadata. This process is a practical case of color space in itself. If the CDN JPEG decoder did not properly take the metadata into account, the left image would also show desaturated.
Working with color encodings
Now that we understand the underlying principles, let’s return to the example. The color values we load in OpenGL and GLSL with texture(iChannel0, vec2(uv)).xyz in Shadertoy are loaded verbatim from the texture (source). The original texture has sRGB values and so does the color of the logo we picked out in the color picker. If we directly blend a white background (255, 255, 255) and our color (90, 0, 255) we end up with (173, 128, 255). However, this value is off as we are blending encoded colors.
We know that the sRGB space is non-linear due to the transfer function. This means the values must be converted to a linear space before mixing. We can look up the desired functions on Wikipedia. After having done the conversion and mixing, the values must be converted back to sRGB for output. The full example is shown in Shadertoy. Now, the halfway point between a white background and our color is computed to be the softer color (196, 188, 255). Now, the odd halo is gone and we end up with a nice-looking image.
Note for the observant reader: the example in Shadertoy is for demonstration purposes. With OpenGL it is possible to signal the input texture as sRGB. In this case, the graphics hardware will take care of the sRGB to linear conversion.
A tradeoff
Understanding the theory behind color enables one to correctly handle all possible situations. However, doing so comes at a cost of complexity and performance. Parsing and interpreting all possible metadata for a file format of over 30 years old is a huge scope. Converting values from an arbitrary color encoding into another is similarly non-trivial. Decoding color values into their linear representations requires representing them with additional precision. If intermediate results are materialized, more memory must be used to represent them.
This is why many brilliant ShaderToy shaders choose not to perform color-correct blending. If the difference is hardly noticeable, why spend additional cycles computing it? In high-performance computing similar tradeoffs between correctness, performance, and complexity must be made.
Wrapping up
Through these examples it should be clear that RGB values are not colors by themselves. To properly work with colors, one needs to determine what the values represent. Deducing this information and correctly handling color comes at a cost. It likely involves a conversion to a linear representation, and the inverse operation before encoding to an image file. Not all software needs implement full color management: knowing when to make the tradeoff between correctness, complexity, and performance is key.
This article was updated on 2026-10-01 based on a comment from an attentive reader, to improve the definitions of color space and color encoding, and their distinction.
References
- International Color Consortium, White Paper #5 Glossary of terms.
- International Commission on Illumination, Terms and definitions of International Lighting Vocabulary.