Skip to main content

How to Use Photoshop's Auto-Colorize Photos Feature

Tim Arends has for many years been an interested observer in the progress of artificial intelligence in areas such as image manipulation.

Adobe Photoshop's Newest AI Feature

There's a new feature in Adobe Photoshop and Photoshop Elements. It goes beyond removing red-eye, obliterating background elements and swapping people's heads around. It's called Colorize Photo, and it uses artificial intelligence to turn your old black-and-white photographs into vibrant color images with the click of a button.

So how well does it work? Let's look at some photos and see what it does well, as well as identify a few weaknesses of the algorithm. We will start with the most excellent results and work our way down to fair or poor results. Knowing this, you can use the feature more effectively in coloring your own black-and-white photos.

Note: The below are royalty-free images from Art Explosion 750,000 Images clip art collection. They were originally color photos which I converted to grayscale for the purposes of testing shown here.

Examples of Excellent Results

Below are some examples of excellent results from Colorize Photos. None of these have had any retouching of my own applied. You can expect results this good about 10% of the time.

Colorize photos did an excellent job here. Notice how it accurately colorized both the model's skin and hair.

Colorize photos did an excellent job here. Notice how it accurately colorized both the model's skin and hair.

The artificial intelligence tinted this Army Nurse Corp's lips red, gave her a natural skin tone, and even added a gold tint to the badge on her cap

The artificial intelligence tinted this Army Nurse Corp's lips red, gave her a natural skin tone, and even added a gold tint to the badge on her cap

Here, the algorithm accurately colored the individual strands of the model’s hair, something that would take infinitely longer if done by hand.

Here, the algorithm accurately colored the individual strands of the model’s hair, something that would take infinitely longer if done by hand.

Colorize Photos always gives four choices. 95% of the time you will likely prefer the first choice. The first choice (shown here) is the one you will usually prefer.

Colorize Photos always gives four choices. 95% of the time you will likely prefer the first choice. The first choice (shown here) is the one you will usually prefer.

Sometimes however, the third choice, which adds more blue to the image, also gives pleasing results.

Sometimes however, the third choice, which adds more blue to the image, also gives pleasing results.

Notice how colorize photos effectively captured the greenery on the mountainside, the blue in the lake and mountains in the distance. It even gave the foreground subject's jacket a nice red tint!

Notice how colorize photos effectively captured the greenery on the mountainside, the blue in the lake and mountains in the distance. It even gave the foreground subject's jacket a nice red tint!

Look how effectively the AI colored the buildings, sky, trees, water and even reflections in the water of this scene in Poland!

Look how effectively the AI colored the buildings, sky, trees, water and even reflections in the water of this scene in Poland!

Colorizing Photos with Artificial Intelligence

So you've seen some examples of Colorize Photos in action. How does it work?

You're probably familiar with Instagram-style filters. These take your images and apply various effects to them, such as giving them a slightly yellowed, old-time vintage effect. Instagram filters are known as "destructive" filters, because, in order to give photos that old-timey look, they throw away data from the image.

You're probably also familiar with the process of colorizing old movies, which uses computers to add color to a vintage film frame-by-frame. The process is made possible by computers, but it still requires a lot of human effort. The initial frame of each scene must be colored by hand. The computer then takes this information and extends it to the other frames of that scene. But the process has to be started and supervised by humans. Colorizing a still image by hand is a similarly laborious process.

However, Photoshop's Colorize Photos promises to turn a photo from dull black and white to vibrant color with a single click of a button. There's no way a piece of software can do this without employing what is known as artificial intelligence, or the science of making computers do things that previously required human judgment and expertise.

If you have performed a Google image search, you have seen a similar technology in action. Drag one of your own photos to the search box, and if the photo exists on the Internet, Google will usually find it. If it doesn't, Google will show images that are thematically similar -- a dancing woman, a couple on the beach, an automobile, etc.

Google is employing a form of artificial intelligence called image recognition, which allows it to identify the subject of your photo and find similar photos. Sites like Pinterest, Apple Photos and others have a similar feature.


How Colorize Photos Works

So how does Colorize Photo work? Through a process of training artificial intelligence known as deep learning. The developers expose the algorithm to thousands of photos and eventually the software learns that a tree is usually green, the sky is usually blue, and so on.

This is a far greater challenge than it seems. Although you may think a tree is a tree is a tree, the reality is that no two trees are alike, and even the same tree can look very different at different times of day and from different angles. So the challenge of image recognition was an incredible one for developers to surmount, but over the years they have made great strides in the process.

Photoshop's Colorize Photos algorithm employs a similar process. In order to work, it must scan the photo and do at least some analysis of the subject matter so as to determine which color should go where. I believe the algorithm also uses facial recognition to identify faces and uses these as a starting point in determining that the photo contains a person, as well as in coloring the rest of the photo. It often works particularly well with portraits.

By the way, unlike Google's image recognition, Photoshop's Colorize Photos algorithm does not require an Internet connection. I tested this by unplugging my computer from the Internet and running the algorithm, and it worked! That means the "brains" of the AI are built into the software and reside directly on your computer when you download and install the program.

Examples of Pretty Good Results

The above examples are of stellar results from the colorizing artificial intelligence. You can expect results that good only about 10% of the time (with the right subjects). About 25% of the time you can expect pretty good results. Below are some examples.

The algorithm did a pretty good job with this photo, but the skin tone on the arm and hands is a little uneven, and the dress lacks color.

The algorithm did a pretty good job with this photo, but the skin tone on the arm and hands is a little uneven, and the dress lacks color.

The algorithm works well with a variety of skin tones. Notice how it started to color the surfboard red but unfortunately wimped out. This is a photo that will need some hand correcting.

The algorithm works well with a variety of skin tones. Notice how it started to color the surfboard red but unfortunately wimped out. This is a photo that will need some hand correcting.

The algorithm didn't know what color to make the subjects' clothing, but it did a pretty good job on their skin tone and the water and sky in the background.

The algorithm didn't know what color to make the subjects' clothing, but it did a pretty good job on their skin tone and the water and sky in the background.

The artificial intelligence did a pretty good job with this photo, but I would make the stripes on the boy's shirt red and the tablecloth yellow.

The artificial intelligence did a pretty good job with this photo, but I would make the stripes on the boy's shirt red and the tablecloth yellow.

The AI has trouble coloring clothing, because it has no way of knowing what color it was originally, but this woman was actually wearing a colorful jacket. Also, the AI missed her hand.

The AI has trouble coloring clothing, because it has no way of knowing what color it was originally, but this woman was actually wearing a colorful jacket. Also, the AI missed her hand.

The algorithm captured the sky, the painters skin tone and even the blue of his bluejeans! Unfortunately, it missed his hand, but I will show how to easily fix and improve this picture later on.

The algorithm captured the sky, the painters skin tone and even the blue of his bluejeans! Unfortunately, it missed his hand, but I will show how to easily fix and improve this picture later on.

Examples of Okay Results

About 35% to 40% of the time, Colorize Photos will generate results that can only be considered "fair," or acceptable as a starting point for your own colorization efforts.

Usually, when Colorize Photos fails, it is because it fails to recognize the object or part of the object being colored. Because Colorize Photos applies a form of facial recognition, it often effectively colorizes faces but misses. limbs or parts of them such as arms, legs and hands. For some odd reason, while it may effectively colorize a face, it will often miss the ears!

Colorize Photos also tends to do a better job with outdoor and natural scenes then with indoor and man-made objects. This is because it has no way of knowing what color a car, a wall, a telephone or an item of clothing was originally.

Here are examples of such results.

Colorize Photo did a good job on the background foliage and the subject's skin tone, but unfortunately missed her hands, a frequent shortcoming of the algorithm.

Colorize Photo did a good job on the background foliage and the subject's skin tone, but unfortunately missed her hands, a frequent shortcoming of the algorithm.

Yellow was a good choice of color for this model's shirt, but the application was uneven. It could also have done a better job on her arm.

Yellow was a good choice of color for this model's shirt, but the application was uneven. It could also have done a better job on her arm.

As is often the case, Colorize Photos didn't know what to do with the subject's clothing, so it just skipped it.

As is often the case, Colorize Photos didn't know what to do with the subject's clothing, so it just skipped it.

The foliage in the background of this shot was actually filled with colorful flowers, but the algorithm had no way of knowing that, so it just made everything green.

The foliage in the background of this shot was actually filled with colorful flowers, but the algorithm had no way of knowing that, so it just made everything green.

The artificial intelligence did a good job with the grass and trees, but ignored the bicyclists entirely. This one will have to be finished by hand.

The artificial intelligence did a good job with the grass and trees, but ignored the bicyclists entirely. This one will have to be finished by hand.

Colorize Photo attempted to color only these astronauts' skin and hair, and it skipped everything else entirely, including one of the subjects' hands.

Colorize Photo attempted to color only these astronauts' skin and hair, and it skipped everything else entirely, including one of the subjects' hands.

The algorithm did a good job of coloring in the mountains and countryside and even correctly deduced, I think, that the foreground consists mostly of dirt. Unfortunately, it ignored the couple.

The algorithm did a good job of coloring in the mountains and countryside and even correctly deduced, I think, that the foreground consists mostly of dirt. Unfortunately, it ignored the couple.

The grass and bushes in the background were effectively colored here, but the woman's face was unfortunately missed.

The grass and bushes in the background were effectively colored here, but the woman's face was unfortunately missed.

The AI often does a better job with outdoor shots than with indoor shots. It knows what color grass, trees and the sky are but has no way of knowing whether a wall is beige or mauve.

The AI often does a better job with outdoor shots than with indoor shots. It knows what color grass, trees and the sky are but has no way of knowing whether a wall is beige or mauve.

Foliage and flesh tone are the only colors the algorithm ventured to guess in this shot, although it seems to have timidly given the dandelions in the background a touch of yellow.

Foliage and flesh tone are the only colors the algorithm ventured to guess in this shot, although it seems to have timidly given the dandelions in the background a touch of yellow.

The algorithm made the roses in this shot a bold shade of red, but unfortunately it also gave the stems the same hue. It skipped the other flowers. And why did it make one of the lady's hands green?

The algorithm made the roses in this shot a bold shade of red, but unfortunately it also gave the stems the same hue. It skipped the other flowers. And why did it make one of the lady's hands green?

The algorithm did a pretty good job on this photo, but unfortunately missed the cowgirl's hands, a common problem.

The algorithm did a pretty good job on this photo, but unfortunately missed the cowgirl's hands, a common problem.

Colorize Photos started to color red the shirt of the man on the right, but wimped out before it was done. It also missed the subjects' hands.

Colorize Photos started to color red the shirt of the man on the right, but wimped out before it was done. It also missed the subjects' hands.

Colorize Photos did a good job on the foliage and the horse, but it colored the hind legs green, and it skipped the face of the rider. I would also remove the red from the officer's trousers.

Colorize Photos did a good job on the foliage and the horse, but it colored the hind legs green, and it skipped the face of the rider. I would also remove the red from the officer's trousers.

Colorize Photos did a pretty good job with the wall in the background. It even correctly colored the wood paneling. Unfortunately, it missed the subjects' hands.A little manual retouching would greatly improve this.

Colorize Photos did a pretty good job with the wall in the background. It even correctly colored the wood paneling. Unfortunately, it missed the subjects' hands.A little manual retouching would greatly improve this.

Colorize Photos didn't know how to color these Polish girls' colorful costumes. It started to make the ribbons red, but ultimately tuckered out before it finished.

Colorize Photos didn't know how to color these Polish girls' colorful costumes. It started to make the ribbons red, but ultimately tuckered out before it finished.

Examples of Disappointing Results

It's important to approach Colorize Photos with realistic expectations. Sometimes (at least 30% of the time) the results of the algorithm are simply mediocre. However, even in such cases, it is often possible to use its results as a starting point for your own hand-colorization efforts. So that you don't get the idea that Colorize Photos always produces brilliant results, let's look at some disappointing efforts.

The AI did a good job on this photo – if you want everything to be pink or pale yellow. Also, it missed one of the models' hands.

The AI did a good job on this photo – if you want everything to be pink or pale yellow. Also, it missed one of the models' hands.

For some inexplicable reason, the AI had trouble recognizing this young lady's arms and colorizing them appropriately.

For some inexplicable reason, the AI had trouble recognizing this young lady's arms and colorizing them appropriately.

Similarly, the AI made this young lady's arms green. Perhaps it confused them with the background foliage.

Similarly, the AI made this young lady's arms green. Perhaps it confused them with the background foliage.

The green of the foliage seems to be seeping into this model's dress, turning it a lovely shade of greenish gray.

The green of the foliage seems to be seeping into this model's dress, turning it a lovely shade of greenish gray.

The AI tried with this photo, it really did! It appropriately colorized some of the cheerleaders' faces and legs, but missed others. Also, it had trouble distinguishing the pavement from grass, and started to color it green, but didn't finish the job

The AI tried with this photo, it really did! It appropriately colorized some of the cheerleaders' faces and legs, but missed others. Also, it had trouble distinguishing the pavement from grass, and started to color it green, but didn't finish the job

The AI had no idea what color to make this vendor's colorful tie-dyed apparel, so it gave up entirely.

The AI had no idea what color to make this vendor's colorful tie-dyed apparel, so it gave up entirely.

Off to a good start, but the AI missed one of the subject's faces and the other one's knees. Also, the grass could have been made green.

Off to a good start, but the AI missed one of the subject's faces and the other one's knees. Also, the grass could have been made green.

It's almost as if the AI had some inkling of what color these American flag should be, but it was timid it its execution.