Tim Arends has for many years been an interested observer in the progress of artificial intelligence in areas such as image manipulation.
Adobe Photoshop's Newest AI Feature
There's a new feature in Adobe Photoshop and Photoshop Elements. It goes beyond removing red-eye, obliterating background elements and swapping people's heads around. It's called Colorize Photo, and it uses artificial intelligence to turn your old black-and-white photographs into vibrant color images with the click of a button.
So how well does it work? Let's look at some photos and see what it does well, as well as identify a few weaknesses of the algorithm. We will start with the most excellent results and work our way down to fair or poor results. Knowing this, you can use the feature more effectively in coloring your own black-and-white photos.
Note: The below are royalty-free images from Art Explosion 750,000 Images clip art collection. They were originally color photos which I converted to grayscale for the purposes of testing shown here.
Examples of Excellent Results
Below are some examples of excellent results from Colorize Photos. None of these have had any retouching of my own applied. You can expect results this good about 10% of the time.
Colorizing Photos with Artificial Intelligence
So you've seen some examples of Colorize Photos in action. How does it work?
You're probably familiar with Instagram-style filters. These take your images and apply various effects to them, such as giving them a slightly yellowed, old-time vintage effect. Instagram filters are known as "destructive" filters, because, in order to give photos that old-timey look, they throw away data from the image.
You're probably also familiar with the process of colorizing old movies, which uses computers to add color to a vintage film frame-by-frame. The process is made possible by computers, but it still requires a lot of human effort. The initial frame of each scene must be colored by hand. The computer then takes this information and extends it to the other frames of that scene. But the process has to be started and supervised by humans. Colorizing a still image by hand is a similarly laborious process.
However, Photoshop's Colorize Photos promises to turn a photo from dull black and white to vibrant color with a single click of a button. There's no way a piece of software can do this without employing what is known as artificial intelligence, or the science of making computers do things that previously required human judgment and expertise.
If you have performed a Google image search, you have seen a similar technology in action. Drag one of your own photos to the search box, and if the photo exists on the Internet, Google will usually find it. If it doesn't, Google will show images that are thematically similar -- a dancing woman, a couple on the beach, an automobile, etc.
Google is employing a form of artificial intelligence called image recognition, which allows it to identify the subject of your photo and find similar photos. Sites like Pinterest, Apple Photos and others have a similar feature.
How Colorize Photos Works
So how does Colorize Photo work? Through a process of training artificial intelligence known as deep learning. The developers expose the algorithm to thousands of photos and eventually the software learns that a tree is usually green, the sky is usually blue, and so on.
This is a far greater challenge than it seems. Although you may think a tree is a tree is a tree, the reality is that no two trees are alike, and even the same tree can look very different at different times of day and from different angles. So the challenge of image recognition was an incredible one for developers to surmount, but over the years they have made great strides in the process.
Photoshop's Colorize Photos algorithm employs a similar process. In order to work, it must scan the photo and do at least some analysis of the subject matter so as to determine which color should go where. I believe the algorithm also uses facial recognition to identify faces and uses these as a starting point in determining that the photo contains a person, as well as in coloring the rest of the photo. It often works particularly well with portraits.
By the way, unlike Google's image recognition, Photoshop's Colorize Photos algorithm does not require an Internet connection. I tested this by unplugging my computer from the Internet and running the algorithm, and it worked! That means the "brains" of the AI are built into the software and reside directly on your computer when you download and install the program.
Examples of Pretty Good Results
The above examples are of stellar results from the colorizing artificial intelligence. You can expect results that good only about 10% of the time (with the right subjects). About 25% of the time you can expect pretty good results. Below are some examples.
Examples of Okay Results
About 35% to 40% of the time, Colorize Photos will generate results that can only be considered "fair," or acceptable as a starting point for your own colorization efforts.
Usually, when Colorize Photos fails, it is because it fails to recognize the object or part of the object being colored. Because Colorize Photos applies a form of facial recognition, it often effectively colorizes faces but misses. limbs or parts of them such as arms, legs and hands. For some odd reason, while it may effectively colorize a face, it will often miss the ears!
Colorize Photos also tends to do a better job with outdoor and natural scenes then with indoor and man-made objects. This is because it has no way of knowing what color a car, a wall, a telephone or an item of clothing was originally.
Here are examples of such results.
Examples of Disappointing Results
It's important to approach Colorize Photos with realistic expectations. Sometimes (at least 30% of the time) the results of the algorithm are simply mediocre. However, even in such cases, it is often possible to use its results as a starting point for your own hand-colorization efforts. So that you don't get the idea that Colorize Photos always produces brilliant results, let's look at some disappointing efforts.