Input image (actual size)

The results of the algorithm. Mouseover to compare with the standard technique  used in Photoshop - bicubic interpolation.

Click to see more results on textures.

You can search over 2 billion images on the web, but most of them are virtually worthless. Google calls images which are 800 pixels wide "large." If you want to print that image at a decent resolution, it's going to be about 2 inches across.

I'm looking for ways to take low-resolution images and turn them into high-resolution images. The trouble is, the details just aren't present, so no matter what you do with the data you already have, new details aren't going to show up. One good answer to this is to copy details from other images. The trick is knowing what details to copy.

I've created a system that chooses the most likely high-resolution patch to augment any given low resolution patch in an image. The program is fed numerous examples and is designed to build up the details of a high-resolution image based on details from patches in its training set.

Puzzle solving, Kung Fu style

There is a scene in the TV series "Kung Fu" where the young acolyte Caine is given the task of assembling a puzzle of a map of the world.  He finds, though, that when the pieces are turned over on their backs, they form the picture of a man.  Since he knows what a man is supposed to look like, this is a lot easier. In the same way, we assemble low resolution patches (we know what the low resolution image is supposed to look like) in order to place high resolution patches.

Background

Single-image super-resolution is the problem of creating a high-resolution image from a low-resolution one. (It is also known as expansion, resolution enhancement, or zooming.)

Simple interpolation methods perform adequately when the dimensions of the image are increased by less than 100%. The most commonly used methods are linear and bicubic interpolation. Higher order poynomial splines perform poorly at the edges of shapes in the image, where the curves overshoot the data points.

Adaptive interpolation methods, which maintain sharp edges, can reduce blurring and ringing artifacts. These methods weight neighboring data points differently depending on the gradient, spreading color along edges but not across them. But at best these interpolation methods simply preserve the high frequency information that is already present. To get more realistic results or higher levels of resolution, it is necessary to introduce new high frequency information.

There are three places the new information could come from. One is to use details of the image itself at the smaller scale. This is the idea behind fractal image compression. This is unlikely to work well except on images that happen to be fractals, such as rocks or clouds. Even then, the image may not contain enough samples to provide realistic details.

A second possibility is generating the details algorithmically. The problem with this approach is a lack of any kind of theory of what kind of detail to generate.

The third possibility is to introduce information from another source, basing details on high-resolution patches whose low-resolution version is similar to the source patch. In the last 5 years, researchers have begun to explore this idea. The program described here is an implementation of this third approach.