The fluctuation of the water surface causes refractive distortions that severely downgrade the image of an underwater scene. Here, we present the distortion-guided network (DG-Net) for restoring distortion-free underwater images. The key idea is to use a distortion map to guide network training. The distortion map models the pixel displacement caused by water refraction. We first use a physically constrained convolutional network to estimate the distortion map from the refracted image. We then use a generative adversarial network guided by the distortion map to restore the sharp distortion-free image. Since the distortion map indicates correspondences between the distorted image and the distortion-free one, it guides the network to make better predictions. We evaluate our network on several real and synthetic underwater image datasets and show that it out-performs the state-of-the-art algorithms, especially in presence of large distortions. We also show results of complex scenarios, including outdoor swimming pool images captured by drone and indoor aquarium images taken by cellphone camera.

Our overall distortion guided network (DG-Net) architecture consists of two sub-nets:

1. a convolutional network for estimating the refractive distortions (Dis-Net) for estimating the refractive distortions;
2. a distortion-guided generative adversarial network for restoring the distortion-free image (DG-GAN).

It is challenging to acquire fluid dataset with ground truth underwater distortion-free images using physical devices. We resort to physics-based modeling and rendering to synthesize a large fluid dataset for our network training. We use fluid equations derived from the Navier-Stokes to model realistic fluid surfaces and implement a physics-based renderer to simulate refraction images. Our dataset contains over 63K distorted refraction images, generated from 6354 unique distortion-free images (or reference pattern). We keep 10 consecutive frames per wave. For each refraction image, we provide the ground truth distortion-free image, distortion map, and height map of the water surface. We divide our dataset as 70% for training (43,600), 15% for validation (9980), and 15% for testing (9960). Sample images from our dataset are shown in the figure below.