Website Conservatoire national des arts et métiers

Computer science laboratory

Location : Cnam, Paris or UBS, Vannes (France)
Salary : ≈ 600€/month
Contract : Internship (4 to 6 months)
Starting date : Flexible in 2023

Research topic

Earth Observation through satellite imagery is a major tool for geoscience. However, there is a dilemma on whether to deploy systems that acquire many images at high frequency but with a low spatial resolution or systems with high spatial resolution but few revisits. The Sentinel-2 constellation, operated by the European Space Agency (ESA), leans towards high frequency for better temporal monitoring of dynamic phenomena. Yet, its 10m/px resolution is often not enough in urban applications since buildings, roads, and sparse vegetation are barely visible in the images. Conversely, commercial satellites, such as SPOT-6/7, often embark very high-resolution sensors. For example, the French mapping agency (IGN) distributes a yearly high-resolution cloud-free mosaic of the entire French metropolitan area. These panchromatic SPOT-6/7 images at a 1.5m/px resolution are helpful for precise land cover and land use mapping but are produced only once a year.

This research work aims to bridge the gap between those two data sources by investigating super-resolution. Super-resolution is an image processing technique aiming to increase the resolution of an image. It does not use a reference high-resolution image, but instead relies on prior knowledge regarding the structure of the data. Using deep learning for super-resolution has been frequently investigated in the last years [1], starting with Convolutional Neural Networks(CNN) [3], then later using Generative Adversarial Networks (GAN) [5, 8]. Nowadays, diffusion models [9, 6], already state-of-the-art generative models for image synthesis, have established themselves as strong candidates for the next generation of super-resolution algorithms.

Some works have transposed those deep architectures to remote sensing [4, 10], with two main shortcomings. First, most focus on single-image resolution, i.e., transforming a single image into a more detailed one. But low-resolution remote sensing data draws its strength from the temporal information contained in the satellite image time series (SITS). Second, super-resolution systems are mostly trained on artificial pairs of (low resolution, high resolution) images where the LR image is a downsampled version of the HR reference. This means that training sets and benchmarks are mostly synthetic and do not actually inform us of the performance of super-resolution systems on actual data. To overcome these issues, new benchmark datasets have been proposed such as Sen2Venus [7] and WorldStrat [2]. The latter includes pairs of a sequence of low-resolution Sentinel-2 images and a high-resolution Spot-6/7 image acquired worldwide. This dataset opens opportunities for the development of new approaches relying on modern state-of-the-art super-resolution techniques such as diffusion models.
The goal of this research project is (i) to leverage the temporal information contained in satellite time series to improve the super-resolution process, and (ii) to train deep diffusion models to power a super-resolution system on real data.

Objectives
The research intern will perform the following tasks:

searching and summarizing the literature for suitable multi-image super-resolution approaches, diffusion models for super-resolution, and super-resolution of temporal data,
collecting and analysing the WordStrat dataset,
implementing and adapting state-of-the-art approaches, such as SRDiff [6], to SITS,
investigating techniques to exploit the temporal structure in SITS, for example by using attention mechanisms and temporal blocks in the conditional encoder of diffusion models.

Applicant profile
The ideal applicant is pursuing an MSc. in Computer Science, Artificial Intelligence or Computer Vision or an equivalent degree (e.g. engineering diploma).

A good grasp of the fundamentals of machine learning and deep learning for computer vision is expected.
The candidate should have an interest in scientific research and good written and oral communication skills.
Knowledge of the Python programming language is a must, including some level of experience with at least one deep learning framework (PyTorch, Keras, TensorFlow, JAX…).
A first experience with time series, image processing or generative models is a plus.
All applications, independently of previous experience, will be considered, provided that the candidate’s motivation and profile fit the internship topic. 

French is not required but can help with everyday life.

Where you will work
This position is for an internship from 4 to 6 months, with a flexible starting date in 2023. The internship will either take place at the Center for research and studies in computer science and communications (Cédric), the computer science laboratory of the Conservatoire national des arts et métiers (Cnam) in Paris or at the Institute for Research in Informatics and Random Systems (Irisa), the computer science laboratory of the University of South Brittany in Vannes.
Depending on their preference, the intern will join the Complex Data, Machine Learning and Representations team in Paris or the Environment observation with complex imagery team in Vannes. In either cases, gratification is approximately 600€/month as imposed by the law for public institutions.

How to apply
Please send your application (CV and a short motivation letter) to: nicolas.audebert@cnam.fr, charlotte.pelletier@univ-ubs.fr 

References
[1] Saeed Anwar, Salman Khan, and Nick Barnes. “A Deep Journey into Super-resolution: A Survey”. In: ACM Computing Surveys 53.3 (May 28, 2020), 60:1–60:34.
[2] Julien Cornebise, Ivan Oršolić, and Freddie Kalaitzis. The WorldStrat Dataset: Open High-Resolution Satellite Imagery With Paired Multi-Temporal Low- Resolution. Zenodo, July 2022.
[3] Chao Dong et al. “Learning a Deep Convolutional Network for Image Super-Resolution”. In: Computer Vision – ECCV 2014. Lecture Notes in Computer Science. Cham, 2014, pp. 184–199.
[4] Juan Mario Haut et al. “Remote Sensing Image Superresolution Using Deep Residual Channel Attention”. In: IEEE Transactions on Geoscience and Remote Sensing 57.11 (Nov. 2019), pp. 9277–9289.
[5] Christian Ledig et al. “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 2017, pp. 105–114.
[6] Haoying Li et al. “SRDiff: Single image super-resolution with diffusion probabilistic models”. In: Neurocomputing 479 (2022), pp. 47–59.
[7] Julien Michel et al. SEN2VENµS, a dataset for the training of Sentinel-2 super-resolution algorithms. Version 1.0.0. Zenodo, May 2022.
[8] Nathanaël Carraz Rakotonirina and Andry Rasoanaivo. “ESRGAN+ : Further Improving Enhanced Super-Resolution Generative Adversarial Network”. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (May 2020), pp. 3637–3641.
[9] Chitwan Saharia et al. “Image Super-Resolution Via Iterative Refinement”. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2022), pp. 1–14.
[10] Kexin Zhang, Gencer Sumbul, and Begüm Demir. “An Approach to Super-Resolution of Sentinel-2 Images Based on Generative Adversarial Networks”. In: 2020 Mediterranean and Middle-East Geoscience and Remote Sensing Symposium (M2GARSS) (Mar. 2020), pp. 69–72.

To apply for this job email your details to nicolas.audebert@cnam.fr