16.02.24 - SpectralGPT: Next-Generation Foundation Model for Hyperspectral Imaging

"- ChatGPT, you said? I heard of it already.
- No, SpectralGPT."

Hashtag#SpectralGPT is a new foundation model for multispectral and hyperspectral images developed by Hong et al. Foundation models are trained on large datasets, which makes possible to apply them for different tasks. While there are several foundation models trained for RGB images, there is no such tool for multispectral and hyperspectral images.

As spectral foundation model, SpectralGPT is the first foundation model specifically developed for multispectral imagery, aware of spatial-spectral coupling and spectral sequence. It has been trained on a large-scale dataset comprising over a million images of different sizes in height, width and number of channels derived from Sentinel-2 imagery. Pre-training and implementation enabled it to process images of varying sizes and spectral composition.

When evaluated for several tasks (single-label scene classification, multi-label scene classification, semantic segmentation, change detection and geo-characteristics recoverability), SpectralGPT performed often better than the state-of-the-art models. In particular, benchmarking with EuroSAT dataset (27'000 Sentinel-2 images from 34 European countries classified in 10 land use classes) for single-label scene classification showed that SpectralGPT obtained the best accuracy (99.15) against state-of-the-art models (ResNet 50 (96.72), SeCo (97.23), ViT (98.73), ViT-22k (98.91), SatMAE(99.09)).

Furthermore, semantic segmentation was also benchmarked on a dataset for Munich and the surrounding area (Sentinel-2 imagery, segmentation mask for 13 land use and land cover classes). The illustration below shows the results of semantic segmentation on sample images.

In further work, the authors aim to increase the size and diversity of the training dataset, as well as develop the model for additional tasks.

Source (content and image): Hong, Danfeng, Bing Zhang, Xuyang Li, Yuxuan Li, Chenyu Li, Jing Yao, Naoto Yokoya, et al. “SpectralGPT: Spectral Foundation Model.” arXiv, November 25, 2023.