Evaluating Text-to-Image GANs Performance: A Comparative Analysis of Evaluation Metrics

Abstract

Generative Adversarial Networks (GANs) have emerged as powerful techniques for generating high-quality images in various domains but assessing how realistic the generated images are is a challenging task. To address this issue, researchers have proposed a variety of evaluation metrics for GANs, each with its own strengths and limitations. This paper presents a comprehensive analysis of popular GAN evaluation metrics, including FID, Mode Score, Inception Score, MMD, PSNR, and SSIM. The strengths, weaknesses, and calculation processes of these metrics are discussed, focusing on assessing image fidelity and diversity. Two approaches, pixel distance, and feature distance, are employed to measure image similarity, while the importance of evaluating individual objects using input captions is emphasized. Experimental results on a basic GAN trained on the MNIST dataset demonstrate improvement in various metrics across different epochs. The FID score decreases from 497.54594 at Epoch 0 to 136.91156 at Epoch 100, indicating improved differentiation between real and generated images. In addition, the Inception Score increases from 1.1533 to 1.6408, reflecting enhanced image quality and diversity. These findings highlight the effectiveness of the GAN model in generating more realistic and diverse images with training progression.  However, when it comes to evaluating GANs on complex datasets, challenges arise, highlighting the need to combine evaluation metrics with visual inspection and subjective measures of image quality. By adopting a comprehensive evaluation approach, researchers can gain a deeper understanding of GAN performance and guide the development of advanced models.