In this paper, we study the spatial averages of the solution to the parabolic Anderson model driven by a space-time Gaussian homogeneous noise that is colored in both time and space. We establish quantitative central limit theorems (CLT) of this spatial statistics under some mild assumptions, by using the Malliavin-Stein approach. The highlight of this paper is the obtention of rate of convergence in the colored-in-time setting, where one can not use Ito calculus due to the lack of martingale structure. In particular, modulo highly technical computations, we apply a modified version of second-order Gaussian Poincaré inequality to overcome this lack of martingale structure and our work improves the results by Nualart-Zheng (Electron. J. Probab. 2020) and Nualart-Song-Zheng (ALEA, Lat. Am. J. Probab. Math. Stat. 2021).