We exploit insights into the geometry of bosonic and fermionic Gaussian states to develop an efficient local optimization algorithm to extremize arbitrary functions on these families of states. The method is based on notions of gradient descent attuned to the local geometry which also allows for the implementation of local constraints. The natural group action of the symplectic and orthogonal group enables us to compute the geometric gradient efficiently. While our parametrization of states is based on covariance matrices and linear complex structures, we provide compact formulas to easily convert from and to other parametrization of Gaussian states, such as wave functions for pure Gaussian states, quasiprobability distributions and Bogoliubov transformations. We review applications ranging from approximating ground states to computing circuit complexity and the entanglement of purification that have both been employed in the context of holography. Finally, we use the presented methods to collect numerical and analytical evidence for the conjecture that Gaussian purifications are sufficient to compute the entanglement of purification of arbitrary mixed Gaussian states.