Analytic solutions to HJB equation in mathematical finance are relatively hard to come by, which stresses the need for numerical procedures. In this paper we provide a self-contained exposition of the finite-horizon Markov chain approximation method as championed by Kushner and Dupuis. Furthermore, we provide full details as to how well the algorithm fares when we deploy it in the context of Merton type optimisation problems. Assorted issues relating to implementation and numerical accuracy are thoroughly reviewed, including multidimensionality and the positive probability requirement, the question of boundary conditions, and the choice of parametric values.