Long-term control of SARS-CoV-2 outbreaks depends on the widespread coverage of effective vaccines. In Australia, two-dose vaccination coverage of above 90% of the adult population was achieved. However, between August 2020 and August 2021, hesitancy fluctuated dramatically. This raised the question of whether settings with low naturally derived immunity, such as Queensland where less than 0.005% of the population is known to have been infected in 2020, could have achieved herd immunity against 2021's variants of concern. To address this question, we used the agent-based model Covasim. We simulated outbreak scenarios (with the Alpha, Delta and Omicron variants) and assumed ongoing interventions (testing, tracing, isolation and quarantine). We modelled vaccination using two approaches with different levels of realism. Hesitancy was modelled using Australian survey data. We found that with a vaccine effectiveness against infection of 80%, it was possible to control outbreaks of Alpha, but not Delta or Omicron. With 90% effectiveness, Delta outbreaks may have been preventable, but not Omicron outbreaks. We also estimated that a decrease in hesitancy from 20% to 14% reduced the number of infections, hospitalizations and deaths by over 30%. Overall, we demonstrate that while herd immunity may not be attainable, modest reductions in hesitancy and increases in vaccine uptake may greatly improve health outcomes. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.