We construct a colored operad whose category of algebras is the category of algebraic quantum field theories. This is achieved by a construction that depends on the choice of a category, whose objects provide the operad colors, equipped with an additional structure that we call an orthogonality relation. This allows us to describe different types of quantum field theories, including theories on a fixed Lorentzian manifold, locally covariant theories and also chiral conformal and Euclidean theories. Moreover, because the colored operad depends functorially on the orthogonal category, we obtain adjunctions between categories of different types of quantum field theories. These include novel and interesting constructions such as time-slicification and local-to-global extensions of quantum field theories. We compare the latter to Fredenhagen’s universal algebra.