Any technology requires precise benchmarking of its components, and the quantum technologies are no exception. Randomized benchmarking allows for the relatively resource economical estimation of the average gate fidelity of quantum gates from the Clifford group, assuming identical noise levels for all gates, making use of suitable sequences of randomly chosen gates. In this work, we report significant progress on randomized benchmarking, by showing that it can be applied individually on a broad class of quantum gates outside the Clifford group, even for varying noise levels per quantum gate. This is possible at little overhead of quantum resources, but at the expense of a significant classical computational cost. At the heart of our analysis is a representation-theoretic framework which we bring into contact with classical estimation techniques based on bootstrapping and matrix pencils. We demonstrate the functioning of the scheme at hand of benchmarking tensor powers of T gates. Apart from its practical relevance, we expect this insight to be relevant as it highlights the role of assumptions made on unknown noise processes when characterizing quantum gates at high precision.