If you want to do it for all buttons in the component you can do this:
:host { // this particular example is for icon button @include mat.icon-button-density(-3);}
This way you are still of course putting it on a 'parent' of the button - but it doesn't require a clumsy explicit wrapper div just to achieve that.
Assumes you're using style encapsulation.