QuoteEDIT: Just remembered... Go to Admin -> Members -> Permissions -> Settings. Make sure to enable the two settings that are there or you will not be able to deny permissions nor set post count based groups.
1. Enable the option to deny permissions.
2. Enable permissions for post count based groups.
Select "No" to enable group to have own permission set."