I'd say you identified the issue right there. It seems to be your "material effect" that you need to work on, of when to blow or when to play away, and think less about the count.
If possible try and get your games videos and you can then see how quick you're on the trigger and how technical you might be. Then compare that with your better/higher referees in your area and what they "let go" and learn from them.