yes - you need to balance the non-offending team's right to go quickly, with the YC player's right to come back on.
scenario
- blue have a player in the bin, 11 mins gone
- you award PK to Red
- there is a pause while red collect the ball / confer about what to do. ie they have NOT tapped and gone
- the blue YC player is standing on the the touchline
- you can just wave him on to the field. Why not? he's done his time.
Red have the right to tap and go, but if they don't do that, I don't think red have the right to keep him off the field....