The more government spends the less money there is for the taxpayers to put to better use when it comes to economic growth.
There is of course some feed back going on here. The higher unemployment goes the more government has to spend in compensation and benefits. However, that doesn't mean that overall spending has to increase; there is always the alternative of better budgeting choices.