I would think it would be a let down to the house guests that di dnot make it to be able to be let into the house to host challeges, what a smack in he face. "See what your missing".
Though there are few of the remaining that did not make it that I would be interested in seeing, it would be an interesting twist to see how they would bring some of them into the house on a more permanent basis to shake things up.