I think the problem with the show, is that they made it all about the viewer control. If you watch the webisodes, the live feeds on Monday night, and the show on Monday night. You see it's all centered around the the viewer's votes. You can't build a show like that. Anytime drama remotely shows up in the show, through a fight they only show like 5 seconds of it. On the feeds if the cast starts getting into a heated debate (fight), the big O, will cut them off and get them onto something else.
It seems that in a effort to build a show that was different than BB they actually went too far and made a show that is so different that its actually turning people off.