Pelosi ran unopposed but number of members who voted against her could signal a messy battle when new Congress convenes