I thought season 2 the weakest, apart from maybe the last. In hindsight, it's obvious that all that tail end second group stuff was a filler option, as pretty much none of it was ever really tied into the overall narrative. The thing is, even upon first watch, I felt that that was the case - it just reeked of filler. I almost gave up on watching it at that stage, but it managed to draw me back in by the end of season three.
Maybe it didn't matter, but Vaughan left before the 6th season, and he wrote some of the best episodes (I forget which now, but he did).
I mentioned that somewhere here (maybe earlier this thread). He wrote one in season five with a lot of unLock talking with Ben that was really good. And some of the best from earlier seasons too. It isn't the only TV show where some of the more capable writers on the staff weren't ultimately in charge of the overall direction. (For example, the X-Files best episodes were usually written by people other than Carter or his main showrunners.)