It isn't about fallibility, it's about authority: Christ's authority. As Church members, we follow the apostles because Christ, not we ourselves, called them to serve. They are Christ’s representatives, not ours. They will be judged for the way they use their authority, but they will not be judged by us. Christ gives the authority to judge and condemn His apostles only to the First Presidency, not to the members of His Church. And so we follow them, with a conviction born of the Holy Ghost that they are indeed Christ's apostles, but without regard to their fallibility or infallibility.