Old E-Soul was the primary antagonist of the second arc. Though the first "X" title holder in history who defeated Zero, he gradually fell out of favor with the public, culminating in him being replaced. (Source: To Be Hero X Wiki)