Colin Baker left Doctor who because he was replaced with Sylvester McCoy. The 6th Doctors time on television saw the shows lowest ratings ever the show to that point.
That coupled with the backlash that was caused by his suddenly very brash, harsh, egocentric personality.. and you have a recipe for replacement.
Eh, no. Cause when McCoy toke over, the Sixth Doctor had only a "normal death" instead of something else.
Michael Grade didn't like Doctor Who, and wanted to cancel it. So Colin Baker was made a scapegoat. In addition, Grade pointed to the "high levels of violence" as justification for his "cleaning up television". Despite FAR more violent stuff airing all the time.
The ratings were not at Tom Baker levels but they were good, and certainly FAR better than what followed with Mccoy. Basically, Colin Baker(who was a superb Doctor btw) just found himself in the wrong place at the wrong time. Joh Nathan Turner had lost the plot, and Michael Grade hated Doctor Who, and was looking for a reason to cancel it. Anyone who has listened to CB's Big Finish Audios realises that calling him "shitty" is the outburst of an imbecile.
Bad ratings for the show and bad reviews.
After season 22, the show was nearly canceled. BBC management agreed to instead put the show on an 18-month hiatus, but season 23 didn't do much better. So, the BBC insisted that Colin Baker be replaced and the show be "refreshed".