He started out as the ghost of a dead knight who could summon his spectral horse. By the time he was approved for production he had become an undead centaur-like being and all the concept art reflected that. He migrated to the Shadow Isles fairly early in development - Riot has been working behind the scenes to develop the idea of undeath in Valoran and they wanted Hecarim to be representative of the tone of the Shadow Isles.
It has been a very, very long time since he was actually alive, but his form suggests that there may have been a race of centaur-like beings in the past. He hasn't retained any memories from his life and he doesn't particularly care about what he used to be.
Hecarim's second joke, "Another blue ribbon for Hecarim," is a reference to the blue ribbon, a term used to describe something of high quality. They are often granted as an award to the winner of a contest showcasing horses or other livestock.
On the Brazilian server, this joke is replaced with a reference to Pôneis Malditos, a Nissan advertising video that has became popular on Brazil as an internet meme.
Hecarim shares a quote with Battlecast Prime Cho'Gath, "Crush them underfoot."
Hecarim shares a quote with Infernal Nasus, "Despair."
His halberd is a reference to the video game Guitar Hero.
The South American version of "Arcade Hecarim" skin was wrongly translated as "Hecarim of Arcadia" ("Hecarim de Arcadia"). Arcades in Spanish are either called "Arcades" or "Fichines", so it doesn't make sense in Spanish either. It may be a nonsensical reference to "Lawrence of Arabia" ("Lawrance de Arabia").
Despite this, however, the two are still listed as friends rather than rivals on the official League of Legends website.
Hecarim existed longer than Yorick. Hecarim's shape suggests that he might've been something that no longer even exists in Valoran, while Yorick can be recognized as human.
On the League of Legends website, both Hecarim and Thresh used to be listed as rivals with Ahri; this might be due to Ahri being a fellow harvester of souls which would interfere with the Shadow Isles' goals.