He was in the wand (specifically the crystal) the entire time. It was reading Eclipsa's chapter that allowed him to possess Ludo and move from the crystal to inside his mind, which is why in the last episode Ludo said the wand no longer worked.
Since the book was blank, Toffee seems to have somehow absorbed the spells and Glossaryck is somehow connected to this.
I'm still confused how Ludo's wand was made. Obviously he partially regenerated around the rock and crystal from the bone, but how did the crystal get on/in the rock in the first place? I guess Toffee could effect some magic through the crystal since it is magical itself; probably how Ludo was led to it too.