My understanding is that as you get fatter and fatter, your adipocytes store more and more fat and grow bigger and bigger, but also, they divide and greatly increase in number.
When you lose weight, the adipocytes lose fat and get smaller and smaller, but they don't go away through autophagy and instead stick around, not only making you flabbier, but also possessing the potential to quickly rebound in size given enough of a caloric surplus.