They literally state in the narration following the invasion that the whole reason they don’t want regular stuff anymore and abandoned “anything cute” is because it attracts the aliens. This is directly after you watch the duplo aliens literally steal lego pieces.
Hell, this is the first conversation Wildstyle has with Emmet right after.
It follows with the sister stealing a handful of his figures and later on him complaining about her always taking his stuff.
And then, Rex speaks of having to see his friends in the hands of the sis.
How can you miss it?