I've speculated this to be the actual cause of people losing their sense of smell and taste rather than COVID-19. As you might know, taste is highly dependent on smell, and if you're deficient in it your taste is dampened significantly. Silica dust in your respiratory tract, ESPECIALLY in your nose, will damage mucous membranes and in the case of your olfactory bulbs, cause a loss of smell.
One thing of note is that the olfactory bulb, as far as I know, is one of the few parts of your body capable of regenerating nerve tissue. It could be a possible explanation for recovery, but I'd need to do more research. Also, depending on the type and size of the fibers, cells might be able to encapsulate and digest them, or you could just expel them via sneezing.