Far be it from me to kill the magic of music, but...
Long story short, there are paired neurons related to hearing that are either in sync or out of sync based on harmonics, and when they are synced, the experience is soothing, and when they are out of sync, disturbing. This is a communicative aid found in most mammals and birds that allows one to distinguish between vocalizations of distress or vocalizations of encouragement. (Some reptiles have a primitive system like this as well, though their tonal perceptions are kinda "shifted" compared to ours.) In our case, it is key to language and the emotional expression behind words.
In this fashion, music communicates to us in a preverbal fashion.
Thus, while emotive memory is a factor (as >>9187615
suggests) it isn't required. Children and animals previously unexposed to music will be lulled or excited by it to the degree of hypnotic effect. (Not that some folks with certain brain disorders don't simply interpret all music as "noise".)
Memory is a factor in that the brain anticipates certain patterns as it is exposed to more and more music over time. And really, particularly in western culture, almost all music shares certain patterns:https://www.youtube.com/watch?v=JdxkVQy7QLM
When the brain runs into a pattern that it's not expecting, the effect similar to that of receiving a dissonant sound, and this is both perplexing and irritating, breaking the effect. This behavior of the pattern recognition system is not restricted to music, of course, but in this case, it takes the brain out of a meditative state, and puts it in analysis mode, and the two systems start fighting with one another, thus annoyance and irritation.
Not that the brain can't learn to like new music, it's just a matter of learning and falling into the new patterns and making new associations.
Long story long:https://www.podtrac.com/pts/redirect.mp3/audio.wnyc.org/radiolab/radiolab042106.mp3
(In audio form, of course.)