The rude part would be, that your response time to verbal questions from your opponents should decrease because you actually don't hear them. Another point is, that it is unpolite to listen to someone/thing else while you should listen to your opponent. Third and last it's sort of paradox to say, that you can concentrate better on the game while you are using parts of your concentration listening to music...
On the contrary, I like listening to music when I build a sealed deck.