I’ve only had that problem with YouTube Music if I’m trying to listen from inside YouTube itself (via clicking Music in YouTube’s sidebar). Both the actual Music app and webpage both seem to play the audio-only versions, not the music video versions. I’ve got plenty of songs that if I try listening through YouTube will have the video version (often with end title cards and whatnot).