I like that EXCEPT..how could he be a world class seeker and ride the broom without magic ability?!
Not sure, but maybe it's something that comes naturally to him... kind of like riding a bicycle, he's had the training so he'll never forget it. I don't recall if Rowling said that only the magically-inclined can ride brooms, it's just assumed so. Maybe it's not the case. Muggles don't consider it since it's all bad, pagan and make-believe.