I've pondered what I can say on this subject.
Okay, we have a number of different kinds of situations in which dragons can be bonded to a person.
For example, children presented with dragon eggs and hatchlings. I can say that in every case in which a healthy hatchling was born, no Targaryen was ever rejected by it, so far as I can recall and find through a cursory search of what I have.
Then we have older dragons, either ones that once had riders or perhaps never had riders. These can be dangerous -- especially if old and ill-tempered -- even to a Targaryen, but there's the sense that it's even worse for non-Targaryens (but hey, I can see the argument that it's just a recognition thing -- dragons may be more tolerant of silver-gold, purple-eyed people through long association with them). The belief is that you have to be of Targaryen blood -- or, lets be more general about it -- dragonlord blood to have a chance with these. But it's impossible to prove this 100% based on just the information provided in the texts, because you can never know if a "black swan" will or has shown up. That said, there's certainly a widespread belief that all dragonriders have had dragonlord blood somewhere in their lineage.
I might suggest that it's more specific than dragonlord lineage, though. If it's a genetic thing, it's entirely possible that the particular genes required for a dragon to "sense" that you are an acceptable rider might skip a person -- they may look 100% Targaryen, and simply not have that particular thing. I can't think of any Targaryen who was actually rejected by a dragon, of course, but that doesn't necessarily mean much. But we may then see that in the dragonseeds, the converse can be true -- they may have very little dragonlord blood, but what portion they have happens to be exactly what's needed to permit them to ride a dragon.
The best evidence, I think, for some magical genetic component to dragonriding is simply the fact that the dragonlords in particular practiced incest -- other Valyrian nobility did as well, I guess, but it seems the dragonlords really led the way. Which suggests that they may well have believed that there was a genetic component, one they wanted to keep strictly to the family. (Alternatively, I guess you can argue it's all just inheritance matter -- marrying family members out might lead to their offspring, members of other families entirely, might be able to claim your dragons for themselves).
For my part, I am quite convinced that GRRM views this business as being a kind of magical fantasy genetics thing -- there's literally something "in the blood". And from that, I am convinced that having no dragonlord blood means that you couldn't ever hope to ride a dragon.
The "dragon dreams" of Daenys the Dreamer, Daeron the Drunken, or Daemon II Blackfyre seems to match well enough with the idea that there's a distinctive, inherited kind of magic in the Targaryen line, possibly a broader dragonlord thing, we don't know. And given the prominence of dragons in such dreams -- even from people who exist after the dragons are dead, who never saw a living one -- it just seems natural to suppose that there's something "there" to it.
I would guess the more dragonlord blood you have, the likelier you'll have those aspects necessary for various expressions of Targaryen/dragonlord magic, whether it's prophetic dreams or sorcery or dragonriding, but having just a drop could in itself -- by the grace of the gods and the whim of GRRM -- be sufficient.
***
Not just "a drop", but the "right drop", I guess.
Are there lots of them? Well, if we accept the dragonseeds had in some cases very distant drops but still had the "right drop", it seems likely that the dragonriding gene is probably pretty well-spread. Like, every noble family who can trace Velaryons or Celtigars or other dragonstone houses can probably notionally have a chance at having that "right drop".
The other aspect of all this is that GRRM's as much as admitted that there's a certain fantasy genetics even in other families -- like Robert's always-black-haired-and-blue-eyed-kids. That's obviously not a trait that necessarily passes to all his offspring, or to the offspring of his ancestors with similar weird dominant traits, because otherwise Westeros should be over-run with black-haired-and-blue-eyed people.
Basically, it's all authorial fiat. George decides who can ride a dragon and who can't, by whatever rules he wants. I think the rule here is that, yes, there's something special about dragonlord blood, but having dragonlord blood in and of itself isn't necessarily enough to convey that specialness.
ETA: I see another argument comes from deformed infants with scales and wings and such. While it's easy to suppose that this is further proof that there's actual magical genetic stuff going on, I wouldn't absolutely rule out the possibility that proximity to dragons -- who are, after all, magical creatures -- could lead to strange things to embryos in the womb. Like ... magical radiation.
***
Про Крапиву, Ульфа и Хью.
I believe they have Targaryen blood besides things not in the texts, yes, related to the origins of the Valyrian-dragon connection. I think The World of Ice and Fire will have one or two tidbits on the subject.
***
The dragonlord aspect of thing is, I believe, necessary -- it's what gets your foot through the door. It's not enough to get you everything. Again, as I noted, it's understood older dragons -- i.e. not hatchlings -- can be rather a lot more difficult to get to accept someone. But in all cases, there's reason to believe that all those accepted had dragonlord descent somewhere in their lineage.
***
That's fine, but there's evidence that the dragonlords were dragonlords for magical reasons tied to dragons. Again, note that the key element in all the known prophetic Targaryen dreamers whose dreams we get any real substantial description of are dragon symbology and so on, even after dragons are long gone from the world.
Jojen and Bran don't dream of wolves or trees or whatever all the time -- Melisandre's flames don't just show her weirdly shaped shadows and fire. The Targaryen dreams we know of always seem to revolve around dragons -- the deaths of them, the hatching of them, and so on. Always, of course, symbolic, but still. It's a strange magic that constantly uses dragons as symbols without actually having anything to do with dragons at all.... unless, of course, the dragon dreams and the dragon riding all have a common source, namely the magical genetics of dragonlords.
***
Well, we don't have an older, riderless dragon actually killing any Targaryens or anything. I think it's more... consider some of he behavior we see from Dany's dragons -- they can be snappish, demanding, aggressive. You don't go messing with a bad-tempered old dragon, generally. They're not pets.
***
Oh, that. No, I didn't mean the .... je ne sais quois of dragonlord genes are what make dragon eggs hatch. I mean, I assume the failure of those later eggs relates to the general degradation of the dragon gene pool following the destruction of the Dance, and whatever else was going on (Dragonpit, the falling off of magic, etc.).