Well, the short answer is that the laws of inheritance in the Seven Kingdoms are modelled on those in real medieval history... which is to say, they were vague, uncodified, subject to varying interpertations, and often contradictory.
A man's eldest son was his heir. After that the next eldest son. Then the next, etc. Daughters were not considered while there was a living son, except in Dorne, where females had equal right of inheritance according to age.
After the sons, most would say that the eldest daughter is next in line. But there might be an argument from the dead man's brothers, say. Does a male sibling or a female child take precedence? Each side has a "claim."
What if there are no childen, only grandchildren and great grandchildren. Is precedence or proximity the more important principle? Do bastards have any rights? What about bastards who have been legitimized, do they go in at the end after the trueborn kids, or according to birth order? What about widows? And what about the will of the deceased? Can a lord disinherit one son, and name a younger son as heir? Or even a bastard?
There are no clear cut answers, either in Westeros or in real medieval history. Things were often decided on a case by case basis. A case might set a precedent for later cases... but as often as not, the precedents conflicted as much as the claims.
In fact, if you look at medieval history, conflicting claims were the cause of three quarters of the wars. The Hundred Years War grew out of a dispute about whether a nephew or a grandson of Philip the Fair had a better claim to the throne of France. The nephew got the decision, because the grandson's claim passed through a daughter (and because he was the king of England too). And that mess was complicated by one of the precedents (the Salic Law) that had been invented a short time before to resolve the dispute after the death of Philip's eldest son, where the claimants were (1) the daughter of Philip's eldest son, who may or may not have been a bastard, her mother having been an adulteress, (2) the unborn child of the eldest son that his secon wife was carrying, sex unknown, and (3) Philip's second son, another Philip. Lawyers for (3) dug up the Salic Law to exclude (1) and possibly (2) if she was a girl, but (2) was a boy so he became king, only he died a week later, and (3) got the throne after all. But then when he died, his own children, all daughters, were excluded on the basis of the law he's dug up, and the throne went to the youngest son instead... and meanwhile (1) had kids, one of whom eventually was the king of Navarre, Charles the Bad, who was such a scumbag in the Hundred Years War in part because he felt =his= claim was better than that of either Philip of Valois or Edward Plantagenet. And you know, it was. Only Navarre did not have an army as big as France or England, so no one took him seriously.
The Wars of the Roses were fought over the issue of whether the Lancastrian claim (deriving from the third son of Edward III in direct male line) or the Yorkist claim (deriving from a combination of Edward's second son, but through a female line, wed to descendants of his fourth son, through the male) was superior. And a whole family of legitimized bastard stock, the Beauforts, played a huge role.
And when Alexander III, King of Scots, rode over a cliff, and Margaret the Maid of Norway died en route back home, and the Scottish lords called on Edward I of England to decide who had the best claim to the throne, something like fourteen or fifteen (I'd need to look up the exact number) "competitors" came forward to present their pedigrees and documents to the court. The decision eventually boiled down to precedence (John Balliol) versus proximity (Bruce) and went to Balliol, but those other thirteen guys all had claims as well. King of Eric of Norway, for instance, based his claim to the throne on his =daughter=, the aforementioned Maid of Norway, who had been the queen however briefly. He seemed to believe that inheritance should run backwards. And hell, if he had been the king of France instead of the king of Norway, maybe it would have.
The medieval world was governed by men, not by laws. You could even make a case that the lords preferred the laws to be vague and contradictory, since that gave them more power. In a tangle like the Hornwood case, ultimately the lord would decide... and if some of the more powerful claimants did not like the decision, it might come down to force of arms.
The bottom line, I suppose, is that inheritance was decided as much by politics as by laws. In Westeros and in medieval Europe both.
