Analysing pedigrees

kidneyfan wrote: »

I don't think that you should necessarily reduce the data files. Have you looked at indexing the database? 400,000 records isn't many.
One thing I would suggest might be to apply the standard dosage method to your dataset and contrast your own method with that. etc

I use indexing extensively. What "slows" the program is the amount of comparison (I'm not saying what with what).
It does the comparisons for 894 horses, stepping down through the load file one at a time.

Do you mean Dosage Index? I wrote a program for that in 1993. It is not worth considering. See elsewhere on board for my comments.

There wasn't much improvement from reducing the data field size, about 10% to 53 seconds a record.
I'm not even sure there was an improvement as I didn't record the timings from earlier.
I might go for a cut down program that looks at less of the pedigree, identify likely candidates, then run the full version on those.

The other option is to do what they do on the cycling forum. If someone gets a puncture people recommend a new bike. Perhaps a new PC?

I'm dragging up an old thread.
For the past few months I've been doing a bit of database programming. The idea is to compare flat (not fences or hurdles) pedigrees to race results.

At this stage I have not done the comparison, but I have the basics on the pedigree side.
I have a program that analyses pedigrees. I can load any number of horses, start it, and come back later for the results.
I'll be tinkering with it a little, adding bits and pieces, putting in timing to find slow spots, and rewriting to speed things.

What gave me a boost was a poster asked me by PM to look at the Tattersalls February catalogue. It has 488 lots.
I could look at each lot on screen in my commercial pedigree program (TesioPower) and pick out the best imo.
Instead I decided to put in some extra effort to finish the database program.

It analysed the 488 Tattersalls lots in 44.53 seconds or approx 0.9 seconds a horse.
That was on my slow seven year old PC.
I bought a new PC in November and that runs things in 32% of the time of the old PC, so it will analyse a horse pedigree in under 0.3 of a second.
First it gathers the 126 ancestors in the first six generations (2,4,8,16,32,64), then analyses them.
Fwiw in a sales catalogue they print the first three generations (14 ancestors).

Plan
Add new features to the six generation analysis (group winners, Derby winners, group winner producer).
Compare the analysis to race performance.
Give up if no link found between pedigree and performance , or makes changes.
My next plan will be to analyse 7, 8, 9, 10, 11, 12 generations. This should be easy as I only have to increase the size of the ancestor database.
Of course if I go from 6 generations from 7 generations the data doubles.
A 12 generation analysis is much bigger (and slower) than a 6 generation, 8,190 horses v 126 horses, 65 times the size.

When running the program a byproduct is it gave some strange results for a few horses, many in the 1800s.
These are horses with incomplete pedigrees (some of these are half-bred non-thoroughbred).
I go back, fix the data if I can, and run it again.
The reason to test the program against so many pedigrees if to test it against complex pedigrees.

I am very interested in full siblings in pedigrees
(horses with same sire and dam e.g. sires Sadler's Wells and Fairy King are both by the sire Northern Dancer out of the dam Fairy Bridge).
And I look for 3/4, 7/8 siblings.
Examples of recent 3/4 siblings are:
Frankel; Highland Reel; Intello; Roderic O'Connor; Sir Isaac Newton; Teofilo, all by Galileo out of a Danehill dam.

Below is a summary decade by decade of full siblings in 6 generation pedigrees from 1800 to now.
It gives an indication that horses were much more closely inbred in the past, probably because you walked your sire to a local mare.
(full siblings A and B: one of horse A, and four of horse B is counted as five)
Horse populations in the early 1800s stayed in the same area (as did humans).
When trains were invented (1830s) you could travel your mare.
Motor transport (1890s) made travel even easier.
Now you can fly the mare or stallion anywhere.

In the table below you will see a spike in full siblings in pedigrees in the 1860s and 1870s.
My guess is brothers Stockwell (1849) ("the emperor of stallions") and Rataplan (1850) are heavily involved.
Galileo traces back in direct male line to Stockwell, as does almost everything else running today.
The low full sibling numbers for the 1990, 2000s, 2010s might be because many of these are low quality running horses, not breeding horses.

Decade Horses Full Sib Average
180? 6 23 3.8
181? 40 121 3.0
182? 311 863 2.8
183? 948 2407 2.5
184? 1285 3642 2.8
185? 1826 8846 4.8
186? 2374 19072 8.0
187? 3145 26993 8.6
188? 4612 26988 5.9
189? 5622 24467 4.4
190? 6543 31694 4.8
191? 7831 35914 4.6
192? 9965 30673 3.1
193? 11835 32875 2.8
194? 16631 40744 2.4
195? 25365 38647 1.5
196? 27277 36406 1.3
197? 36611 82902 2.3
198? 50620 128464 2.5
199? 69801 107813 1.5
200? 83274 52267 0.6
201? 15249 6288 0.4

381171 3.4

I don't know much about statistics but I think the Chi Squared test might be the way to see if my results are relevant.
I've been swotting up on it in the last few days so caution is advised.

First I split the data into three files: colts, geldings, fillies.
There are 37,057 colts and that is where I started the tests.
I split the colts into two groups 10,212 and 26,842 = 37,054 (3 missing?).

Group A are the lower rated colts, Group B the higher rated colts.
Of course the highest rated colt in group A will only be a fraction below the lowest rated colt in Group B.
In hindsight I could have picked a higher point so that the split would be closer to 50/50.

Actual Frequency Group A Group B Total
0 4453 11394 15847
1 3876 10158 14034
2 1505 4116 5621
3 317 983 1300
4 58 171 229
5 3 20 23
10212 26842 37054


Expected Frequency Group A Group B Total
0 4,367.40 11,479.60 15847
1 3,867.74 10,166.26 14034
2 1,549.14 4,071.86 5621
3 358.28 941.72 1300
4 63.11 165.89 229
5 6.34 16.66 23
10212 26842 37054

p-value 0.01806328 <--- chitest (actual: expected)


Chi-Square Terms Group A Group B
0 1.68 0.64 = (4453-4367.4)^2/4367.4
1 0.02 0.01 ( differences squared
2 1.26 0.48 ( to turn them positive
3 4.76 1.81 ( then divide by expected
4 0.41 0.16
5 1.76 0.67

Chi-Square 13.64 <--- sum above Chi-Squared Terms values
Degrees of Freedom 5 <--- (data rows -1) * (data columns -1)
Alpha 0.01 <--- 1% level
Critical Value 16.81 <--- critical chi-square value (x2 distribution table)

Decision Reject - Group A & Group B are not different at 1% level

Explanation ( If "Chi-Square" number is bigger than "Critical Value"
( then differences are large between the groups
( and not caused by chance


This is from an Excel spreadsheet.
The actual numbers are at the top.
Then there is a calculation of the "expected" numbers.
(4367.40 is 4453 x 15847/37054 and so on)
The Chi-Squared Terms are the differences squared (to turn negative differences positive).
The idea is if the difference are big then Group A and Group B are very different, and the difference is not due to chance.

The "Decision" near the end is a comment saying if Group A and Group B are similar or different.
What I want is the pedigree factor I am testing to show a difference between the groups.
I want the Group B to have more of the factor, and that difference to be so large that if is not caused by chance.
The above test "failed" to prove that there is a one in a hundred chance that Group B are better due to the factor.
But if I change the Alpha to 0.05 it changes to
"Accept - Group A and Group B are different, - significant at 5% level"
or in other words there is only a one in twenty chance that Group B has more of the factor due to chance.

Then I thought I would see what the average ratings were for the colts with this factor.
Please remember that there are 37 pedigree factors, and this is just one of the 37 factors, a factor I think might produce better colts.
The other 36 factors are not yet tested statistically.
I will try to write a database program to calculate the statistical result at 1% and 5% for colts, geldings, fillies, and for all 37 factors.
It might be worthwhile to split the data further into 10% chunks from slowest horses to fastest.
Another possible is to use the random factor to split the horses randomly so that I do not use my opinions to select groups to test.

What would happen if I calculated the average ratings of the 37,057 colts who have the factor one, twice, three times and upwards.
Almost half the colts do not have this factor in their first six generations.
The colts without this factor have an average rating of 74.36.
The higher the number of factors it appears the higher the rating.
These are the ratings of the 23 horses with 5 occurences that average 106.40
45, 65, 74, 83, 88, 90, 96, 107, 111, 112, 115, 118, 120, 121, 124, 127, 129, 131, 132, 140
I should point out that much of the data is of the best horses over the past forty years, and small numbers are unreliable.


occurrence Count Average Diff
0 15,847 74.36
1 14,034 76.32 1.96
2 5,621 79.12 2.80
3 1,300 85.05 5.93
4 229 89.69 4.64
5 23 106.40 16.71
6 3 91.00 -15.40

37,057

I'm staying in the house today waiting for a DHL courier.

This are an extract from the results, and might be of interest. There are four sex groups and 18+ factors in each, so 72+ pieces of information.

One piece of information was picked at random (and luckily it is informative)
Average rating (factor occurence): 80.7 (0); 81.2 (1); 81.9 (2); 82.7 (3); 83.6 (4); 85.4 (5); 87.2 (6); 91.1 (7); 105.1 (8); 75.5 (9); 76.5 (10)
Average rating increase cumulatively: n/a (0); +0.5 (1); +1.2 (2); +2.0 (3); +2.9 (4); +4.7 (5); +6.5 (6); +9.4 (7); +24.4 (8); -5.2 (9); -4.2 (10)

You can see there is an increase in rating for each increase in this factor count, from 0 up to 8. Then there is a massive drop for counts 9 and 10.
One of the earlier groups has over 25,000 horses so the average is reliable, group 9 has only 4 horses and group 10 has 2 horses. Group 6 has 1,496, group 7 has 313, group 8 has 32 horses.

Some people say only one generation matters, the sire and dam. Sales catalogues show a 3 generation pedigree. I'm working with 6 generations. Is it time to look at 7, 8, 9, 10, 11, 12 generations?

Francie Barrett wrote: »

It'd be interesting to do a distribution of the various combinations.

Good mare/good stallion.
Average mare/good stallion.
Good mare/average stallion.
Average mare/average stallion.

Maybe you could run a query, get the average ratings of the offspring for each combination?

I know you've talked about buying a horse, has your data analysis narrowed down the criteria on what you'd be using to select something?

“Good” and “average” are not easy to define. I tend to use numbers. I produced foal averages for stallions. People would be surprised at how little the averages differ, only a few points. And of course we do not know how many foals were culled. We may just be seeing the best on the racecourse. It is a business.

A while back I grouped mares into 5 rating points bands (e.g. 1-6, 6-10, 11-15 up to 140+). The higher the dam ratings the higher the average foal rating. I confess I made errors and would have to run it again. The link between sire rating and foal rating was similar. Most sires are 120+. The dam is the weak part of the pedigree, and the dam’s dam the weakest.

The variation in foal ratings for a sire’s lifetime crop is very large, and most of the variation imo due to good/bad pedigrees instead of the dam rating. For bad pedigree I mean little in common between the horses in the sire and in the dam pedigree i.e. little or no duplications, or male duplications of a sire only. The concept of good sires, good broodmares does not make sense to me. Good horses are the product of good pedigrees that match the ancestors of sire and dam (others may disagree).

Ratings used might be suspect. They are a combination of well known ratings, the highest gained by the runner as a 3yo or older. Earlier ratings (1960s, 1960s, 1970s) seem to have been reviewed and lowered, so if you take them from old books you might get 135 for a horse, and if you see the same horse now it might be a 127. And 2yo ratings (free handicaps) were used when no 3yo+ rating found. An example here is Fairy Bridge, the dam of Sadler’s Wells. She only ran as a 2yo, was rated 124 (actually stones and pounds) in the Irish Free handicap, and retired to stud.
The USA experimental free handicap has an upper limit of 126 which might under rate those horses. The introduction of International Classifications helps even out the ratings of the major racing countries. Some countries may have been a optimistic in the past, and their ratings useful in promoting local bloodstock.

Has my analysis narrowed down my criteria?
(1) avoid numerous duplications of a sire that produces only male offspring e.g. Northern Dancer, especially if these are the only or the majority of duplications in the pedigree.*
(2) if buying to breed buy fillies with horses in their 3rd, 4th 5th generation that are full siblings of horses in the pedigrees of stallions at stud (or ¾, 7/8 siblings). These are not as common as you might think.
(3) breed the filly on paper with all sires at stud before you buy her (test-matings).

* I started to record the sire lines for each of the 393 stallions now at stud, and the sire line of the dams of those 393 stallions. After 22 stallions I stopped to do other work. 17 of those 22 stallions were Northern Dancer sire line, and 12 of their dams were Northern Dancer sire line (9 had both). Careful pedigree planning is needed to avoid this.

The example I gave of El Condor Pasa should be examined. His dam Saddlers Gal has the full siblings Special and Lisadell (both By Forli out of Thong) in her pedigree 3 x 2, too close to make her a good runner, but this close inbreeding often makes the mare a good producer. The Japanese breeder knew what he was doing. He bred her to the sire Kingmambo, who also has Special in his pedigree.

The next work is to go beyond the basic 6 generation analysis and prepare something that goes deeper into the pedigree, isolating the features that top males have and top females have (identified visually on screen). The analysis so far was a ready reckoner.

Just got knocked out of a tournament on PokerStars (5:20 am). Time for a cup of coffee and a donut.

Found this on the internet a few minutes ago

Takashi Watanabe's interest in horseracing is driven by his knowledge of pedigrees which resulted in his breeding of El Condor Pasa, the newest Japanese star to shine in Europe. The owner of a trucking and transportation company in Tokyo, Watanabe was introduced to the sport by his father.
In 1992, he commissioned an agent to attend the Tattersalls December Sales to purchase the mare Saddlers Gal-a daughter of Sadlers Wells-who had failed to win in nine starts in Ireland.
The mare was withdrawn from the catalogue, but he was so determined to secure her that the representative, Morio Sakurai, was ordered to track her down.
''Mr Watanabe asked me to find her,'' said Sakurai. ''I located her on a farm in Ireland and he told me to buy her.''
The subsequent mating between Saddlers Gal and the top French miler Kingmambo was to produce El Condor Pasa, who Watanabe named after a Simon and Garfunkel hit.
The colt was placed with Yo****aka Ninomiya and last year became the first three-year-old to win the Japan Cup, doing so by two and a half lengths-the widest margin ever.
Ninomiya, a trainer since 1990, has a team of just 10 horses.
Approaching 50 years of age, he had previously served as assistant to renowned horseman Teruo Hashimoto for 12 years.
But he has also gained valuable experience for El Condor Pasa's current international programme during spells with Sir Michael Stoute in Newmarket and with US trainer Bruce Headley in California.

* Yo****aka (Yosh1taka)

This is the pedigree of Mary Sea, the average filly I chose for the test-mating experiment mentioned in previous posts.
Note the few connections between the pedigrees of her sire Selkirk and dam Mary Astor.

These are the average ratings for each pedigree position for a large number of pedigrees.
Shown are average rating & number of rated horses averaged for that position.
What is amazing to me is that the average for sires in the 3rd, 4th, 5th, 6th generation are all over 130.
The 1st generation at 124.3, and 2nd generation at 129.6 and 127.6 suggest that many of the current horses will not survive in future pedigrees.
Of course many of the current horses making up the 78.9 average are geldings or colts that will not be used for breeding.
Lower counts for dams probably reflect: unraced dams; dams of one foal and rating not found / investigated; imported dams from USA and other.
The lowest average dam ratings are on the bottom dam line as anticipated.

These are a few more ideas that are work in progress.

PREPOTENT
You often hear about prepotent horses (showing great effectiveness in transmitting hereditary characteristics to its offspring).
The term chef-de-race is used for these horses in the Dosage Index calculation. They are super sires.
The Dosage Index calculation uses chef-de-race sires in the first four generations of a pedigree.
Some of these sires have sired winners of great races.
Is there evidence that their influence lasts many generations? I have not seen the analysis.

I have a big file of the six generation pedigrees of 159,230 horses.
The file is about 20 million lines (159,230 x 127).
The idea I have is to give all the 127 horses in a six generation pedigree the same rating.
I’ll use the example of Mary Sea (2000) above to explain. Her rating is 71.
I’ll give her sire Selkirk 71, and her dam Mary Astor 71, and all the other horses in her first six generations 71 for this exercise.
Selkirk is actually 129 and Mary Astor 86.

Selkirk will get a different rating for every pedigree he appears in as a sire, whatever generation he is in.
For Mary Sea he gets 71, for Altieri he gets 122, for Emerging (by Mount Nelson) he gets 90 (Selkirk is in the pedigree of Mount Nelson).
At the end of the work all the Selkirk ratings are averaged, and Selkirk gets a number.
The average of all the 159,230 horses is about 80.
If a sire is “prepotent” his average should pop up above all others like a cork in water.
His influence will have boosted the rating of every pedigree in which he appears.
This should work for dams too, although they appear much less in pedigrees as they produce fewer offspring.
Sires and dams that appear in only a few pedigrees will be filtered out of the results.
I have about half the work done on this, the difficult part.


AGE
In this week’s Irish Field trainers and breeders were asked “would they buy the produce of an old mare”. A variation on this is do you believe birth order is important.

I could easily produce information on these questions, but I think other factors are ignored in these studies.
I’ve seen a study where the sample was only a few hundred, and the conclusion was the status quo: earlier birth rank is better; younger broodmares are better.

My guess is the owner of a new broodmare sends his expensive purchase to the best sire he can afford, and probably repeats this for a few years.
After a few disappointing foals he loses faith in the mare (and loses his money) and sends her to cheaper stallions, and still cheaper stallions until the end of her productive life.
Have you every heard of a mare owner starting her out with the cheapest stallion, and sending her to the champion sire in her old age?

I am interested not in the age of the sire and dam but in the dates of birth in the pedigree.
A few numbers might throw some light on this.
These are the average age of all horses (by generation) in my large data sample.
There are more horses in the 6th generation so I'll say the average age of all in about 10.5 years.
I'm not using this average age for anything at present. I just thought people might be interested in it.

Gen Age
0
1 11.25
2 10.02
3 10.00
4 10.11
5 10.34
6 10.51



The table below needs a bit of explaining.
These eight horses (A B C D E F G H) are four super sires, three disappointing sires, and one unproven recent sire.
Five of the sires were excellent on the racecourse, but two of these disappointed at stud.
One of the super sires was a disappointment on the racecourse.

The numbers are the dates of birth of horses in the same pedigree positions, in the same generation of their pedigrees.
The first date of birth (DOB) is in the first quarter of the pedigree, the next in the second quarter (1st and 2nd quarters are in the sire pedigree).
The last two DOBs are in the dam pedigree of these horses.
I could have listed all the DOBs in that generation but am using four only for demonstration.

A B C D E F G H
Sire's sire 1/4 1925 1935 1913 1920 1913 1935 1935 1920
Sire's dam 1/4 1920 1961 1935 1919 1935 1935 1927 1935
Dam's sire 1/4 1908 1954 1913 1942 1942 1919 1919 1937
Dam's dam 1/4 1909 1954 1913 1934 1930 1917 1928 1935


Comments:
A) notice the imbalance between the 1925, 1920 and the 1908, 1909 in the dam side. This is like the Leaning tower of Pisa.
My untested theory is that this imbalance often acts like an outcrossed pedigree, and might be a reason for the high ability.
But my idea is this imbalance is a negative when that horse goes to stud.
Half his pedigree doesn't match the pedigree of his mares because it is too old.
Unproven sire as yet. Excellent runner.
C) A super runner and super sire. Good DOB balance for three of the four ancestors.
D) A very good runner, but a major disappointment as a sire.
The sire side of his pedigree is much older, about twenty years.
E) A high class runner and super sire. One horse is a good bit older (1913 DOB), but three are about the same age.
The big worry is when the two on the sire side, and the two on the dam sire differ greatly.
F) This was a very good juvenile, but a failure as a sire.
Again note the large gap between the sire side (1935, 1935) and the dam side (1919, 1917).
G) A poor runner but a sensational sire.
H) A good runner, not a top runner, and a very successful sire.

I haven't given the name of the horses as I don't want to offend people.
This is just an idea from looking at many pedigrees, and trying to figure out why some very good horses didn't make the grade as sires.
These sires and runners did not get their class just from a few matching dates of birth.
My point is unmatched DOBs might be a negative.

It might be a tricky task to program and test this. How do I extract "bad" DOBs?

Peintre Celebre wrote: »

Diomed re a good horse from a bad mare. Hyperion's grand dam won a seller

Good horses have run in sellers (and bad ones).

Virago (1851) ran in a seller as a two year old and lost.
In John Porter's autobiography he says " I have in my time seen many great fillies, but I regard Virago as perhaps the greatest of them all".
Virago won the English 1000 Guineas as a 3yo.
John Porter trained 21 English Classic winners.

"The filly's participation in the event (a 2yo selling race) was a colossal piece of bluff, the purpose of which was to deceive those whose duty it was to frame the big handicaps of the following spring. .... "entered to be sold for £80, a bit of bunkum ... Day's head man accompanied the filly to the starting-post, ostensibly with a view to ensuring her getting well away ..... Goater appeared to be taken by surprise when the starter dropped his flag, and Virago was "left" a long way behind the others. She of course finished "nowhere" as intended. .... She was not among the first three, though she could have carried eleven stone and won. She could not have been bought for £5,000."

I have many horses in my data with the runner rated over 100 and its dam in the 20s. Often a dig into the low rating shows a horse who ran well but disappointed many times later and ended the year with a low rating (that I've used) although it ran at a higher rating during the year. The most unusual case was a rating of 3 as a three year old. The horse finished its two year old racing with a 99 rating, but finished last (?) every time as a 3yo and was retired.

I use very large volumes of data in an attempt to even out the questionable ratings.