UPDATE:
The decent performance I get for DRC+ projecting low-PA players is from assigning pitchers terrible DRC+s no matter how well they hit. The rest of the post is fine, but this is all even more bullshit than I realized at the time of writing.
/UPDATE
Required knowledge: wOBA and DRC+
Part 2: The DRC+ team-switcher claim is utter statistical malpractice
TL;DR: raw DRC+ is a little better overall than projecting everybody to be league average, but actually worse than that for team-switchers. Best-regressed DRC+ has about a 2.5-point MAE improvement over “everybody hits league average” for switchers and a 1 point MAE improvement over best-regressed wOBA. Regressed DRC+ has a huge advantage projecting very-low-PA seasons and starts losing to regressed wOBA around the 300 PA mark.
Having already demonstrated and explained why DRC+ is structurally unfit for use in WAR/BWARP, the purpose of this next experiment was to test the claims here and here that DRC+ was something special when it came to projecting next year’s wOBA for team-switchers. It fails that test convincingly, but a little regression work gives a decent projection for players that *don’t* switch. Unfortunately for DRC+, that prediction is only marginally better overall than the same methodology using wOBA, and only has a real advantage at the low-PA end. Regressed wOBA rules the high-PA end.
Methodology Overview
To test their claim, and to account for leaguewide wOBA changing every year, I normalized every batter-season’s wOBA onto a 100 scale by taking (batter wOBA)/(league average wOBA for that season) * 100. I’ll call that wOBA% from now on. Normalization to wOBA% makes sense because many of the factors that influence leaguewide wOBA changes in the upcoming year, from the changing strike zone to the changing baseball itself are not something DRC+ tries to predict- or ever should try to predict. Using wOBA% instead of raw wOBA removes a good deal of nonsense noise at no cost.
Team switchers were not explicitly defined, but since the test sample must be composed of players who had PAs in consecutive seasons, I’m defining a team switcher as anybody who didn’t appear entirely for the same team in both years (e.g. half a season for team A followed by 1.5 seasons for team B is a team switcher).
I also normalized DRC+ to DRC+% similarly since it was coming out a bit under 100, but since DRC+ is on the run scale, I used 100*Sqrt(Max(DRC+,0)/100) to put it on the same scale as wOBA. Seasons from 2010-2018 were used, although 2010 was only used to project 2011. Every pair of consecutive seasons where a player had at least 1 PA was eligible.
MAEs were calculated weighted by the harmonic mean of the PAs in year T and year T+1. Best regressions were determined the same way (using all pairs of seasons, not just switchers). Since MAEs are calculated in wOBA%, I multiplied by 3.20 (i.e. a wOBA of .320) to put them back on the wOBA-points scale to report.
Tests and Results
The first thing I tried was simply using year T DRC+% as the projection for year T+1 wOBA% and benchmarking that against an “everybody 100 wOBA%” (LgAvg) projection and restricting it to pairs of qualified seasons (500+ PA in each). DRC+ had an MAE of 25.1 points of wOBA against LgAvg’s 34.3 overall, but 26.5 vs 26.1 on team switchers.
In the attempt to see if the signal could be improved, I regressed year T DRC+% with league average (100 wOBA%) PAs and the minimum weighted MAE came with adding 89 average PAs. Doing the same optimization with year T wOBA% came up with adding 332 average PAs. For reporting purposes, I broke the players up into 3 groups
- 0-99 PAs in year T, which is just enough to capture all pitchers (2016 Bumgarner, 97 PA) as well as a bunch of callups and fill-ins who aren’t really MLB quality.
- 400+ PAs in year T, which is all full-time players and primary sides of platoons, etc. That number is kind of arbitrary, but it’s a little over 50% of the average PA per position, assuming some PHing, and moving it around 25 PAs isn’t really going to affect the big picture analysis anyway
- 100-399 PAs in year T to cover everybody else.
This is a sample report.
Table 1. wOBA MAEs, 400+ PA in both seasons. LgAvg=100
Min 400 PA both seasons | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year T+1 wOBA% | year T wOBA% | N |
all | 26.0 | 32.3 | 25.1 | 24.5 | 106.6 | 107.5 | 1152 |
switch | 28.4 | 26.7 | 26.9 | 25.8 | 103.6 | 105.3 | 303 |
same | 25.2 | 34.3 | 24.4 | 24.1 | 107.7 | 108.2 | 849 |
Any PA cutoff biases the sample, but using a PA cutoff in both seasons is especially bad form because it excludes players who would have reached the cutoff in year T+1 if they hadn’t been benched for sucking. Even with artificially tight performance constraints, the regressions are virtually useless for team switchers- only 1 point of MAE improvement for wOBA and nothing for DRC+. To avoid the extra bias problem, future results will include all (1+ PA) year T+1 seasons.
Table 2. wOBA MAEs, 400+ PA in year T, any PA year T+1. LgAvg=100
Min 400 PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 27.6 | 32.6 | 26.6 | 26.3 | 105.0 | 106.6 | 1597 |
switch | 30.3 | 28.8 | 29.0 | 28.2 | 101.0 | 103.9 | 473 |
same | 26.4 | 34.2 | 25.6 | 25.5 | 106.6 | 107.7 | 1124 |
The bias in Table 1 is apparent now. This population is simply worse to begin with in year T (marginal hitters were more likely to suck and get benched in year T+1 and not show up in Table 1) and dropped off more to year T+1. Back to the post topic, neither DRC+ nor wOBA are any good for switchers, wOBA is a bit ahead of DRC+, and the projection for same-team players is a clear improvement on LgAvg.
Table 3. 100-399+ PA in year T, any PA year T+1. LgAvg=100
100-399 PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 36.8 | 36.2 | 34.7 | 34.5 | 98.5 | 97.8 | 1574 |
switch | 38.5 | 38.3 | 36.3 | 36.5 | 94.3 | 95.2 | 627 |
same | 35.7 | 34.9 | 33.6 | 33.2 | 101.4 | 99.5 | 947 |
The league average errors are a good bit worse and now DRC+ and wOBA are pretty useless for everything, offering at best a 2-point improvement over LgAvg. Also, the quality of the players here is clearly worse because….. better players get more PAs and make it into Table 2 instead.
Table 4. wOBA MAEs, 1-99+ PA in year T, any PA year T+1. LgAvg=100
1-99 PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 110.2 | 102.2 | 62.9 | 92.5 | 84.3 | 69.8 | 2409 |
switch | 91.4 | 96.7 | 64.3 | 87.7 | 73.3 | 69.8 | 841 |
same | 120.3 | 105.2 | 62.2 | 95.1 | 90.2 | 69.7 | 1568 |
And this is interesting. Garbage hitters, giant MAEs, and regressed DRC+ winning by a mile for a change. The other interesting thing here is that the players teams keep improve *a ton* and the ones they let go keep being godawful at the plate. Somebody should look into that in more detail.
Seeing that the 100-399 PA group at least resembled MLB-quality hitters, albeit not the good ones, and that the 1-99 PA group was an abomination at the plate (it did include all the pitchers), I wondered what would happen if I cheated a little and tried to optimize on the 100+ PA group instead of everybody. That group looks like
Table 5. wOBA MAEs, 100+ PA in year T, any PA year T+1. LgAvg=100
Min 100 PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 30.4 | 33.7 | 29.1 | 28.8 | 102.7 | 103.9 | 3171 |
switch | 33.3 | 32.2 | 31.6 | 31.2 | 98.6 | 100.8 | 1100 |
same | 28.9 | 34.5 | 27.7 | 27.5 | 104.9 | 105.6 | 2071 |
Again, useless for switchers, solid improvement for the ones who stayed. Based on this, I decided to reoptimize based on a LgAvg of 103 using only players with 100+ PA year T just to see what would happen.
Trying a different league average
This is starting to look down the rabbit hole of regressing to talent (more PA is a proxy for more talent as we’ve seen) instead of to pure league average, but let’s see what happens. The regression amounts came out to 243 added PA for DRC+ and 410 added PA for wOBA. Doing that came up with
Table 6. wOBA MAEs, 100+ PA in year T, any PA year T+1. LgAvg=103
Min 100 PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 30.4 | 33.2 | 28.7 | 29.1 | 102.7 | 103.9 | 3171 |
switch | 33.3 | 33.7 | 31.3 | 31.9 | 98.6 | 100.8 | 1100 |
same | 28.9 | 32.9 | 27.2 | 27.6 | 104.9 | 105.6 | 2071 |
Well, that’s not an auspicious start, marginally helping the stayers (the switchers were closer to 100, so regressing towards 103 isn’t any help). Let’s see if there’s any benefit in either group individually.
Table 7. wOBA MAEs, 100-399 PA in year T, any PA year T+1. LgAvg=103
100-399 PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 36.8 | 38.7 | 34.5 | 35.6 | 98.5 | 97.8 | 1574 |
switch | 38.5 | 41.9 | 36.7 | 38.0 | 94.3 | 95.2 | 627 |
same | 35.7 | 36.6 | 33.0 | 33.9 | 101.4 | 99.5 | 947 |
Well, that was a disaster for these guys. The top line using 100 LgAvg was 36.8 / 36.2 / 34.7 / 34.5 before, and regressing them further from their talent shockingly didn’t do any favors. Was this made up for by the full-time players?
Table 8. wOBA MAEs, 400+ PA in year T, any PA year T+1. LgAvg=103
400+ PA year T | raw DRC+% | LgAvg | regd DRC+% | regd wOBA% | year t+1 wOBA% | year T wOBA% | N |
all | 27.6 | 30.8 | 26.1 | 26.3 | 105.0 | 106.6 | 1597 |
switch | 30.3 | 29.1 | 28.3 | 28.5 | 101.0 | 103.9 | 473 |
same | 26.4 | 31.5 | 25.2 | 25.4 | 106.6 | 107.7 | 1124 |
Not really. This is a super marginal improvement over the previous top line of 27.6 / 32.6 / 26.6 / 26.3. Only the LgAvg projection really benefits at all, and that’s not what we’re interested in. Changing league average a little and optimizing over only MLB-quality hitters doesn’t seem to really accomplish anything great for the DRC+ or wOBA regressions.
Conclusion
There’s little-to-nothing to BP’s claim that DRC+ is something special for team switchers. Raw DRC+% is worse than league average, and the MAE for best-regressed DRC+ is about 2 points better than league average overall, and that entire benefit is from the low-PA end. It projects team-switching full-time players worse than assuming they’re league average. However, for the really low-PA players, it is more accurate raw and much more accurate regressed than league average.
Likewise there’s also absolutely nothing to the claim that DRC+ is a significant improvement over wOBA for predicting year T+1 wOBA for switchers- the gap is actually *smaller* for switchers. 1.1 points of MAE for switchers and 1.5 points for stayers. The best regression of DRC+ absolutely does shine in the very-low-PA group, but it’s also not good in the full-time player category. Regressed DRC+ and regressed wOBA actually do make fairly decent, much better than league average predictions for full-time players who *stay*, for whatever an untested model fit to in-sample data is worth.