Limitations of DEV
By Zac Urback
The main issue with using a model to evaluate prospects is, in certain instances, context that may affect results is lost. DEV is meant to help - not replace - scouts, and in order to most adequately apply the model, one must first understand the flaws. A good analyst and/or scout applying DEV must be able to identify which players are not adequately valued by DEV.
Quality of Line Mates
The first flaw I want to discuss is the impact of line mates. All players are influenced by their line mates, but certain cases are more extreme than others (for better or for worse). Zack Phillips is an example of a player whose production was significantly inflated by playing with Jonathan Huberdeau on a stacked Saint John Sea Dogs team. According to DEV, Phillips was worth a top 7 pick in his draft year, due to his exceptional production, but scouts knew Phillips was benefitting from playing on a great line. DEV cannot currently quantify quality of line mates, which is why assigning subjective context is essential to interpreting results.
Another issue with DEV is that it looks at players' individual seasons in a vacuum. Anomaly seasons can lead to players being over or under valued by DEV. One way of addressing this concern is to analyze players track records.
16-year-old season DEV suggested he was worth a 7th round pick, but
his 17-year-old season DEV indicated he was worth a top 10 pick. Prospects
showing substantial improvement is typical, but any time a prospect makes such
a considerable jump by DEV I'd be at least a little bit cautious. False
positives are rare, but they occur and must be considered in any analysis. A
smart scout must question results and determine just how much a player actually
Unique Players with Cohorts
DEV projects players based off of their cohorts, but certain prospects cohorts do not adequately depict the type of player they are. Gabriel Landeskog was praised for his elite defensive abilities, his character, and his style of play. DEV does not account for these factors. DEV estimates the likelihood that a prospect will make the NHL and projects their point production. Ultimately, if a player is defined by something significant other than their individual production, DEV may improperly value them.
Adjusted point production is, currently, the best quantifiable measure for projection, but it is not necessarily the "be all end all" for every prospect. If you value a more subjective projection method, your opinions will not always match with DEV. Sometimes the subjective approach will be right (i.e. Landeskog) but, sometimes it will be wrong (i.e. Kyle Beach: selected 11th, ranked 25-29th by DEV). DEV tends to do a better job of evaluating the typical players, rather than the outliers like Landeskog. It is often the consensus top 5-10 picks that DEV struggles to properly evaluate.
Unique Players Without Cohorts
On the other hand, there are players that are unique to the point where they do not have many cohorts. For example, Colton Horn went undrafted in his draft year despite being valued as a top 7 pick by DEV. With that said, Colton Horn only had 3 comparable players (Tyler Ennis, Derek Roy and Eric Himelfarb) making it extraordinarily difficult to put much stock in his DEV ranking.
Interpreting Results - 2015 NHL Draft
Now that I've demonstrated some of the flaws of DEV, I want to take a closer look at a few of the CHL forwards selected in the first round of the 2015 NHL draft.
Connor McDavid & Dylan Strome
Connor McDavid and Dylan Strome were both unranked by DEV, as neither of them had any cohorts, but obviously they were deserving of their selections.
Lawson Crouse & Travis Konecny
Lawson Crouse and Travis Konecny are both examples of prospects that scouts praise for bringing more to the table than just production. Defensively responsible power forwards like Crouse are extremely coveted in the NHL, thus their market value may exceed their production value, making his DEV slightly misleading.
Likewise, Konecny plays a gritty style of hockey that is atypical of undersized players. DEV is a product of two inputs: likelihood of making the NHL and expected NHL production. Players of Konecny's size tend to struggle to make the NHL, but if they make the NHL, they usually produce at a high rate. For comparative sakes, Matthew Barzal, by DEV, was 3x more likely than Konecny to make the NHL, but their expected production was virtually identical. Thus, Barzal had a far higher DEV than Konecny, making him a safer pick, but if they both make the NHL, they are expected to produce at similar rates.
Konecny's successful (4/32) DEV draft-year cohorts
This article focuses on the areas where DEV struggles, but these instances tend to be relatively uncommon. Nevertheless, it is important to identify these outliers and to reconcile context and traditional scouting with DEV in order to achieve the most optimal results.