This day on November 20
None

Happy Birthday To: d_ottosen, lambofgad, Ofer, JayStyles, oilpride5

F.A.Q. Terms of Use F.A.Q. F.A.Q.
Members Members   Search Search     Register Register   Login Login   Home Home
 Speculation » Advanced stats, causality, and why numbers matter
Switch to flat viewSwitch to tree viewCreate a new topicSubmit Reply
 Advanced stats, causality, and why numbers matter [message #696596]
Tue, 27 June 2017 01:49 Go to next message
ziltoid  is currently offline ziltoid
Messages: 211
Registered: January 2011

No Cups

If this is in the wrong section please move it to wherever you see fit.



This is going to be a bit of an essay, but I think it touches on an important topic that has kicked up in the wake of recent events (e.g., Russell signing, the Eberle trade, and how different posters draw their conclusions re: all these moves). In short, I am going to provide a case for using "advanced statistics," as-well-as what we can reasonably conclude from them.


Descriptive versus Inferential

I think any advanced stats discussion needs to begin by delineating between descriptive and inferential statistics. As you would expect, descriptive statistics are simply metrics that describe some population, with the overwhelming majority of advanced stats falling into this category (e.g., shots, goals, assists, points, cori, fenwick, WOWY, zone entry and exit data, etc.). On the other hand, inferential statistics make predictions based on sample/population data. In the case of hockey, these would include things like ridge regression models prediction wins, or modelling goal-scoring as a semi-Markov process. To distill it down even further, descriptive statistics count and group things, whereas inferential statistics make heavy use of probabilities.

Main point #1 (MP1): most advanced statistics in hockey are simply counting and grouping things we directly observe.

Take corsi-for per 60; we simply count how many shots and shot attempts (blocked and missed) occur over a specific interval (e.g., a game, a season, a career, etc.), then divide by the total TOI, then multiply that result by 60.

60( (shots + shot attempts) / TOI)

There is nothing predictive about CF60; it is just a number that describes what we directly observe (corsi), but converted into a different unit (per 60). Given MP1, it is hypocritical to say these statistics are not valid but metrics such as goals and assists are (more on this later); all are direct observations, some of which simply get converted into different units and/or grouped into different sets, much in the same way kilometers can be converted into miles, or how Statistics Canada calculates things such as X per Y (e.g., debt per household, children per family).

Of course, there are people who recognize the above argument, and instead reject stats on the grounds that they, to paraphrase, "don't tell us anything about why."

The Problem of Causality

Conventional wisdom would state that even though we can measure all off these "advanced statistics," it takes an "expert" (aka, a "hockey guy") to be able to infer anything meaningful from these observed variables. That is, even though we can measure a player's corsi, it takes a trained eye to understand what causes that number to be what it is. In a sense this is true; it is hard to determine causality even under experimental control, let alone with correlational data. Thus, the bigger argument is the extent to which (i) correlational data can inform our understanding of the game, and (ii) we can reasonably infer causality.

The argument in favour of (i) is as follows:

(1) Goals, Assists, and Points all inform our understanding of team and player performance.

(2) If a measure is as valid as Goals, Assists, and Points, then it can inform our understanding of team and player performance.

(3) Advanced statistics are as valid as Goals, Assists, and Points.

Therefore, given (1-3), advanced statistics can inform our understanding of team and player metrics.

Now, the elephant in the room is, "what constitutes a valid observation." For example, we can count how many strides each player takes per shift, but one may be hard-pressed to view that as a valid metric, especially when we consider it along side metrics like goals, assists, and points. On the surface this seems like a complex and difficult question to answer, but in reality it is very simple. A valid observation is anything we directly observe and measure about the game. So yes, number of strides per shift IS as valid as goals, assists, and points, but valid does not entail useful, it only entails informative. That is, if we can directly observe and measure it, then it tells us something about the game, though whether this constitutes a useful piece of information remains to be seen. Thus, the next question with respect to (i) is, "in what way can advanced statistics usefully inform our understanding of the game."

If we take the conservative view that no correlational data can infer causality, then we are forced to view advanced statistics much in the same way we view lots of medical evidence. For example, the link between smoking and cancer is correlational; we can't randomly select 500 kids and have half of them smoke for 20 years and the other half not, then see who gets cancer; we have to correlate cancer rates with smoking while statistically controlling for mediating variables (though there are lots of other statistical ways of assessing the link between smoking and cancer, I don't want this to turn into a dense stats lecture). Even animal studies are correlational as a causal effect in rats does not entail a causal relationship in humans; it correlates with a causal relationship, but it is an imperfect proxy. Thus, from the most conservative viewpoint, advanced statistics in hockey tell us about the strength and direction of relationships between variables. Corsi has a moderate positive correlation with goals, therefore we can conclude that players with high corsi values generally score more goals than those with poor corsi values, but we cannot stake any claims on why this is (as an aside, for the 2012/13 - 2014/15 seasons, corsi-for percentage has a correlation of 0.35 with GF per 60). Clean zone exits correlate with corsi, therefore we can conclude that defensemen who produce clean zone exits generally produce more scoring chances as measured by shots and shot attempts. We can take this a step further when we delve into inferential statistics. For example, the stuff I am currently working on shows, among other things, that offense mediates the relationship between possession and defense. We cannot make any definitive claims as to why this occurs (at least not under this conservative view re: causality), but the relationship clearly exists in the data and should not be easily dismissed.

Main point #2 (MP2): advanced statistics in hockey are useful because they afford us the ability to assess the strength and direction of relationships between increasingly specific aspects of the game.

Let us now turn our attention to causality.

In short, there are 3 main criteria for determining causality such that X causes Y:

(1) X has to be correlated with Y
(2) X has to precede Y, temporally.
(3) Competing explanations of causality (within reason) must be ruled out.

Criteria (1) and (2) can, at times, be satisfied via hockey data (e.g., a goal must be proceeded by a shot), but the biggest problem we face is (3). Here, the problem is that advanced stats in hockey exhibit a high degree of multicollinearity, which is just a fancy way of saying that metrics are often highly correlated with each other. Thus even though criteria for (1) and (2) can be reasonably satisfied when trying to establish a causal relationship between corsi and goals, the fact that corsi is highly correlated with lots of other metrics means that we cannot directly rule out other competing explanations of causality. That said, there are statistical techniques we can use to control for the effects of other variables, but it is still an imperfect system. However, I do believe that if we (i) collect and analyze the data carefully, and (ii) repeatedly find strong convergent evidence for an effect/relationship, then we can say with a high level of confidence that there is very likely a causal relationship between X and Y. This goes back to the old adage that a single study does not mean anything, but 100 of them do. In reality, nothing in hockey is caused by a single factor, but rather by a combination of factors. Thus, we talk about "how much" of a factor X is in causing Y.

Main point #3 (MP3): although we can never be 100% certain with respect to causality, in the face of strong convergent evidence we can become highly confident that there is a causal relationship.

Advanced Stats versus Seen 'em Good

At the end of the day, it seems people's view of stats boils down to what they think is going to be more detrimental: the inherent biases of personal opinions, or our limited ability to determine causality via stats. If a person views our limited ability to determine causality via stats as being more detrimental than the inherent biases we all carry when we watch hockey, then they will likely side more-so with the "seen 'em good camp" (and visa-versa for the stats camp). Personally, I side heavily with the stats camp, but I will be the first to acknowledge that if you do the statistical analyses wrong, then your conclusions are going to be invalid and you are going to make bad decisions. Regardless of which camp a person situates themselves in, it is an objective truth that the stats are significantly less biased than personal opinions (in fact the biggest source of bias in stats comes from rink-to-rink variability in how they count each measure, but that bias largely washes out at the end of the day).



Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696597 is a reply to message #696596 ]
Tue, 27 June 2017 02:47 Go to previous messageGo to next message
George  is currently offline George
Messages: 336
Registered: October 2009

No Cups

Really interesting post, thank you. I’m still learning the different metrics, but my own view has been that we’re still in the early stages of advanced descriptive stats and are too reliant on corsi, which I always thought would be a bit too influenced by the context you are put in. So I found it interesting that you said that corsi is highly correlated with lots of other metrics, like clear zone exits. Maybe I’m not giving corsi enough credit.

I wonder how much scope there is for grouping player type/usage. For instance, I would think it would be hard to get any meaningful measure that compares all out defensive defensemen (e.g., Russell) with all out offensive defensemen (e.g., Schultz). To get something meaningful you would have to make within group comparison of players (e.g., Schultz vs Barrie). But then this leaves open the question as to whether each player type is necessary; I’ve seen a few bloggers suggest that all six of your defensemen should have puck-carrying ability, presumably based on analysing corsi. So is the role of defensive defensemen now becoming obsolete is today’s game, or is this collateral of our reliance on shot-based metrics? I don’t know, I’m just wondering out loud here, but I find all of this really interesting.



Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696647 is a reply to message #696597 ]
Tue, 27 June 2017 12:39 Go to previous messageGo to next message
ziltoid  is currently offline ziltoid
Messages: 211
Registered: January 2011

No Cups

George wrote on Tue, 27 June 2017 02:47

Really interesting post, thank you. I’m still learning the different metrics, but my own view has been that we’re still in the early stages of advanced descriptive stats and are too reliant on corsi, which I always thought would be a bit too influenced by the context you are put in. So I found it interesting that you said that corsi is highly correlated with lots of other metrics, like clear zone exits. Maybe I’m not giving corsi enough credit.

I wonder how much scope there is for grouping player type/usage. For instance, I would think it would be hard to get any meaningful measure that compares all out defensive defensemen (e.g., Russell) with all out offensive defensemen (e.g., Schultz). To get something meaningful you would have to make within group comparison of players (e.g., Schultz vs Barrie). But then this leaves open the question as to whether each player type is necessary; I’ve seen a few bloggers suggest that all six of your defensemen should have puck-carrying ability, presumably based on analysing corsi. So is the role of defensive defensemen now becoming obsolete is today’s game, or is this collateral of our reliance on shot-based metrics? I don’t know, I’m just wondering out loud here, but I find all of this really interesting.



There are statistical tools to group "things" (in this case player types). Again, I don't want to turn this into a stats lecture, but things like factor analyses and principle component analyses can be used to group similar "playing styles."

At the end of the day the biggest consideration (from my experience) is not individual skill, but the interactive effects between players. Long story short, no one player's performance can be attributed solely to them as an individual; their performance is a byproduct of their statistical interaction with other players on the ice. Thus you want to find players that interact with each other such that they both perform at an elevated level. So defensive d-men are not necessarily obsolete, it is just that they tend not to interact well with other d-men (though, ironically, Klef and Larsson represents a case where a defensive d-man is clearly beneficial to the pair as a whole; though Russell + anyone is a bad, bad, bad idea).



Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696603 is a reply to message #696596 ]
Tue, 27 June 2017 08:50 Go to previous messageGo to next message
Adam  is currently offline Adam
Messages: 11761
Registered: August 2005
Location: Edmonton, AB

6 Cups

Thanks for this. Interesting read. I'm always fascinated by how many brilliant people from different backgrounds we get in here.

I was re-reading this piece ( http://edmontonjournal.com/sports/hockey/nhl/cult-of-hockey/ analytics-expert-says-kris-russell-of-edmonton-oilers-is-und errated-and-one-of-nhls-top-defensive-d-men) this weekend and I've been puzzling over it.

Werenka's made a believer out of Chiarelli and David Staples, but this part has bothered me:

Quote:

This kind of work comes down to dozens and dozens of decisions and judgements made by TruPerformance game analysts on every player every game. The decisions are subjective, but Werenka says training, guidelines and auditing is in place to make sure that the raters are reliable, that one rater will watch the same game and make almost all the same calls on the same plays and players.

TruPerformance has a set standard of rules analysts apply to judge each play. Auditing is done to make sure the raters are consistent. “Is there a level of variance? Absolutely. We operate on a level of five or eight per cent variance. We would like to keep it within five per cent.”

The key is to get good people to rate, Werenka says. “There are two types of people that are very good. Experienced people that have open minds. They decide through their experience that, you know, being physical or a check is one of the most important things that will happen in a game, and if they’re not open minded to change that then we’re in trouble. So we want experience guys that are open-minded, or guys that have a good basic understanding of the game and have an extreme learning curve, guys that can learn. One of the best compliments we can pay to those guys is, ‘He never asked the same question twice.’ Those are the guys we love.”

When it comes to the issues of rater reliability and subjectivity, Werenka has come to a few conclusions.

“You can say this is subjective, but there’s been a couple phrases we like to use: Consistent subjectivity equals objectivity. And how do you know if that consistent subjectivity is meaningful, if it makes sense, is right? That’s where your correlation to winning comes in.”


There is subjectivity in Corsi analysis as well - what constitutes a shot? And scoring chance metrics are even more so, but I wonder with Werenka's system if there isn't some built in biases. If for example, you were a defensive defenceman when you played and believed that your practice of off the glass and out was a valuable skill - after all, you're relieving pressure and possibly allowing linemates to change - might you rate similar players high?

They're very proud that their analysis has the top players in the game consistently at the top of their charts, but since there's a high level of subjectivity, might part of that be due to marker bias?

I've also been thinking about how this plays in to the hiring of General Managers. I think the mistake that Arizona and Florida made is that the spreadsheet guy is really just a part of your scouting department. Advanced stats is about player analysis, which is only one piece of the puzzle when it comes to being a GM. What does a 26 year old number cruncher know about negotiating contracts or managing the salary cap? I think your best GM has a negotiating background, as he has to handle much of that himself, and is a good leader with great delegation skills and the ability to hire well and trust those he brings in. Having Lamoriello as your GM with Dubas as the assistant is a much better route than say, hiring Tyler Dellow to run your team.

Anyhow, I'm getting off topic, but I appreciated the post on stats, and it will be interesting to see how it evolves in the NHL, and how much we'll actually get to see (since the best analytics guys keep getting hired away).



"This team needs an enema!"
#FireLowe #FireMacT #FireHowson #FireBuchberger #FireHowsonAgain #FireChiarelli #FireMcLellan

Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696649 is a reply to message #696603 ]
Tue, 27 June 2017 12:42 Go to previous messageGo to next message
George  is currently offline George
Messages: 336
Registered: October 2009

No Cups

Adam wrote on Tue, 27 June 2017 08:50

Thanks for this. Interesting read. I'm always fascinated by how many brilliant people from different backgrounds we get in here.

I was re-reading this piece ( http://edmontonjournal.com/sports/hockey/nhl/cult-of-hockey/ analytics-expert-says-kris-russell-of-edmonton-oilers-is-und errated-and-one-of-nhls-top-defensive-d-men) this weekend and I've been puzzling over it.


I really like that piece, though I’m probably biased as it speaks to my own concerns about corsi. It does raise a good point, corsi is embraced so much partially due to the fact that it is public data. It is a pretty objective metric that can be scraped from an NHL website. Conversely weighting each play in a game for a pre-determined value requires teams of people and an agreed upon standard.


Quote:

Werenka's made a believer out of Chiarelli and David Staples, but this part has bothered me:

Quote:

This kind of work comes down to dozens and dozens of decisions and judgements made by TruPerformance game analysts on every player every game. The decisions are subjective, but Werenka says training, guidelines and auditing is in place to make sure that the raters are reliable, that one rater will watch the same game and make almost all the same calls on the same plays and players.

TruPerformance has a set standard of rules analysts apply to judge each play. Auditing is done to make sure the raters are consistent. “Is there a level of variance? Absolutely. We operate on a level of five or eight per cent variance. We would like to keep it within five per cent.”

The key is to get good people to rate, Werenka says. “There are two types of people that are very good. Experienced people that have open minds. They decide through their experience that, you know, being physical or a check is one of the most important things that will happen in a game, and if they’re not open minded to change that then we’re in trouble. So we want experience guys that are open-minded, or guys that have a good basic understanding of the game and have an extreme learning curve, guys that can learn. One of the best compliments we can pay to those guys is, ‘He never asked the same question twice.’ Those are the guys we love.”

When it comes to the issues of rater reliability and subjectivity, Werenka has come to a few conclusions.

“You can say this is subjective, but there’s been a couple phrases we like to use: Consistent subjectivity equals objectivity. And how do you know if that consistent subjectivity is meaningful, if it makes sense, is right? That’s where your correlation to winning comes in.”


There is subjectivity in Corsi analysis as well - what constitutes a shot? And scoring chance metrics are even more so, but I wonder with Werenka's system if there isn't some built in biases. If for example, you were a defensive defenceman when you played and believed that your practice of off the glass and out was a valuable skill - after all, you're relieving pressure and possibly allowing linemates to change - might you rate similar players high?


Presumably they have some safe guards in place. They talk about variance levels, I'm sure they rotate guys so it's not the same one or two people marking the same teams, and they probably have an inter-rater reliability measures that are regularly updated. With all of that they /should/ get reliable data. If they're carrying out the analyses properly, they can really partial out a lot of subjectivity.

Quote:

They're very proud that their analysis has the top players in the game consistently at the top of their charts, but since there's a high level of subjectivity, might part of that be due to marker bias?


Of course, these players actually are the best in the league so it seems they're doing something right ;) I actually liked their argument here: the corsi players are obviously way off suggesting that it is a not a strong a predictor as some may believe. Again, if TruPerformance are collecting data using proper protocol, they should be able to hone in on objective values.

For those who haven't read the article, Top 5 Centres as rated by each metric:

TruPerformance: Sidney Crosby, 93%, Evgeni Malkin, 92%, Anze Kopitar, 91%, Patrice Bergeron, 89%, Tyler Seguin/Connor McDavid, 88%.

Corsi: Nick Shore, 61%, Pavel Datsyuk, 58%, Mike Ribeiro, 58%, Mathieu Perreault, 58%, Eric Staal, 57%,




Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696657 is a reply to message #696649 ]
Tue, 27 June 2017 12:50 Go to previous messageGo to next message
Adam  is currently offline Adam
Messages: 11761
Registered: August 2005
Location: Edmonton, AB

6 Cups

George wrote on Tue, 27 June 2017 12:42

Adam wrote on Tue, 27 June 2017 08:50

Thanks for this. Interesting read. I'm always fascinated by how many brilliant people from different backgrounds we get in here.

I was re-reading this piece ( http://edmontonjournal.com/sports/hockey/nhl/cult-of-hockey/ analytics-expert-says-kris-russell-of-edmonton-oilers-is-und errated-and-one-of-nhls-top-defensive-d-men) this weekend and I've been puzzling over it.


I really like that piece, though I’m probably biased as it speaks to my own concerns about corsi. It does raise a good point, corsi is embraced so much partially due to the fact that it is public data. It is a pretty objective metric that can be scraped from an NHL website. Conversely weighting each play in a game for a pre-determined value requires teams of people and an agreed upon standard.


Quote:

Werenka's made a believer out of Chiarelli and David Staples, but this part has bothered me:

Quote:

This kind of work comes down to dozens and dozens of decisions and judgements made by TruPerformance game analysts on every player every game. The decisions are subjective, but Werenka says training, guidelines and auditing is in place to make sure that the raters are reliable, that one rater will watch the same game and make almost all the same calls on the same plays and players.

TruPerformance has a set standard of rules analysts apply to judge each play. Auditing is done to make sure the raters are consistent. “Is there a level of variance? Absolutely. We operate on a level of five or eight per cent variance. We would like to keep it within five per cent.”

The key is to get good people to rate, Werenka says. “There are two types of people that are very good. Experienced people that have open minds. They decide through their experience that, you know, being physical or a check is one of the most important things that will happen in a game, and if they’re not open minded to change that then we’re in trouble. So we want experience guys that are open-minded, or guys that have a good basic understanding of the game and have an extreme learning curve, guys that can learn. One of the best compliments we can pay to those guys is, ‘He never asked the same question twice.’ Those are the guys we love.”

When it comes to the issues of rater reliability and subjectivity, Werenka has come to a few conclusions.

“You can say this is subjective, but there’s been a couple phrases we like to use: Consistent subjectivity equals objectivity. And how do you know if that consistent subjectivity is meaningful, if it makes sense, is right? That’s where your correlation to winning comes in.”


There is subjectivity in Corsi analysis as well - what constitutes a shot? And scoring chance metrics are even more so, but I wonder with Werenka's system if there isn't some built in biases. If for example, you were a defensive defenceman when you played and believed that your practice of off the glass and out was a valuable skill - after all, you're relieving pressure and possibly allowing linemates to change - might you rate similar players high?


Presumably they have some safe guards in place. They talk about variance levels, I'm sure they rotate guys so it's not the same one or two people marking the same teams, and they probably have an inter-rater reliability measures that are regularly updated. With all of that they /should/ get reliable data. If they're carrying out the analyses properly, they can really partial out a lot of subjectivity.

Quote:

They're very proud that their analysis has the top players in the game consistently at the top of their charts, but since there's a high level of subjectivity, might part of that be due to marker bias?


Of course, these players actually are the best in the league so it seems they're doing something right ;) I actually liked their argument here: the corsi players are obviously way off suggesting that it is a not a strong a predictor as some may believe. Again, if TruPerformance are collecting data using proper protocol, they should be able to hone in on objective values.

For those who haven't read the article, Top 5 Centres as rated by each metric:

TruPerformance: Sidney Crosby, 93%, Evgeni Malkin, 92%, Anze Kopitar, 91%, Patrice Bergeron, 89%, Tyler Seguin/Connor McDavid, 88%.

Corsi: Nick Shore, 61%, Pavel Datsyuk, 58%, Mike Ribeiro, 58%, Mathieu Perreault, 58%, Eric Staal, 57%,




One of my issues with Staples though is that he seems to group all shot-related data in with Corsi. I don't think it's a perfect metric by any means. We saw with Eakins what happens when a coach tries to run a system designed to improve shot metrics without regard to shot quality. It was a horrendous failure because correlation does not equal causation. Good teams shoot more, but just increasing the number of shots doesn't necessarily equate to more goals and more wins...Smytty clappers from the half-boards just surrender possession for a low percentage shot.

Corsi is just the beginning though and most of the stats guys are diving a lot deeper than Corsi now.

I think the work being done by Steve Valiquette has more relevance than the Werenka stuff (in part because it doesn't have a super secret metric that says Russell is a top-ten d-man in the league on a level with Marc-Edouard Vlasic).

http://www.thehockeynews.com/news/article/whats-the-next-dim ension-of-analytics-former-nhl-goalie-steve-valiquette-knows

He's spent a lot of time on what happens just before a shot, and how scoring chances are developed.




"This team needs an enema!"
#FireLowe #FireMacT #FireHowson #FireBuchberger #FireHowsonAgain #FireChiarelli #FireMcLellan

Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696679 is a reply to message #696657 ]
Tue, 27 June 2017 14:21 Go to previous messageGo to next message
George  is currently offline George
Messages: 336
Registered: October 2009

No Cups

Quote:

I think the work being done by Steve Valiquette has more relevance than the Werenka stuff (in part because it doesn't have a super secret metric that says Russell is a top-ten d-man in the league on a level with Marc-Edouard Vlasic).


Yeah, that part made me cringe a bit.


Quote:

http://www.thehockeynews.com/news/article/whats-the-next-dim ension-of-analytics-former-nhl-goalie-steve-valiquette-knows

He's spent a lot of time on what happens just before a shot, and how scoring chances are developed.




The Valiquette analysis is really interesting and would seem to have a more direct application to coaching than the other measures used. The Royal Road idea also seems to sum up all of Letestu’s goals this year! (not that Valiquette’s analysis contains 5-on-4 data, but it essentially explains the sequence involved). It also summed up about 50% of the goals scored against the Oilers during the days of Eakins' swarm defense….

I’d love to see a more detailed analysis like this, but focusing on defensive plays. So much of corsi promotes maintaining puck possession, but inhibiting the other team’s possession and re-gaining possession are often over-looked. Can we identify unheralded forwards or defenders that excel at keeping the puck out of their green zone or from crossing the Royal Road. And even better, do they gain possession or at least get the puck out of their zone and relieve pressure on a consistent basis.



Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696680 is a reply to message #696657 ]
Tue, 27 June 2017 14:35 Go to previous messageGo to next message
Burgeoboy  is currently offline Burgeoboy
Messages: 26
Registered: July 2006
Location: Burgeo, Newfoundland

No Cups

Adam wrote on Tue, 27 June 2017 16:20

George wrote on Tue, 27 June 2017 12:42

Adam wrote on Tue, 27 June 2017 08:50

Thanks for this. Interesting read. I'm always fascinated by how many brilliant people from different backgrounds we get in here.

I was re-reading this piece ( http://edmontonjournal.com/sports/hockey/nhl/cult-of-hockey/ analytics-expert-says-kris-russell-of-edmonton-oilers-is-und errated-and-one-of-nhls-top-defensive-d-men) this weekend and I've been puzzling over it.


I really like that piece, though I’m probably biased as it speaks to my own concerns about corsi. It does raise a good point, corsi is embraced so much partially due to the fact that it is public data. It is a pretty objective metric that can be scraped from an NHL website. Conversely weighting each play in a game for a pre-determined value requires teams of people and an agreed upon standard.


Quote:

Werenka's made a believer out of Chiarelli and David Staples, but this part has bothered me:

Quote:

This kind of work comes down to dozens and dozens of decisions and judgements made by TruPerformance game analysts on every player every game. The decisions are subjective, but Werenka says training, guidelines and auditing is in place to make sure that the raters are reliable, that one rater will watch the same game and make almost all the same calls on the same plays and players.

TruPerformance has a set standard of rules analysts apply to judge each play. Auditing is done to make sure the raters are consistent. “Is there a level of variance? Absolutely. We operate on a level of five or eight per cent variance. We would like to keep it within five per cent.”

The key is to get good people to rate, Werenka says. “There are two types of people that are very good. Experienced people that have open minds. They decide through their experience that, you know, being physical or a check is one of the most important things that will happen in a game, and if they’re not open minded to change that then we’re in trouble. So we want experience guys that are open-minded, or guys that have a good basic understanding of the game and have an extreme learning curve, guys that can learn. One of the best compliments we can pay to those guys is, ‘He never asked the same question twice.’ Those are the guys we love.”

When it comes to the issues of rater reliability and subjectivity, Werenka has come to a few conclusions.

“You can say this is subjective, but there’s been a couple phrases we like to use: Consistent subjectivity equals objectivity. And how do you know if that consistent subjectivity is meaningful, if it makes sense, is right? That’s where your correlation to winning comes in.”


There is subjectivity in Corsi analysis as well - what constitutes a shot? And scoring chance metrics are even more so, but I wonder with Werenka's system if there isn't some built in biases. If for example, you were a defensive defenceman when you played and believed that your practice of off the glass and out was a valuable skill - after all, you're relieving pressure and possibly allowing linemates to change - might you rate similar players high?


Presumably they have some safe guards in place. They talk about variance levels, I'm sure they rotate guys so it's not the same one or two people marking the same teams, and they probably have an inter-rater reliability measures that are regularly updated. With all of that they /should/ get reliable data. If they're carrying out the analyses properly, they can really partial out a lot of subjectivity.

Quote:

They're very proud that their analysis has the top players in the game consistently at the top of their charts, but since there's a high level of subjectivity, might part of that be due to marker bias?


Of course, these players actually are the best in the league so it seems they're doing something right ;) I actually liked their argument here: the corsi players are obviously way off suggesting that it is a not a strong a predictor as some may believe. Again, if TruPerformance are collecting data using proper protocol, they should be able to hone in on objective values.

For those who haven't read the article, Top 5 Centres as rated by each metric:

TruPerformance: Sidney Crosby, 93%, Evgeni Malkin, 92%, Anze Kopitar, 91%, Patrice Bergeron, 89%, Tyler Seguin/Connor McDavid, 88%.

Corsi: Nick Shore, 61%, Pavel Datsyuk, 58%, Mike Ribeiro, 58%, Mathieu Perreault, 58%, Eric Staal, 57%,




One of my issues with Staples though is that he seems to group all shot-related data in with Corsi. I don't think it's a perfect metric by any means. We saw with Eakins what happens when a coach tries to run a system designed to improve shot metrics without regard to shot quality. It was a horrendous failure because correlation does not equal causation. Good teams shoot more, but just increasing the number of shots doesn't necessarily equate to more goals and more wins...Smytty clappers from the half-boards just surrender possession for a low percentage shot.

Corsi is just the beginning though and most of the stats guys are diving a lot deeper than Corsi now.

I think the work being done by Steve Valiquette has more relevance than the Werenka stuff (in part because it doesn't have a super secret metric that says Russell is a top-ten d-man in the league on a level with Marc-Edouard Vlasic).

http://www.thehockeynews.com/news/article/whats-the-next-dim ension-of-analytics-former-nhl-goalie-steve-valiquette-knows

He's spent a lot of time on what happens just before a shot, and how scoring chances are developed.





That article does not say that Kris Russell is a top ten d man, it clearly says he's not even a top pairing guy ( which puts him outside the top 62 at least) it says he's in the next level, with guys like Vlasic , who I was suspirsed to see rated that low. I am by no means an advance stats guy, this is the first time I even heard of top performance, but at first look, it seems to do a much better job then Corsi , based on who it list as the top centers and d men in the League .

One says Kris Russell sucks and shouldn't be in the nhl , it also doesn't rank Crosby or Mcdavid in the top 5 centers . The second says Kris Russell is a second pairing d man and that Crosby and Malkin are the top 2 centers....clearly the first one is better , because there's no way Kriss Russell is a second pairing guy ?

There is no prefect stat/metric yet, but if I had to pick between those two based on thier top rated guys , it's not even close . I am sure top performance as it issues , it sounds subjective, but let's not dismiss it cause it's doesn't say Kriss Russell is the worest hockey player alive. Is it possible these guys that apparently break down each player , play by play and assign a score to each play , might be on to something , rather then just looking at a stat sheet and counting shots for and against ?



Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696686 is a reply to message #696680 ]
Tue, 27 June 2017 15:01 Go to previous message
Adam  is currently offline Adam
Messages: 11761
Registered: August 2005
Location: Edmonton, AB

6 Cups

Burgeoboy wrote on Tue, 27 June 2017 14:35


That article does not say that Kris Russell is a top ten d man, it clearly says he's not even a top pairing guy ( which puts him outside the top 62 at least) it says he's in the next level, with guys like Vlasic , who I was suspirsed to see rated that low. I am by no means an advance stats guy, this is the first time I even heard of top performance, but at first look, it seems to do a much better job then Corsi , based on who it list as the top centers and d men in the League .

One says Kris Russell sucks and shouldn't be in the nhl , it also doesn't rank Crosby or Mcdavid in the top 5 centers . The second says Kris Russell is a second pairing d man and that Crosby and Malkin are the top 2 centers....clearly the first one is better , because there's no way Kriss Russell is a second pairing guy ?

There is no prefect stat/metric yet, but if I had to pick between those two based on thier top rated guys , it's not even close . I am sure top performance as it issues , it sounds subjective, but let's not dismiss it cause it's doesn't say Kriss Russell is the worest hockey player alive. Is it possible these guys that apparently break down each player , play by play and assign a score to each play , might be on to something , rather then just looking at a stat sheet and counting shots for and against ?


It's Werenka's stats that Chiarelli's touting and he's said a couple of times that in some super secret stat category, Russell is among the very best in the league.

You'll note that I've not defended Corsi as the be-all-end-all of stats. Like most, it tells you something, and there's a value in knowing it. I'm more of a believer in deeper numbers, because I think Corsi doesn't tell enough of the story.

My issues with Russell started when he was with Calgary. He was regularly watching as McDavid blew around him. We've seen why since he came here - his gap control is an issue. He backs off attacking forwards so as to better position himself for shot blocks. If they don't just shoot, that can be an issue. McDavid was extremely fast so giving him extra space is suicide. He just went around Russell again and again.

I believe he gives up the zone too easily and that he doesn't apply adequate pressure on the puck carrier. That's mostly from watching him - not from advanced stats. I don't know what the advanced stat would be that would prove that out.

I also think he's a terrible puck mover. He doesn't connect with his breakout passes, and often he doesn't even look to try...he just fires it up the boards and hopes for the best. If I was going to check some fancy stats based on breakouts, I'd like to know what the number of completed breakout passes are. That's not easily acquired, but I'd be asking for it if I was a GM.

The term 'rigid subjectivity' is what gives me the most pause. They are talking about extraordinary plays, like lunging to break up a two-on-one, but maybe the best players don't have to lunge - maybe they're in the right position and just break it up without leaving their feet (which creates its own issues - one of my problems with the Kris Russell shot-blocks). How much credit are they giving players for diving in front of shots? Is that why Russell scores high?

It's possible that the best players score the best because they're the best, but it's also possible that the model is skewed, and that they've designed it to measure based against the players who they thought were the best.

You are right that there's no perfect system, but I distrust the numbers that Werenka is doing, because Chiarelli talks about controlled zone entries and passing metrics, and that jars with both all the stats I've seen and the eye test.



"This team needs an enema!"
#FireLowe #FireMacT #FireHowson #FireBuchberger #FireHowsonAgain #FireChiarelli #FireMcLellan

Send a private message to this user  

 Re: Advanced stats, causality, and why numbers matter [message #696658 is a reply to message #696603 ]
Tue, 27 June 2017 12:50 Go to previous messageGo to next message
ziltoid  is currently offline ziltoid
Messages: 211
Registered: January 2011

No Cups

Adam wrote on Tue, 27 June 2017 08:50

Thanks for this. Interesting read. I'm always fascinated by how many brilliant people from different backgrounds we get in here.

I was re-reading this piece ( http://edmontonjournal.com/sports/hockey/nhl/cult-of-hockey/ analytics-expert-says-kris-russell-of-edmonton-oilers-is-und errated-and-one-of-nhls-top-defensive-d-men) this weekend and I've been puzzling over it.

Werenka's made a believer out of Chiarelli and David Staples, but this part has bothered me:

Quote:

This kind of work comes down to dozens and dozens of decisions and judgements made by TruPerformance game analysts on every player every game. The decisions are subjective, but Werenka says training, guidelines and auditing is in place to make sure that the raters are reliable, that one rater will watch the same game and make almost all the same calls on the same plays and players.

TruPerformance has a set standard of rules analysts apply to judge each play. Auditing is done to make sure the raters are consistent. “Is there a level of variance? Absolutely. We operate on a level of five or eight per cent variance. We would like to keep it within five per cent.”

The key is to get good people to rate, Werenka says. “There are two types of people that are very good. Experienced people that have open minds. They decide through their experience that, you know, being physical or a check is one of the most important things that will happen in a game, and if they’re not open minded to change that then we’re in trouble. So we want experience guys that are open-minded, or guys that have a good basic understanding of the game and have an extreme learning curve, guys that can learn. One of the best compliments we can pay to those guys is, ‘He never asked the same question twice.’ Those are the guys we love.”

When it comes to the issues of rater reliability and subjectivity, Werenka has come to a few conclusions.

“You can say this is subjective, but there’s been a couple phrases we like to use: Consistent subjectivity equals objectivity. And how do you know if that consistent subjectivity is meaningful, if it makes sense, is right? That’s where your correlation to winning comes in.”


There is subjectivity in Corsi analysis as well - what constitutes a shot? And scoring chance metrics are even more so, but I wonder with Werenka's system if there isn't some built in biases. If for example, you were a defensive defenceman when you played and believed that your practice of off the glass and out was a valuable skill - after all, you're relieving pressure and possibly allowing linemates to change - might you rate similar players high?

They're very proud that their analysis has the top players in the game consistently at the top of their charts, but since there's a high level of subjectivity, might part of that be due to marker bias?

I've also been thinking about how this plays in to the hiring of General Managers. I think the mistake that Arizona and Florida made is that the spreadsheet guy is really just a part of your scouting department. Advanced stats is about player analysis, which is only one piece of the puzzle when it comes to being a GM. What does a 26 year old number cruncher know about negotiating contracts or managing the salary cap? I think your best GM has a negotiating background, as he has to handle much of that himself, and is a good leader with great delegation skills and the ability to hire well and trust those he brings in. Having Lamoriello as your GM with Dubas as the assistant is a much better route than say, hiring Tyler Dellow to run your team.

Anyhow, I'm getting off topic, but I appreciated the post on stats, and it will be interesting to see how it evolves in the NHL, and how much we'll actually get to see (since the best analytics guys keep getting hired away).


Inter-rater reliability is, as I mentioned, something to think about. But there are ways to assess it and take it into account.

But I think you are spot on with you GM view. GMs need strong negotiating skills, but they also need an open mind re: stats. With Chia, I fear that he does not have an open mind; he has preconceived notions about what metrics he thinks are important, then directs the stats department to assess those metrics (that is the only way I can imagine Russell gets 4x4), which causes the big picture to get obscured. Instead, he should give the stats department a blank slate to analyze the numbers however they see fit, then have them report back to him.



Send a private message to this user  

 
Previous Topic:Offers on table for Eberle, as per Dreger...
Next Topic:Treenas oil is back!!!!!!
Oilers NHL Minors Speculation For Sale 


Copyright © OilFans.com 1996-2017.
All content is property of OilFans.com and cannot be used without expressed, written consent from this site.
Questions, comments and suggestions can be directed to oilfans@OilFans.com
Privacy Statement


Hosted by LogicalHosting.ca