Saturday, January 28, 2012

Quantifying the Value of a Football Manager

In today's FT Magazine Simon Kuper has an essay on the evaluation of football managers in England's professional leagues based on the work of his Soccernomics co-author Stefan Szymanski, of the University of Michigan.

Here how Kuper describes Syzmanski's methodology:
[F]ootball does a bad job of valuing managers. Football managers are modern celebrities, yet the vast majority appear to add no value to their teams, and could probably be replaced by their secretaries or stuffed teddy bears without anyone noticing. Only a few managers, such as Sturrock and Alex Ferguson, consistently improve their teams. Yet some of these excellent few get overlooked. All these findings emerge from a 37-year study of English football managers...

If players’ wages determine results, it follows that everything else – including the manager – is just noise. Most managers are not very relevant. In the long run, they will achieve almost exactly the league positions that their players’ wages would predict.

Still, there is an important caveat. Players’ wages don’t explain everything – merely almost everything at most clubs. That leaves room for a few good managers to make a difference. The question then is: which managers finish consistently higher with their teams than you would expect given their wage bills? Or, to borrow a phrase from Real Madrid’s manager José Mourinho, who are the special ones?

Szymanski has tried to answer that question. He analysed the financial accounts of four-fifths of English professional clubs from 1973 to 2010, and identified the managers who consistently overachieved. These men are the elite.

We should note right away that Szymanski’s model gives more credit to overachieving managers at the top of football than at the bottom. England’s 92 professional teams are spread over four divisions. A manager in League Two who has the 90th smallest budget in England but manages to finish 80th nationwide is overachieving. However, a manager with the third-highest budget in England who wins the Premier League is probably overachieving even more.
Syzmanski looks at team performance (focused on 251 managers out of 699 total who held the position for 5 years or more and for which financial data was available for the period 1974 to 2010),  after accounting for the expected position given the wages, which exhibit a strong correlation with league table position, as shown in the graph below from UEFA (here in PDF).
After accounting for wages, Szymanski then attributes the balance of unexplained variance in final league position to the "manager effect," an assumption with shortcomings that Szymanski is well aware of -- Kuper explains "factors besides the manager might have caused each club’s overachievement."

The attribution of unexplained variance to the "manager effect" is a serious weakness in such studies.  This can been demonstrated quantitatively by looking at another recent effort to quantify the value added by the manager.

Bell at al. (2011) attempt to evaluate managers in a similar fashion to Szymanski, using less data and a more complex statistical methodology. Their paper, titled, "The Performance of Football Club Managers: Skill or Luck?" evaluates managerial performance 2004 to 2009 in the Premier League at the match level and account for a range of variables, such as injuries, suspensions, transfer spending . They find, as did Szymanski, that weekly wages alone explains more than half the variance in points awarded (56% to be exact).

But then they do something that I don't quite follow -- they create a complex multiple regression model that throws in a suite of variables (some of which are not statistically significant) and end up explaining only 20% of the variance (see their Table 1, p. 21), which represents a severe degradation from the simple bivariate model. They then attribute the remained 80% of unexplained variance to the "manager effect." If I've understood their methods correctly, this simply seems implausible.  (I've emailed the authors and they are welcome to correct any misinterpretation.)
Despite the different approaches, the results of Bell at al. and Szymanski are similar -- for instance, the best managers in the Premier League appearing on both lists are Ferguson, Wegner, Beneitez, Moyes.  Each of these managers has led teams with unusual success (Everton was top 6 in 3 of the 5 years looked at by Bell et al. and won promotion with Preston North End in the longer period looked at by Szymanski). So yes, these teams outperformed, with either Manchester United or Arsenal winning the Premier League championship in 15 of 20 seasons.

But were these teams successful because of the manager?  Or were the managers successful because of the teams? What if it was both? And more? The only way that these studies can answer this question is by assuming the role of the manager in the variance to be explained, which is, unfortunately, the exact relationship that these studies are trying to pin down. So if you buy the assumptions of how to attribute the "unexplained variance" then these studies provide an answer. But if you don't buy the assumptions, then you wind up right where you started.

Consequently, I am not yet convinced that anyone has solved the riddle of effectively quantifying the value added by a manager, though the efforts by Syzmanski and Bell et al. represent a good start.

2 comments:

  1. Comment by email for posting from Stefan Syzmanski ...

    "I'm not sure I exactly agree with the way you explain the problem, but I agree there is a problem. I would start with wages.

    How do we know that wages cause success rather than successful teams choose to pay high wages (reverse causation)? We can argue from theory - why would owners choose to pay high wages if they didn't need to- why not use the money for something else? I think this is pretty persuasive, but it's not a statistical argument.

    The statistical problem is that there is no way of separating high wage paying teams from successful teams. Imagine we could randomly assign players to teams, then we would observe the impact of high wages on team performance.
    Obviously we can't do this, but there is a potential fix. If we can find a variable that is correlated with wages but not causally linked with performance then we can use that in the regression instead of wages.

    It's tricky to think of such a variable. One example might be climate- teams based in locations with worse climates will usually have to pay more for the same player, so this will be correlated with wages, but presumably is not very closely correlated with performance.

    However, for my English football club database it would be hard to collect the data over 37 years (does it even exist? And there probably isn't enough variation in climate between teams to make this viable (how would you compare Everton and Liverpool- 1 mile apart?).

    I'm currently working on the problem with Thomas Peeters, a PhD student from Antwerp who's spending some time over here. Because the football club accounts have so much other information (e.g. balance sheet and profit and loss data), we're hoping that one of these variables might fit closely with wages while having no obvious link to performance.

    So, if we can solve that statistical problem, I don't think there's too big a problem to attrbuting residual success to the manager. In general a manager is with a club for only a few years, so I think it's hard to believe that there are other factors that will influence performance of the team but only during the tenure of the manager. There are so many managers, and so many of them statistically insignificant, that I think there genuinely is something special about the individual on those rare occasions when something out of the ordinary happens."

    ReplyDelete
  2. Stefan-

    Many thanks for the comment ... a few thoughts in reply:

    1. If you look at the figure above from UEFA on the relationship of wages and success, you can see evidence that the wage effect diminishes as wages increase, which would tend to support the idea of bi-directionality in wages <--> success (or at least an intervening variable, like a Sheik!).

    2. With so much variance explained by wages (in the data that you cite, reproduced at the top of this post from your book), the "manager effect" is likely to be small in many cases and difficult to detect in the best of situations (small N problem).

    3. I agree that a "manager effect" that persists over time and (especially) across teams is the most likely to be identified.

    4. Given the non-stationary nature of the data and underlying relationships, I am not sure that aggregate statistical methods are possible of resolving the "manager effect." Perhaps a focus on manager decision making (e.g., evaluating the consequences of decisions on line-ups, substitutions, transfers, etc.) might also bear some fruit. Data which shows that manager that (a) performs above average according to aggregate statistics, and (b) can be shown to make good decisions based on their outcomes (over time), would be convincing to me, in fact, I'd expect (b) to largely cause (a).

    Thanks again!!

    ReplyDelete