Confessions of an Index Investing Skeptic — Part I: Diversification Ain’t What it Used to Be

Image by cliff1066

Nobody goes there anymore. It’s too crowded.
— Yogi Berra

I first heard of index funds in the 90s, although they had been around since 1975. Their popularity grew relatively slowly at first, but has skyrocketed in the last 20 years or so.

A number of factors have contributed to this growth, but one of the biggest has been the steady trend away from defined-benefit pensions to defined-contribution retirement plans like 401(k)’s. These plans typically offer limited choices, most often comprising a defined set of mutual funds to choose from. As word got out that few managed funds could consistently outperform the indexes (a point I certainly do not dispute), it’s no surprise that more and more Americans have elected to invest their retirement savings in them.

Millions upon millions of people now steadily pour money into index funds every month. While it’s great that so many people have figured out that it’s a good idea to avoid the high fees and spotty performance of actively managed mutual funds, this also presents a problem.

The Crowd Ruins Everything

The trouble with great investment ideas is that they almost always contain the seeds of their own destruction. In anything resembling an efficient market, as soon as word gets out about any edge that can be gained, hordes of people pour in and shift prices until the edge disappears.

I hate to say it, but I’ve come to suspect that index investing is no exception. To see why, have a look at the following chart: 1

S&P 500 Member Correlations over time

This chart shows correlation levels among the stocks that make up the S&P 500 index since 1993. Correlation essentially measures the extent to which stocks move together as opposed to independently.

During period the chart covers, the percentage of all U.S. equity assets being passively managed grew from roughly 10% to about 30%. As you can see, during the same period, correlations have trended steadily upward, hitting all-time highs over the past couple of years.

This is hardly surprising, when you think about it. Almost a third of all equities in the market are now bought and sold in massive quantities, all together, based on absolutely no analysis whatsoever of their individual worth. That is, after all, what index investing is. And when all stocks in an index are bought and sold together, the natural result is that their prices will tend to move together.2

Not Your Father’s Diversification

One of the biggest selling points of index funds is the great diversification they provide. Diversification helps mitigate risks, lowers volatility, and smooths out returns, at least in theory. Unfortunately, that all starts breaking down when correlations rise. The more stocks move together, the less protection from volatility buying them all at once provides.

Does that mean diversification no longer has any benefit? Of course not. But the same level of diversification offers much less protection now than it used to. In my mind, that raises a question: What would happen if everyone invested exclusively in index funds?

Well? What do you think? What would happen?

I will leave you with a story to reflect on. It’s called the Parable of the Ox. Read it, and see what ideas it gives you.

See you next time.

  1. Source: SSgA by way of Seeking Alpha.
  2. I don’t claim that the rising popularity of index investing is the sole cause of this trend, but it would be difficult to argue that it isn’t a major factor.


  1. What would happen if everyone only bought index funds?

    Well if that happened I’d guess it’d follow similar to what happens when “everyone” buys into the same investing “fad”. Everyone suggests to everyone that “this is it!” or whatever and more and more people pour their money in which pushes the “fad” investment up and up in price into a bubble.

    At some point though the fad investment gets a bit shaky and for one reason or another starts to decline a bit. Maybe it’s a world shift or maybe consumer opinion changes – whatever – but the investment goes down maybe just a little… maybe a lot. Then the people all get scared and the mad pull out occurs sending the investment tumbling down.

    If we put that scenario to index funds though imagine if you had $1,000,000 in the S&P500. Over the years more and more people say it’s the bee’s knee’s and pour their money into it (what’s happening now). Say there’s then a MASSIVE world crash (bigger than the GFC even), think circa 1929 or 1987. Your $1mil is now worth say $50,000… but you’ve still got just as many shares in the S&P500 as you had before… and the companies that make up the S&P500 are the same, massive, international conglomerates that make dependable brands that everyone the world over not wants but often NEEDS to buy every year.

    As a result I’d expect for the S&P500 to start slowly going up as those companies continue to make profits and sell goods. People would realise that these companies produce dependable results and produce a valid product. Sure many would have sold their shares by that point and “lost it all” but if you held on the prices should rise again. If they don’t… then you’d have the case where the worlds biggest corporations are still running… it’s just their “imagined net worth” as that Ox story goes is said to be less. However as this is an imagined value, once everyone’s over the big media hype etc shouldn’t their faith in the companies (that would be still running fine mostly) return and the price go back up? Or are my assumptions a bit off?

    Hmm… that’s a long comment lol

    • Sean Owen

      I should have worded the thought experiment a little better. I know I said imagine if people only BOUGHT index funds, but that’s not quite what I meant. What I meant to ask was, what if people only INVESTED in index funds. They could buy or sell, but only indexes.

      I’ll update the wording so it’s a little more clear.

  2. First post ever

    My strategy is to invest in index funds, but simply because I can’t come up with a better way to invest. I’m really looking forward to this series, and I hope you’ll make me realise something new.

    That being said, I disagree with the hypothesis in this post. The fact that correlation increased over time is interesting, but could be caused by a number of factors. Some of these factors would take away from the power of diversification. If index investing is one of the driver, it doesn’t bother me at all.

    Your thought experiment is the best way to demonstrate why. Assume 100% of the investment in the stock market is m&m index funds. What happens? One clever guy can make insane return by applying some basic fundamental analysis. Is the world short of clever fundamental analysts? No. (Please don’t read this as me implying that the average analyst is clever)

    Playing your thought experiment in the long run, it seems that the market will balance between a bunch of dumb lambs following the average (me) and a bunch of analysts (some clever, some not) who will drive each stock to its true value.

    • Sean Owen

      To be fair, I noted in a footnote that I don’t think index funds are the only thing driving the rise in correlations. The death of pensions and rise of the 401(k) is another big factor. I’m sure there are others as well.

      Regarding my thought experiment, I think your conclusion is absolutely correct, apart from one quibble: I asked what would happen if EVERYONE only bought index funds. EVERYONE means there are none of those clever actors you mentioned left.

      Of course that will never happen, so setting that extreme condition aside, your conclusion is correct, and it’s at the very core of where I am going with this. It’s the silver lining, in fact.

      Spoiler alert: The vast herd driving up correlations and distorting market prices will create fantastic opportunities for those willing to educate themselves. You can BE one of those clever actors you mentioned.

      Thanks for the excellent comment.

  3. Hi Sean….

    Just put this up on my current post in response to your comment there and thought I’d share it here as well:

    My pal Sean runs a fine blog and recently has been posting a series of “Confessions of an Index Investing Skeptic.” Here’s a link to the first:

    Several readers here have asked for my response to it, including Sean. Unfortunately, while I am keenly interested in his ideas, his series comes at a time when I am just slammed. Moreover, as thoughtful as they are, they deserve a close reading.

    In the comments below you’ll find some of this conversation, but I wanted to include my last here so that, for those of you interested, it’s not buried:

    Hi Sean…

    I gave your first three parts a quick read and, while I’d want to read more closely and think a bit more deeply about your ideas, the points you make seem valid. But they wouldn’t dissuade me from the index strategy I suggest.

    You appear to be exercising a bit of philosophical “what if” analysis. What if the pace of index investing grows to overwhelm the market. If, as you say, “Everybody” indexed, we might have to reconsider. But I don’t see that happening. Two reasons:

    1. As I’ve said before, human nature being what it is, people will always look for the angle. Even now, when the evidence is clear that out-performance is virtually impossible, there is a HUGE Wall Street industry based on the premise that it can be done.

    2. If Indexing were ever to reach levels where it truly distorted the market and opened up genuine out-performance opportunities, the tide would turn on a dime.

    Further, if Indexing grew to 100% of the market stocks would still be competing with other asset classes –Bonds, precious metals, commodities, REITs and the like — for investment dollars. This would require some relative valuation analysis, so that wouldn’t entirely disappear. But, again, it ain’t gonna happen.

    As you say, as usual, we seem to agree far more than we disagree. And, as I predicted, your take is thoughtful and insightful.

  4. Unless the government mandates that people are only allowed to invest in index funds (and then takes over their finances so they can’t make side bets) I think a more practical but still extreme question is “what happens if 99% of people chose to invest in index funds?” In that case there may be a huge rise in correlation and easier profits for the few who remain as active investors.

    But you also need to ask what happens to the active investors in this scenario. It is a mathematical requirement that half of the people who try to do better than the index will in fact do worse (and that’s before they pay for expensive experts). The underperformers are giving away money to the indexers and outperformers, and in turn the indexers are giving away money to the outperformers.

    The problem is that if there are only 2 active investors in the world, one will still be an underperformer and you probably can’t guess which one it will be. To be the best you have to be better than everyone else. Who is everyone else and how good are they? I have no clue since I don’t know even a tiny fraction of a tiny percentage of the market participants.

    Even a handful of active investors could keep markets balanced if they have enough capital. Like arbitrageurs and market makers, they provide a service that helps other investors by making prices more accurate and they can make a profit by doing so.

    To those like me who don’t want to spend the time needed to declare that one company is better than another, indexes will always provide valuable diversification even if they get a lot worse than they are now (note the wild swings in correlations in your chart – I imagine indexing popularity as more of a smoothly growing line). At the extreme, bankruptcy would still serve to remove unfit businesses if active investors weren’t doing enough. As an investor I am completely satisfied with getting a little bit of profit from a lot of businesses and not having to care if one or two of them go out, rather than the other way around. It’s the best worst option available 🙂

    • Sean Owen

      “It is a mathematical requirement that half of the people who try to do better than the index will in fact do worse”

      The index does not represent the median result achieved by market participants. Remember that individual participants trade in vastly different amounts. You can easily have 99 winners and 1 big loser.

      “The problem is that if there are only 2 active investors in the world, one will still be an underperformer and you probably can’t guess which one it will be.”

      Not so. If all but two people agree to sell an asset for less than its worth, the remaining 2 can both profit enormously.

      “Even a handful of active investors could keep markets balanced if they have enough capital.”

      What constitutes “enough” capital will continue to grow as more and more passive investors enter the market. And rising correlations suggest (to me, at least) that the pricing power of the smart money is already beginning to be overwhelmed by the sheer volume of passive and other artificial trades.

      “indexes will always provide valuable diversification”

      Diversification is only valuable to the extent it isn’t nullified by correlation.

      • Good points – for practical purposes I assume that there will be a distribution of active investors that each have lots of capital rather than one big loser and a lot of small winners. You could imagine a much more extreme scenario than I did 🙂

        If there are only two active investors they have to trade with each other, otherwise they are both just buying the index. One will do better than the index and the other will do worse. They may both have a net gain (the indexers would too then) but one of them could have done better with less work which is the whole point of buying the index.

        An alternate cause for rising correlations might be things like HFT and hedge funds. Even as they make the market more efficient overall they can cause correlations to increase based on their algorithms or the simple fact that a hedge fund that loses in China has to sell assets in Europe.

        It’s true that diversifying into assets with perfect correlation is pointless. So you have to ask, correlation over what time period? Will the total return of all stocks in the S&P 500 from today to Dec 31 2023 be 70% correlated?

        In the end this highlights same potential risk factors for indexing but I think they will be highly visible before we get anywhere close to that point because it requires such an unlikely extreme (just like when the index is trading at a P/E of over 35 or under 5 and you can tell something is going on). The extremes it takes to disprove indexing are why I like it.

  5. Raymond

    A point which I find interesting regarding almost universal indexing is that even indexes aren’t truly passive.

    From time to time, a stock will be removed from an index as its market cap no longer makes it suitable for it, while allowing it to enter another index. Small/Mid/Large caps as an example

    In order for indexing to still work, each investor would have to invest in each index in the proportion of the total value of its constituents. If everyone wishes to overweight small caps then new shares of these companies will have to be issued at a premium to provide the required liquidity, then the biggest small caps would be removed from the index, etc.

    Also it would open huge arbitrage opportunities if the index reconstitution date is known as people who could guess the next stock to be part of the S&P500 could purchase a lot of shares and wait for ETFs to fight each other to buy from them.

    Which opens another potential issue, as most indexes are based on the “float” which is the portion of the capitalisation of a business which is traded publicly, therefore a big chunk of the shares in some corporations, owned by institutional investors or founders is not taken into account in the index… until they wish to sell and convert their shares to a different class.

    Also, new public offerings of shares, which are dilutive to existing shareholders in an efficent market, would force the ETFs to buy more shares at the market price, although each share represent a smaller part of the business, basically creating value out of thin air.

    The opposite of previous idea is concerning share buybacks. For a business with stable earnings, buying back its own shares is a good way to put a floor under its share price by removing shares from weak hands, and boosting earning per share for the remaining investors. However if indexing is universal, such share buyback would be a losing proposition as the company will be a smaller part of the index.

    The combination of issuing new shares and buying them back based on the value of the index would allow a business to always have buyers and sellers available, allowing to force investors to buy high and sell low, which would make a really inefficient market.

    Overall the performance of any business would depend more on investor sentiment than on management performance, unless index managers create many more rules for constituents ponderation to prevent this abuse, which would result in active indexes… based on analysis of the merits of each company… and people choosing carefully which index to follow!

  6. HFT dude

    From the perspective of someone who worked in the field of algorithmic trading for five years, my opinion is that the increase in correlation between stocks probably has more to do with institutions running correlation trading algos than with retail investors buying more index funds.

    Pair trading is maybe the most basic form of correlation trading. It monitors two historically correlated stocks (e.g. Pepsi and Coca Cola). When the correlation between them weakens (say one ticks up while the other ticks down) the algo will then buy the stock that went down and short the stock that went up. This is an oversimplified example, but such a strategy will increase correlation between stocks.

    The trading volume generated by those algorithms is immense and dwarfs the impact of retail indexers. Interestingly, when trading was halted on the NASDAQ last week, a significant drop in volume was observed on the NYSE because correlation algorithms had no data to work with and just stopped trading.

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>