Michel de Nostredame (1503–1566). Portrait by César de Nostredame. Public domain, via Wikimedia Commons.
And why focusing on evidence rather than prediction may offer a more reliable path for investors.
For centuries people have searched for individuals capable of predicting the future. From ancient oracles to Nostradamus, the promise of foresight has always been alluring.
Financial markets attract some of the brightest minds in finance. Portfolio managers, analysts, economists and quantitative researchers dedicate enormous resources to analysing markets and forecasting what might happen next. Occasionally, some active managers do outperform the broader market.
The challenge for investors, however, is not determining whether outperformance exists. The real challenge is identifying in advance which managers will deliver it in the future.
Identifying yesterday’s winners is easy. Identifying tomorrow’s winners is far more difficult.
Financial markets are also remarkably effective at incorporating information into prices. Every day millions of investors analyse data, interpret news and adjust their expectations, with those collective views quickly reflected in market prices.
This reality sits at the heart of one of the most enduring debates in investing.
More than a century ago, the British statistician and scientist Francis Galton, a cousin of Charles Darwin and grandson of Erasmus Darwin, observed something remarkable at a country fair in Plymouth, England. Visitors were invited to guess the dressed weight of an ox. Nearly 800 people participated, submitting estimates that varied widely. Yet when Galton analysed the entries in a paper titled Vox Populi, or “the voice of the people,” he found that the crowd’s median estimate was within roughly one percent of the correct weight. The collective judgement of the group proved more accurate than almost any individual guess.
Financial markets operate in a similar way. Across dozens of exchanges and tens of thousands of listed companies, millions of investors and institutions analyse information and express their views through millions of transactions each day. The result is a powerful system that continuously aggregates vast amounts of information into market prices.
Outperforming such a system consistently requires predicting the future more accurately than the collective judgement of the market itself, time and time again.
That is a formidable challenge.
Markets today are more competitive than at any time in history.
Institutional investors, hedge funds, quantitative managers and algorithmic trading firms analyse markets continuously. Information travels instantly and the tools available to investors have never been more powerful.
At the same time, the rise of passive investing has transformed the competitive landscape. Index funds and exchange-traded funds now represent a growing share of global equity ownership. As Morningstar recently noted, passive investing has reached a point where index funds now manage a larger share of U.S. equity fund assets than actively managed funds.
Market Share: All Funds
Active v passive market shares, all long-term US mutual funds and ETFs, February 1993 to January 2024.
As passive investing grows, a larger portion of the market simply accepts prevailing prices rather than attempting to forecast them. This leaves a smaller group of active managers competing with one another to identify mispriced securities and generate returns above the market benchmark.
In effect, the competition for these excess returns becomes more concentrated and more intense.
Over time, this dynamic raises the bar for active management. Managers who consistently underperform tend to lose assets or exit the industry, leaving a smaller group of increasingly sophisticated competitors.
Research from S&P Dow Jones Indices, through its SPIVA scorecards, illustrates the implications. The findings are strikingly consistent, repeating themselves across different time horizons. Over long periods the majority of active managers underperform the market benchmarks used in the study. In Australia, more than 80% of active equity managers have underperformed the S&P/ASX 200 over the past decade.
This does not mean skill does not exist. But it does highlight how difficult it is to identify that skill in advance. Short-term performance can easily be mistaken for skill when it may simply reflect luck.
A useful analogy comes from basketball. Imagine two players attempting three free throws. One player is highly skilled, the other simply fortunate. Over a few attempts the lucky player might appear superior. But with each additional shot the odds increasingly favour the skilled player while the chances of the lucky player maintaining their streak diminish. Over hundreds of attempts the probabilities reveal themselves.
Investment performance often behaves in a similar way.
Another challenge lies in the structure of equity markets themselves. Returns are highly skewed.
Research by Hendrik Bessembinder has shown that only a very small number of companies account for the majority of long-term wealth creation in equity markets. In fact, his work suggests that roughly 4% of listed companies have generated the entire net wealth creation of the U.S. stock market over many decades.
Recent analysis titled Skewing Success by Anu Radha Ganti, CFA from S&P Dow Jones Indices highlights the same phenomenon, noting that equity market outcomes are heavily skewed toward a relatively small number of extraordinary long-term winners.
S&P 500 Stock Returns Are Positively Skewed
This dynamic is also visible in periods of heightened market concentration. At various points in history, a small group of companies has come to represent a significant share of the market’s value and performance. The dominance of railroads in the early twentieth century, technology companies during the late 1990s, and the large technology firms of today all illustrate how market leadership often becomes concentrated in a relatively small number of companies.
While these periods can appear unusual at the time, history suggests they are a recurring feature of equity markets rather than an anomaly.
For active investors this creates a difficult challenge. To outperform the market, a portfolio manager must successfully identify and hold these rare outliers in advance. Missing even a handful of these exceptional performers can materially reduce long-term returns relative to the market.
In markets where outcomes are so unevenly distributed, diversification becomes critically important. Broad exposure increases the probability of capturing the companies that drive long-term wealth creation rather than relying on the difficult task of predicting which individual companies will become the market’s future winners.
Investor behaviour can make this challenge even more difficult.
Periods of strong performance often attract attention and capital. Successful strategies can appear like bright lights drawing investors in. Yet markets rarely move in straight lines. Strategies that experience dramatic short-term outperformance may later experience equally dramatic reversals.
In some cases, this pressure can lead to style drift. Investment managers may gradually move away from their stated philosophy in an attempt to capture what is currently working in markets. While this may produce periods of short-term success, it can introduce additional risks and make it harder for investors to understand how a strategy is likely to behave through different market environments. Research from S&P Dow Jones Indices through its SPIVA Persistence Scorecards shows that fund performance frequently migrates across quartiles over time, highlighting how difficult it is for managers to remain consistently among the top performers.
Investors can find themselves enduring significant volatility only to arrive years later at returns that closely resemble the broader market. It is a little like travelling to the same destination by two different routes. One road is a smooth, paved highway. The journey may be slightly longer, but it is predictable, fuel-efficient and easier on the vehicle.
The other route is more scenic and perhaps more exciting. But it involves dirt roads, potholes and constant detours. The ride is rougher, fuel consumption is higher and the wear and tear on the vehicle is greater.
If both journeys ultimately lead to roughly the same destination at about the same time, many travellers might reasonably ask why they would choose the more stressful path.
For investors, the same principle applies. If similar long-term outcomes can be achieved with a smoother and more predictable journey, it is often easier to remain disciplined and sleep well at night along the way.
Markets also generate an extraordinary amount of commentary. Forecasts, headlines, predictions and opinions appear daily across financial media. During periods of volatility this noise can become even louder.
The documentary Tune Out the Noise, produced by Dimensional Fund Advisors, highlights an important lesson for investors: financial markets function as vast information-processing systems. As David Booth, Founder and Chairman, explains in the film, “the market is a big information-processing machine.”
Prices continuously incorporate the expectations of millions of participants, each acting on their own information, analysis and beliefs. Rather than trying to outguess those collective expectations, investors can instead allow the market to do the heavy lifting. As Booth puts it, investors can simply “sit back and let them duke it out.”
Attempting to react to every headline or forecast can easily lead investors to abandon long-term strategies at precisely the wrong time. Maintaining discipline often requires the ability to tune out the noise and remain focused on long-term evidence.
If predicting the future is so difficult, investors might reasonably ask whether there is another way to approach markets.
One alternative is to focus less on forecasting what might happen next and more on what research and data already tell us today. Decades of academic work have identified characteristics that have historically influenced investment returns. Much of the data and research that underpin systematic investing are publicly available.
The challenge lies not in accessing this information, but in applying it effectively. Financial datasets contain their own “gremlins”, and designing robust processes to address them requires expertise and discipline. The intellectual property for systematic investment firms often lies not in the raw data itself, but in how that data is applied in a disciplined and repeatable way.
Systematic investment strategies attempt to translate these insights into repeatable portfolio construction processes. Rather than relying on discretionary forecasts, systematic approaches follow defined rules grounded in research and data. This is not a new or revolutionary idea. In many respects it reflects decades of academic research into how markets behave.
Systematic investing often attracts less attention because it lacks the drama associated with bold predictions or star portfolio managers. Instead, it focuses on disciplined processes designed to capture small, repeatable advantages over time.
By following clearly defined rules, systematic strategies can reduce behavioural biases and allow investors to understand how a strategy is expected to perform through different market environments, setting expectations in advance.
The growing recognition of research-driven investing can also be seen across the investment industry.
Dimensional Fund Advisors, founded on academic research and the work of leading economists, has built a global investment business with more than a trillion dollars in assets.
Other firms have begun incorporating systematic capabilities alongside traditional discretionary approaches. For example, Magellan Financial Group acquired a majority stake in Vinva Investment Management in October 2024, recognising the complementary role that systematic investing can play alongside traditional portfolio management. Vinva has since made a positive contribution to Magellan’s balance sheet, highlighting the growing commercial importance of systematic investing within diversified investment management businesses.
These developments reflect a broader shift within the industry toward combining research, data and disciplined implementation.
Technology is continuing to reshape the investment landscape.
Artificial intelligence and advances in data science are already influencing how investment firms analyse markets, process information and construct portfolios. Large investment firms, hedge funds and quantitative managers are using increasingly sophisticated tools to analyse vast datasets and process information faster than ever before.
One important implication of this evolution is that financial markets are becoming even more competitive. When more advanced technology is analysing the same information simultaneously, market prices may reflect that information more quickly and efficiently.
Paradoxically, this suggests that the rise of artificial intelligence may not make forecasting markets easier. Instead, it may make it more difficult for any individual investor or manager to consistently outperform, as the collective intelligence embedded in market prices becomes increasingly powerful.
Some observers speculate that AI could transform the investment industry entirely. It may evolve as a powerful tool within investment firms, augmenting human decision-making and enhancing research and portfolio construction. Others suggest it could go further, potentially replacing elements of the traditional investment management model over time.
Yet predicting exactly how these technologies will evolve may prove just as difficult as forecasting equity markets themselves. In that sense, the discussion around artificial intelligence reinforces a broader reality: predicting the future, whether in markets or technology, is inherently difficult.
A more practical approach may be to focus on what these tools can do today, applying them where they enhance research, portfolio construction and risk management, while recognising the limits of prediction.
Even in a world of increasingly sophisticated technology, the human element remains important. Investors entrust managers with their savings, their retirement capital and often the financial security of their families. Stewardship of that responsibility requires judgement, transparency and respect for the people whose assets are being managed. Technology may enhance investment processes, but trust, accountability and human understanding remain central to the relationship between investors and those entrusted to manage their capital.
Markets incorporate enormous amounts of information and reflect the collective expectations of millions of participants. Consistently forecasting those outcomes more accurately than the market itself is a formidable challenge.
Investors may therefore be better served focusing less on predicting what markets will do next and more on building robust portfolios grounded in research, diversification and disciplined processes.
At Hamilton12 we have taken a similar approach. Our research has focused on what long-term data already tells us about equity markets rather than attempting to forecast what markets might do next.
The S&P Dow Jones Indices-calculated Hamilton12 Australian Diversified Yield Index, which underpins our investment strategy, was developed using this philosophy. The objective is simple: deliver income and total returns above the benchmark through a systematic and diversified process grounded in research and data.
This same approach also underpins the Hamilton12 Australian Shares Income Fund, which applies the index methodology in a portfolio designed to provide investors with strong income while maintaining broad exposure to the Australian equity market, importantly without sacrificing total returns.
For centuries people have searched for individuals capable of predicting the future with certainty. Nostradamus captured the imagination of generations with cryptic predictions that continue to be debated today.
Financial markets, however, offer a powerful reminder of just how difficult forecasting the future can be.
Rather than relying on predictions, investors may be better served focusing on what research and evidence already tell us about how markets behave. In investing, disciplined processes grounded in evidence may prove more reliable than attempts to predict the future.
Over the long term, evidence and discipline may prove far more valuable than prediction.
