First, a few words about favorable bets. Popular books often show excessive fascination with the rare cases where one can exploit casino games (card-counting in blackjack) or badly-designed lotteries. Regarding traditional betting against bookmakers, on sports and horse racing, there is a lot of mythology and not much hard data on long-term success of professional gamblers. To overcome the bookmaker's margin you need to be exceptionally good at assessing probabilities, and I don't recommend this career choice. However the typical reader has a more readily available favorable bet: increase your insurance deductibles to the maximum you can comfortably afford to lose. Alas this does not save you enough money for a comfortable retirement.
People seem to categorize stock market investment as different from gambling, and it is important to recognize both similarities and differences. By history and in accord with the economic logic of capitalism, stock market investing has been an example of a favorable game, and unless one anticipates some revolutionary change one may reasonably expect it to remain so remain in our lifetimes. While we explicitly recognize sports outcomes -- next year's Superbowl winner, for instance -- as unpredictable and suitable for assessing probabilities, we are less conscious of the range of possibilities for the S&P500 index after a year, because we cannot readily see explicit odds. When media pundits predict a Superbowl winner we don't believe they really know, but predictions for the stock market have often been assumed more authoritative, even though evidence suggests they are totally useless.
Anyway, here is my suggested starting point for thinking about the stock market. The idea
to invest successfully in the stock market, you need to know whether the market is going to go up or go downis just wrong. Theory says you just need to know probability distributions. That is, probability distributions of future returns of different possible investments. So suppose (a very big SUPPOSE, in practice!) at the beginning of each year you could correctly assess the probability distributions of the return of different possible investments over the coming year. Then you can use the Kelly criterion (which is described below) to make your asset allocation, that is your "portfolio" of different investments. The fact that the distribution, and hence your asset allocation, would be different in different years is not a drawback – this strategy is still optimal for long-term growth.
The Kelly Criterion. Suppose you have a range of possible investment portfolios \(\alpha\), which will produce return \(X_\alpha\), with known distribution. Then choose the portfolio \(\alpha\) that maximizes the quantity \(\mathbb{E} [\log (1+X_\alpha)]\).The key background assumption is compounding -- any interest is reinvested. In the "i.i.d. version", that is if returns in successive years are independent and have the same probability distribution, the math argument is a simple consequence of the law of large numbers, because \(\log\)(value after \(n\) years) is a sum of independent random variables, and the quantity \(\mathbb{E} [\log (1+X_\alpha)]\) is the long-term growth rate.
Whether this mathematical fact is useful to you in the real world depends on two judgments. One, which can be said mathematically, is the long term issue; as described below, following the Kelly strategy involves more short-term risk than you might be prepared to take. The second is that to use the criterion one needs to know the true probability distribution of future return, and of course no-one knows this. I state my own view as follows.
As a default, assume the future will be statistically similar to the past. Not because this is true in any Platonic sense, but because anyone who says different is trying to sell you something.
The mathematical analysis starts with a model of a very simple setting: one risky asset (an index fund) and one non-risky asset (a short-term U.S. government bond), and the math is simplest if we measure returns relative to the interest rate on the latter. In this setting the only choice you can make is what proportion \(p\) of your portfolio to allocate to the risky asset. In the traditional random walk (or Brownian motion) model, returns for the next time period are random. with some mean \(\mu > 0\) and some variance \(\sigma^2 > 0\). Here the Kelly criterion says that the optimal proportion to choose is roughly \[ p = \mu/\sigma^2 . \] Stating meaningful historical values for these parameters for the S&P500 index is much harder than you might think (they are sensitive to start- and end-points, and to inflation adjustments) but typical stated values are \(\mu = 5.6\%\) and \(\sigma = 20\%\), as excess over the short-term interest rate. This makes the optimal \(p \approx 140\%\), meaning you should borrow money and invest your own and the borrowed money in the risky asset. But note this involves the unrealistic assumption that interest rates for borrowing and lending are the same; for many reasons I don't recommend actually doing this, though historically it would have worked well.
Moving on from the simple setting above, the Kelly criterion still works however many "asset classes" are available, and generally reinforces a common-sense "diversify your portfolio" principle. A quick way to see this how this works is to type some stocks into WolframAlpha, such as "apple IBM microsoft" and scroll down the resulting page to see "mean-variance optimal portfolio" which suggests how to divide your hypothetical million dollar portfolio amongst the chosen stocks, plus the index, bonds and T-bills.
40% chance that at some time your wealth will drop to only 40% of what you started with.The magical feature of this formula is that the percents always match: so there is a 10% chance that at some time your wealth will drop to only 10% of what you started with. For an individual investor, it is perfectly OK to be uncomfortable with this level of medium-term risk and to be less aggressive (in investment jargon) by using a partial Kelly strategy, that is using some smaller value of \[ \mbox{$p = $ proportion of your assets invested in stocks} \] than given by the Kelly criterion. Theory predicts you will thereby get slower long-term growth but with less short-term volatility. This leads to a memorable quote:
The Kelly strategy marks the boundary between aggressive and insane investing.The historical trade-off between long-term growth and volatility over the last 50 years is illustrated by this graphic. The insane label above is not mere hyperbole, because being more aggressive (in the specific sense here of allocating more money to risky assets) will by definition make your growth rate decrease, and indeed will at some point turn negative. This is in fact one of the more interesting paradoxes in probability theory -- here greed is not good!
The Kelly criterion provides a different perspective on this issue. The historic "140% invested" optimal strategy, and the fact that downward fluctuations of the index have historically been substantially less than theory asserts for a Kelly strategy, imply that index fluctuations have been smaller than in a Kelly-optimal portfolio. What this implies is debatable, of course. The simplest explanation would be that investors are not sufficiently long-term oriented. Another perspective is that participation in the stock market implies that investors have two beliefs: not only the “obvious” belief that long-tern stock returns will be greater than for risk-free alternatives, but also the less obvious belief that it is intrinsically risky. Because if they didn’t believe the latter, then they would bid up prices until stocks gave the same return. So the “climb a wall of worry” cliche has a converse: when stocks are not perceived as risky then they are likely over-valued.
In the simplest random walk/Brownian motion model for short-term behavior ignoring "Black Swans", the Kelly criterion is equivalent to the more classical "mean-variance optimal portfolio" given by WolframAlpha. But conceptually Kelly strikes me as much preferable -- thinking about what might happen over the next year forces you to think about what has happened in the past. Basing longer term investment decisions on random walk theory rather than empirical past data exemplifies a triumph of hope over experience (to steal a phrase).
I will not debate the efficient market hypothesis, but let me quote one of my heroes, John Bogle (letter to the Economist, November 2013):
Sometimes markets are efficient, sometimes they are not, and it is not possible to know which is which.