Avoiding biases and handling delisting

Back tested performance should always be taken with a pinch of salt as they could easily inflate performance if not accounted for biases.

For a back test to be a reliable indicator of historical performance it should be free of biases. Note that using back test as a reliable indicator of future performance is way more dangerous. But a back test that is free of biases is a good starting point.

In this article, we discuss some of these biases and how we avoid them.

Avoiding look ahead bias

We have already given a flavour of how look-ahead bias can creep into your back test. Suppose you are back testing a screen that has a filter based on EOD closing prices (say P/E). If your back test also transacts on the same day, it induces a look-ahead bias.

Even more glaring look-ahead bias can creep in when dealing with financial metrics. Remember, companies report every quarter but the reports are released with a lag. Therefore, financial statements for that quarter or that financial year are not available on the day that quarter/year ends.

Let’s take a concrete example. Suppose I am back testing a screen where I pick stocks with the highest yoy EPS growth in the last quarter. On a particular rebalancing date (say 31st March 2020), I cannot pick stocks that delivered the highest EPS growth in March 2020 quarter because that information will only be available in April and May 2020. If your back test does not account for this, you will induce a huge look-ahead bias and hence the results won’t be reliable.

The same can be the case for Mutual Funds. Portfolio holdings are reported monthly, but with a lag. The same needs to be accounted for when conducting a back test on mutual funds.

We have carefully mapped our historical data to avoid look-ahead biases. The historical database used for back testing is indexed in such a way that “information” is indexed to a date only when that information is available. For example, RELIANCE EPS of March 2022 is not indexed to 31st March 2022 but to 6th May 2022. RELIANCE reported for March 2022 quarter on 6th May 2022 and hence the information was available only on that date.

Avoiding survivorship bias

Yet another bias that can creep in your back test is survivorship bias. When back testing any dynamic strategy like screen, signal or dynamic baskets (where instruments are selected based on some rule), survivorship biases can overstate your back tested result.

Survivorship bias occurs when we only include the stocks (or ETFs/MFs) that have survived until the end of the back testing period while ignoring those that have been delisted or bankrupted during that time.

In order to deal with survivorship bias, we maintain a comprehensive historical database of all instruments and not just the ones active today. In our back test, during rebalancing, we allow the strategy to pick instruments that existed on that particular rebalancing date. It is quite possible that instrument picked on that date subsequently got delisted (in case of stocks) which needs to be handled (we discuss this next).

Although we have tried to eliminate survivorship bias as much as possible, we do accept that it is not entirely eliminated because of lack of high quality data for non-surviving stocks/ETFs/MFs. Needless to say, we are actively working to make this more robust.

Note that survivorship bias will not be an issue for static baskets where instruments are fixed and do not change with time. The very construct of this strategy makes it work only with current survivors. We discuss static baskets in detail here and back testing static baskets here.

Handling delisting/suspension

Since our back test includes stocks that have been delisted over time, it could run into a scenario where a stock picked on a particular rebalancing date gets delisted during the period till the portfolio is rebalanced again.

For starters, we do not pick stocks that are proposed to be delisted as on the rebalancing date. This is the reason why we have “exclusion list” filter in our screener and signal. Note that this does not amount to “looking ahead” since, generally, we have the information about future delisting.

However, we could still run into scenarios where a stock gets picked and during the period till its next rebalancing, it gets delisted or suspended.

In this case, we assume that the stock was sold at the last available price on the exchange available cash and will be used in the next rebalancing. Again, this is not “looking ahead” since there is a prior notice before delisting/suspension which allows investors to exit the stock.

This is not a perfect implementation. But given the data constraint that we have, perhaps the best one.