False factors: Be aware

Investment data is now easier to access and the tools used to analyse it have progressed dramatically. This, combined with the growing financial incentives behind factor investing, means that investors are increasingly presented with results that display impressive in-sample performance.

However, investors should develop a healthy degree of scepticism and be aware that without careful statistical adjustments, appropriate cross-validation and the imposition of realistic investment constraints many of these results may just be too good to be true.

Xavier Gerard, senior research analyst, recently presented Invesco Quantitative Strategies’ (IQS) stand on the issue at the latest Invesco Asia Investment Lecture Series on Factor Investing. The IQS team’s concerns echo those from prominent academics in the field such as Campbell Harvey and others.¹  

Here are some ways in which the team thinks that they can help investors spot poor data analysis:
 

• Question multiple testing: With more access to investment data, better analytical tools, and financial interests driving the need for breakthroughs in factor-investing research, the problem of data mining is becoming more acute today. As a result, countless factors are being tested, with many false factors that appear to work getting undue attention. It is therefore more important than ever for investors to scrutinize the basis for each factor. They should also look for a healthy and robust research environment, question what is it that incentivized the analysts to produce the research and keep track of how many factors are being tested.
   
• Be sceptical of over-complexity: This is because slight tweaks to a model can alter results. To avoid this trap, investors should scrutinize the rationale more and ask how robust the results are to different signal constructions.
   
• Don’t be fooled by overfitting: With multiple testing, this is perhaps the biggest problem facing investors today. In particular, investors should look out for combinations of vaguely-related and at best barely significant factors that are combined together into what looks like a strong predictor of future returns. The open secret about these models is that the weights on the factors are often chosen with some hindsight bias. Investors should avert this by scrutinizing the rationale for bundling weak signals together, ask for the full set of considered factors, question why plausible candidates are omitted, demand to see the individual statistical significance of each factor in the model and that the significance of the combined factors is tested under a multivariate t-test (e.g. Hotelling’s T-squared).
   
• Beware the limits of out-sample testing: Though out-sample validation is the easiest way to rule out spurious factors, a true out-sample test is hard to find. Experienced researchers know too well how the historical environment impacts the performance of their factors. Moreover, a typical problem is when a factor is researched in a country with a view of testing its robustness in a separate market, assuming that this provides a valid out-sample. Unfortunately, there are often significant cross-country correlations in factor performance. To cut down the chances that a false factor data-mined in-sample is validated out-sample, investors should ask whether factor dynamics are different enough between markets so that there is more information on the signal’s predictive power.² 
   
• Research environment vs. reality: An additional difficulty that investors must overcome in order to form realistic expectations from factor investing is the use of often unrealistic implementation constraints. Investors should keep an eye out for good results from research that does not take seriously into account liquidity restrictions, transaction costs or turnover constraints.
   

To conclude, the IQS team argues that factor research has significantly improved our understanding of how the market works and that factor investing constitutes an important pillar of modern investing. But they warn that the growing financial incentives behind factor investing are such that investors should display a healthy degree of scepticism when assessing research results. The team’s view is that factor investing is here to stay, but it must be done in the right way and investors must form realistic expectations.

Xavier Gerard is Senior Research Analyst at Invesco Quantitative Strategies.

^{^1 Arnott, Robert D. and Harvey, Campbell R. and Markowitz, Harry “A Backtesting Protocol in the Era of Machine Learning” The Journal of Financial Data Science, vol. 1, no. 1 (Winter): 64-74.}  

^{^2 See Xiaomeng Lu, Robert F. Stambaugh, and Yu Yuan “Anomalies Abroad: Beyond Data Mining” NBER Working Paper No. 23809 (September 2017) available at https://www.nber.org/papers/w23809.pdf}
 

Factor Investing Invesco