Professors Nick Taylor and Ian Tonks, of the University of Bristol Business School, formed part of a major international collaborative project assessing the variations in results across research teams in providing answers to typical finance research questions. The paper, Non-standard errors, features 250 joint authors, and explores the difference in results found by researchers when independently testing the same hypothesis on the same data sample. This is important for research in financial economics because variability in data cleaning and choice of an estimation method adopted by a research team is not usually allowed for in establishing the significance of empirical research.
The experimental project design required 164 research teams and 34 peer-assessors to evaluate the same set of market microstructure hypotheses examining 720 million trades in EuroStoxx 50 index futures from 2002 to 2018. Examples include whether market efficiency had increased or decreased over time; what were the estimates of the realised bid-ask spreads charged by financial intermediaries when trading; and whether patterns in measured trading volumes had changed over time.
The project started with the view that using more research teams to test the same hypotheses on the same data should reduce the standard errors. Usually, standard errors in empirical research relate to variability in estimation due to the data sample being representative of the underlying population. However, potential sources of non-standard errors include the research decisions made in the evidence-generating process, for example the model, method and execution used by the research team. This is distinct from the data-generating process that is responsible for the sample being different from the population.
The project identified the magnitude of these non-standard errors and went on to examine what could be done to reduce them. The paper reports that non-standard errors were related to research team quality (measured by previous publication history). These errors were reduced after additional stages of the research process taking into account the peer-review process. For example, the research teams received feedback from peer reviewers who will have identified anomalous procedures in the evidence gathering process, which the researcher could then amend.
The published paper documents the experimental setting, explains the methodology and concludes that non-standard errors add uncertainty to the outcome of hypothesis testing. The results suggest these non-standard errors are the same order of magnitude as standard ‘standard errors’; they co-vary with team quality and paper quality; they decline significantly after peer-feedback; and they are typically underestimated by research teams.
Professor Ian Tonks said, ‘The results from this research project illustrate some of the pitfalls of empirical research in finance and emphasise the importance of peer-review in the publication process for academic research.’
Professor Nick Taylor comments, “Reliable empirical research in finance is not for beginners! High quality research relies on up-to-date quantitative techniques and an understanding of the relevant theoretical background.”
The work is forthcoming in the Journal of Finance, the world’s leading outlet for publications in financial markets.
Find out more about the University of Bristol’s research in Financial Markets.
Menkveld, Albert J., Dreber, A., et al. (2023) “Non-standard errors”, forthcoming Journal of Finance. (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3961574)