Arguments against systematic back-testing [a blog follower’s email]

A few days ago, over a weekend morning coffee, a fellow retail/discretionary trader and I discussed some aspects of my recent testing work.  He proceeded to sent me a detailed email explaining his view that testing in this manner is unlikely to yield useful insights.

He brought up some excellent points – and subsequently also agreed that I post a copy of his email for all readers to see:

Hi George,  just some comments on your blog and our coffee on Saturday:

  • So I don’t personally believe straightforward very systematic trading rules work well for independent traders. Major algorithmic traders  have the resources to try out thousands of different quantifiable tactics against many markets, and select the best. This worked very well some years ago, but works less well now that everyone is doing it. Also it may require very short timescales to be profitable, or constant updating of trading strategy to suit market regimes(and this selection process may be guided by the expert judgement of experienced traders). Bigger companies with larger accounts can survive on 10% return, but I don’t think strict rules based trading leads to returns 25% + pa over an extended time period. But it depends what the rules are   – maybe complex AI rules could work well, but a rule easy to program in Excel will I think be too simple to be consistently effective.
  • Your resources are very limited when it comes to back-testing, an algo company may have terabytes of data and AI systems to search for any small edge.
  • Non-stationarity – the markets are evolving all the time, what worked  2 or 5 or 10 years ago may not work now as the markets have changed and will keep changing, so I question how valuable back testing is when it comes to a trading plan.
  • I do see the value of back-testing to give certain probability insights  – e.g. if ATR is 50% lower than average on 80% of Fridays I won’t expect  a big move on a Friday, if  1% move on DAX is reversed the same day 80%  of the time I would not expect a 2% move after a 1% move.[not real  data]. FT71 uses this approach. But these probability insights feed into a discretionary system rather than form trading plan.
  • You link to Lance Beggs – interesting content by the way (his books look helpful) – and he calls himself a ‘technical, discretionary trader’  – I suspect many successful private traders are also discretionary to some extent relying on expert judgement  / trained intuition in addition to rules. And my thought is that it is developing expert judgement using deliberate practice techniques that creates the results. So interesting to check the methods used by successful independent traders.

Just a few counter – hypotheses to throw into the mix during your research project!


This entry was posted in Testing. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.