20170407 Shiny: day trading forex
April 7, 2017•650 words
Watching my artificial creatures watch
Rainbows of waves dancing on the grid
Today's events condensed into numbers
Twitching and writhing to invisible beats
Each measure different than the last
Window sliding into the future: inexorable
All observed opportunities: irretrievable
Dogma and ritual abound, causality abandoned
Reduction of pattern arbitrary unto madness
Solutions spread and fill the sky
Precipitation floods the arrays
Final selection? Position available now
Because there will always be rainbows
2004
"If you generate more than one reason to do something, you're trying to convince yourself; this is a strong sign to wait or even reject the plan; that said, if you have a single excellent reason, go for it and don't be afraid of failure." —Nassim Nicholas Taleb
"With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." —John von Neumann
After the dot bomb in 2001 I took some time off and considered financial instrument trading for a living. There isn't an engineer in the world who hasn't thought at one time or another, "I can beat the market."
Having a long interest in evolutionary computation, I did some research and used it to study the spot currency (FOREX) market. I implemented parallel-hybrid genetic algorithms based on concepts from the literature: steady state, variable length real-value genome, roulette-wheel and genetic variance selection for breeding and culling, 2-parent propagation with modified uniform crossover, local hill-climbing via parthenogenesis, self-adaptive mutation rate, etc. This was an unnecessarily complicated way of having fun, but the results were beautiful and occasionally astonishing.
My techniques and creatures improved in fits and starts, eventually generating models that back-tested well and sometimes did very well in live trading. But sometimes they didn't. The colorful charts were hypnotizing, the unpredictable success tantalizing, and I was naïve to the addictive power of variable rewards. My improvised server farm grew until PG&E bills were ~$500/mo and I worried it would trigger a DEA raid for a different kind of farm. I continued trading and almost broke even. For months I endured paging at all hours to sanity-check and execute trades, because I couldn't find a solid reason for the erratic performance.
A fortunate accident revealed the cause: models with too many degrees of freedom overfitting the data. Unlike stock exchanges generating public "ticks" of exact bid/ask prices for every instrument at a precise frequency, there is no coordinated, authoritative source of FOREX quotes. Multiple entities each generate their own stream without official cadence—really just the exhaust trail from big private trades; the data are slightly different depending on query timing and source. Apart from rare blips—explained or not—the streams appear identical when plotted on an overview graph; however, they are subtly different. When I mistakenly fed my creatures a secondary source for routine back-testing, they performed completely differently—revealing the over "optimization." Mystery solved! Had I been testing and trading on a stock exchange, who knows how long it would be until I realized this or just rage quit in frustration?
Welp. Lessons Learned:
Evolutionary computation provides efficient methods to generate overfit models
In fact ECs are suitable for generating specific solutions, not models (remedial statistics!)
Nonlinear systems are hard to model dynamically by any means (doy)
The only ways to consistently profit in the markets are to be a broker, exploit insider information, get fresher data (a special case of insider information), shift risk to someone else, or be big enough or clever enough to manipulate the market. Oh, wait, these are all ways of shifting risk—ranging from somewhat to extremely shady, and incredibly to heart-stoppingly expensive to deploy. Arguably, anyone besides a broker or cheater who outperforms a monkey is still standing only by limited grant of survivorship bias.
It is, seriously, totally worth exploring, questioning, experimenting, and learning things the hard way. I highly recommend it.
2017 Apr 7