An example output from my backtesting program, animated over time
Objective
To develop a trading algorithm that outperforms the S&P500 for use in passive wealth growth.
Custom-built Backtester
Overview
The best available method for measuring the success of a trading strategy is to 'backtest' it on historical data. I originally did my backtesting online with Quantopian and QuantConnect, but decided to break away from them due to their limited logging and bans on data export (due to licensing contracts).
Instead, I developed my own backtesting environment in Python from scratch and began scraping data from brokerage APIs.
Algorithm Architecture
An information flow diagram is provided below outlining the design of the backtester. The key design principle here is segmentation. By granting each element of the backtester only the information it needs and standardizing what data flows between them, each element is entirely interchangable on command. This allows for quick and efficient experimentation of different ideas.
The information flow diagram for my stock algorithm, whereby all data from all stocks in the market are
reduced down to a couple stocks to be bought or sold on a given day.
Filter: Defines a universe of stocks as a subset of all stocks by a simple metric, such as top 100 market cap, and sends the subset to the SignalProcessor.
SignalProcessor: Uses more complicated metrics to identify potential buy or sell signals, and sends the subset to the Strategy.
Strategy: Takes all the signals and decides which ones to buy or sell, and sends orders to the Brokerage.
These three elements comprising an algorithm are then plugged into the backtester to analyze the algorithm's performance
Backtester Architecture
With algorithms completely interchangable by changing a single line of code, this system is robust and ready to get testing right away.
The information flow diagram for the backtester, designed for efficiency and segmentation
DatabaseManager: Queries the database for the data necessary for testing the current day.
StockAlgorithm: Shown in the diagram above, comprised of a Filter, SignalProcessor, and Strategy.
Portfolio: Records the number of shares and purchase price as determined by the brokerage. Measures success of algorithm.
Brokerage: Receives orders and models the price at which the stocks are purchased or sold. Minute data can be used to model slipping prices, or a no-slip assumption can be made.
Data Architecture
My stock market data is saved partially in an SQLite Database (for datasets too large to hold in RAM) and partially in Pandas dataframes saved as pickles (for datasets < ~1GB). This data storage setup, combined with efficiently written and heavily multi-threaded code, allows for backtesting across ten years of market data in about 1-2 minutes on a laptop. I can test ideas faster than I can generate them... a wonderful issue to have for a project with so much potential.
The basic data features include open, high, low, close, volume, dividends, split_factor, market cap, market evaluation, p/e ratio, and p/b ratio. From these, many other features can be calculated such as moving averages, volume weighted price, resistance, momentum, etc.
Data Sources
I get my data from three sources. First, I've written a scraper that sends API calls to TDAmeritrade's API a saves data to my database. This is to acquire recent data. To acquire historical data from before I was scraping my own, I pay for access to Tiingo for end-of-day data and fundamentals and Polygon for intra-day data.
Disclaimer I am not sponsored by or affiliated with either company. I just like their services at their price point, and would recommend them.
An Example
Obviously, I won't be sharing any details on the specific strategy I am developing because increasing volume traded through a strategy will eventually change your supply and demand curves enough that the strategy no longer works. But here's an example of using my backtester to test a very simple algorithm.
The Hypothesis
Let's say my hypothesis is "If AAPL increased in price yesterday, then it will increase in price again today". To test this, I could design a strategy that buys AAPL at market open if yesterdays returns were positive, and sells everything I own at the end of each day. (This is obviously a terrible hypothesis, used for illustrative purposes only.)
The Code
We'll just use a pass-through filter for this since we won't be running any computations to decide which stocks to buy. The signal processor will simply signal AAPL each morning where the previous returns were positive:
The strategy will buy all signals in the morning, if any, and sell all open positions at market close:
Results
Plug the SignalProcessor and Strategy into an algorithm and you're good to test it, revealing pretty poor results for the period from 2019-01-01 to 2020-10-31. You'd be better off just throwing your money in a mutual fund than using an algorithm like this.
Conclusions
Financial freedom comes from passively generated income and capital growth. I started the very very beginning of this project in September 2019, fueled by a desire for financial freedom. This backtesting architecture above is the refined product of many iterations and lots of deprecated code, but the result is a setup where I can test ideas faster than I can generate them.
As always, I record all my research and findings in journals for easy access to my collection of ideas. Most of those ideas are failures, but those many failures are just as valuable as the few successes.