JACOB ROBINSONOpen.
← Blog ArchiveARTICLE 252 / 265

Blog Archive / JACOB ROBINSON

Converting a $100k+ Trading Algorithm from Excel to Python

When I was young, I made the mistake of getting a degree from a business school instead of an engineering school. It did serve to make my academic life very easy, but part of that was because we didn’t learn anything that instructors deemed “too difficult for a business student”.

So, imagine my surprise when I got into the world of professional trading, and found out everyone was using Python and not Microsoft Excel.

I had to get good. And fast.

The Lies of Business School

It still sort of irritates me that I went to college for Finance/Economics and not Computer Science. Not that I don’t feel like “a real programmer” – most good programmers I know didn’t even go to college in the first place – but rather that I feel business school genuinely hindered my learning of how the real world worked, rather than helped it.

I have a lot of examples and stories of this, but the focal point of this article is one that is rather small and petty. You see, the Finance department did have a mandatory data analysis class. It was a single semester, and spent the entire time talking to us about Microsoft Excel. No mention of any programming – not R, not Python, not even SQL of all things. They phrased the entire class as if all people ever did in these analysis jobs was use Excel: add data to Excel, crunch data in Excel, make visualizations in Excel. Hell, now that I’m remembering it, I don’t even think we learned DataViz software like Tableau. It’s fucking Tableau, for christsakes! It’s not rocket science!

So, obviously, with my teachers telling me all this, I assumed the only data I’d ever be working with was in Excel. Now, don’t get me wrong – there’s plenty of jobs that don’t deal in datasets large enough to use anything but Excel, and honestly I envy them (If you’re hiring for one of these jobs, let me know. I’d love to use what I actually learned in school). But most real jobs either use massive datasets requiring knowledge of query language, or utilize APIs to get their data.

This, as it turned out, included finance jobs.

My first work requiring real-world finance knowledge was still at the school, but as an analyst on the endowment’s investment team. Obviously these guys dealt with real money – large amount of real money – and so they weren’t fucking around. After getting the job, I learned that this role – despite being heavily marketed towards finance students – usually only hired computer science majors since it required knowledge of derivative mathematics [1] and the ability to use the APIs of Bloomberg and CapIQ [2]. The only reason I had gotten the job was because I had heard of R and Python and had the sense to add them to my application alongside every Python data package I knew. I didn’t actually know how to use any of them.

You’d think that during this job I would have learned to use these skills and become a better, more well-rounded person. Now to my credit, I did try to do this – I was the only person in the entire class [3] who read all four recommended textbooks in full – but it all went way over my head. I ended up just letting the CS guys do all the work while I nodded in the background and pretended to know a single thing they were saying.

So at this point you’re probably wondering what the title was all about. Obviously the $100k algorithm isn’t the endowment, because it was all written in Python in the first place and there was no need to convert it. Instead, this algorithm came from a poor bloke who was trying his hand at professional trading and quickly learned why all these professionals were using Python to begin with…

That’s right: the $100k+ trading algorithm was my own personal funds.

Why Excel Worked (Up to a Point)

Skip a few years later: I got my first job out of college (in tech, after I learned business/finance were all around terrible) and got my first $100k. Given my lifelong interest in trading, I only put half of it in a SPY IRA and the other half I wanted to use for testing algorithms. So I guess it’s more of a $50k algorithm, but that doesn’t affect what we’re talking about.

Since I never had any formal experience in Python, I built the entire thing in Excel – more specifically Google Sheets, where I could use the Google Finance API [4]. This meant I had to pre-screen my stocks on a different software (I chose Koyfin) and I also… well… couldn’t backtest. The lack of proper backtesting was a pretty big black hole – it was something cemented in my investment learning as essential, but I figured that with the amount of capital I had (and my care when it came to risk) I could just stick with well-known trade methodologies and be fine.

Besides all that, it worked! Google Finance would update the close/low/high listings, and my functions would do the rest. As long as I stuck to the same suite of 100-150 stocks (since I was trading ETFs this wasn’t a problem) the system would work well enough. But as I would end up finding out, “well enough” in trading still isn’t enough to make the difference.

The number one issue with the spreadsheet approach – and the reason I ended up finally moving on from it – was that it wasn’t practical to use historical data, something that pretty much every trading algorithm relies upon. Sure, you could use historical data if you wanted to log in every morning and spend four hours downloading CSVs from Yahoo and implementing them into your model. But I sure as hell didn’t.

It got to a point where my lack of being able to use historical data was getting pretty ridiculous. I knew that, ideally, I wanted both my portfolio weights and stops to be dynamic and automated by the system. On paper, this is a very simple operation to do. But if you were using the manual spreadsheet process, it’d require hours of tedious labor that most traders these days figured out how to automate by just learning Python.

In other words, it was time to learn Python.

The Python Conversion + Lessons Learned

Funnily enough, I think my trading algorithm is the only thing I’ve coded without any help from an LLM. That’s mostly because I assumed that my data package of choice – yfinance – was too obscure for an LLM to have accurate knowledge about. I later found this is the standard for pretty much any hobbyist trader and ChatGPT knows it inside out.

Before we got to the Python conversion proper, we needed to figure out what algorithm we wanted to make. For me, it went something like:

  1. Screen a subsegment of ETFs (same as before – this is still done outside the algorithm, but takes virtually no time to do)
  2. Find the Kelly weights of chosen ETFs (NEW – this requires historical data from your trading window (mine is 30 days). Without getting too technical, the Kelly criterion weighs your stocks by the historic frequency and magnitude of winnings. You can read this as: stocks with a long, rich history of making money get first dibs in your portfolio)
  3. Build dynamic trailing stops (NEW – you can set a generic stop (usually the previous day’s low) but it doesn’t account for stock volatility as well. Historical data can be used to fix this. To be fair this was a later addition and I did have ChatGPT help on the math for this)
  4. Export the whole thing as a CSV so that I can easily read it and submit the trades (standard practice – most brokerages even have a way to read CSVs for you, but I like the manual process here)

Now, like I said, this is pretty easy: we aren’t doing any advanced calculus or reinforcement learning to get our portfolio churning out results. Because of this, the only real packages I needed were yfinance (as mentioned previously), pandas (to mess with the data from yfinance), and numpy (a prerequisite for pandas). All I have to do is cycle through the ETFs, download 30 days of data (open/high/low, as it was in the spreadsheet) and do the calculations.

One challenge with programming is that, unlike spreadsheets, you don’t get immediate feedback. With Excel I can see the data right in front of me, so if I make a mistake or error it’s immediately apparent. With Python, you’ve gotta do things step by step: spend too much time crunching the numbers and your output might lead to nonsense with you having no idea what you did wrong.

Fortunately, Python users already solved this problem with something called Jupyter. Jupyter is a way of converting your code into snippets so you can playtest pieces of it over time. It’s considered a de-facto standard in the data analysis world, and while I had known about it I really didn’t come to appreciate how useful it was until this project.

While it took a few tries, I was eventually able to get the whole thing working. It takes in a CSV (the pre-screened ETFs) and then 1) calculates the Kelly weights, 2) adjusts it to the portfolio (given the weights and our cash, how many should we buy of each), 3) creates volatility-adjusted trailing stops for the chosen stocks, and 4) exports the output as a CSV with the ticker, the amount of shares to buy, and the stop.

Interested in the specifics of the code? I have good news! The entire algorithm is free and open-source, and you can look at it here. This is indeed the exact version I use for trading, and I update it whenever I make updates to the live algorithm.

Next stop… well, uh… oh dear, I never did do backtesting, did I? You can look forward to that one in a later article.

[Image omitted]

[1] – That is the mathematics behind financial derivatives, not derivatives in general. It is a bit more complicated, requiring knowledge of concepts like Brownian motion and Ito calculus. I did not know this (I still do not know this) but I pretended to throughout the entire job.

[2] – I’ve never come across this ever before or after getting this job, so I feel the need to explain: CapIQ is S&P’s competing software to Bloomberg. You can read this as “The university didn’t want to pay for Bloomberg and instead got the cheaper CapIQ but then all the endowment guys complained about it so they bought Bloomberg Terminals anyway and now we had subscriptions to both which was a total waste of money and resources but no one bothered to fix it”.

[3] – Like most jobs, once you got in you had a “training academy” that would teach you the basics behind what you were going to do in-role. I didn’t learn anything from this either.

[4] – Yes, obviously Microsoft Excel and Google Sheets aren’t the same app, but they might as well be. Take it as another simplification for the post title.

Select text in the article to highlight it.

Your saved highlights

    Article contents

    Blog Archive · 265 articles

    1. 01 Five Takeaways from #100DaysofCode
    2. 02 Getting Past the Bump
    3. 03 How Important are Principles?
    4. 04 How to Avoid Writer’s Fatigue
    5. 05 The Dangers of Catastrophizing
    6. 06 State of the Union 2019
    7. 07 The Rise of the Personal Brand
    8. 08 Finding Adventure in 2019
    9. 09 5 Ways to Use Automation to Vastly Increase Your Efficiency
    10. 10 What you Should be Doing Instead of Networking
    11. 11 Creatives vs. Technicals: Which Should You Focus On?
    12. 12 The Difference between Quitting and Giving Up
    13. 13 3 Ways To Get Value Out Of Nature
    14. 14 What is the Purpose of Dreams?
    15. 15 How to develop intellectual success in future generations
    16. 16 What the Definition of Luck Really Is
    17. 17 Who Makes the Culture?
    18. 18 Sexual Frustration and Trying to Find Love in the Age of the Internet
    19. 19 The Lazy Man’s Guide to More Efficient Research
    20. 20 How One Joke Can Ruin Your Life
    21. 21 Why IQ is BS
    22. 22 Why it’s Impossible for the Illuminati to Exist
    23. 23 Some of my Favorite Blogs
    24. 24 The Most Important Indicator on Why a Company Succeeds
    25. 25 Returning to Tesla
    26. 26 Living in a Post-Truth World
    27. 27 The Function of Success
    28. 28 Lessons at 20
    29. 29 If the Recession Happened This Year…
    30. 30 Why People Don’t Read
    31. 31 Social Media and Truth
    32. 32 Making Things 1% Better
    33. 33 What the Book of Job Really Means
    34. 34 Life as a Game
    35. 35 The Function of Opportunity
    36. 36 Exponential Growth
    37. 37 The Machine Learning Revolution
    38. 38 Practical Thoughts on Blockchain
    39. 39 Why the Humanities Are Just as Important as STEM
    40. 40 Dow 30,000
    41. 41 The Shame Economy
    42. 42 You Can’t Do It All Alone
    43. 43 Favorites of 2019
    44. 44 State of the Union 2020
    45. 45 The Tolerance Paradox
    46. 46 Null
    47. 47 Social Game Theory
    48. 48 3 Tips for Developing a Positive Mindset
    49. 49 Why Social Gaming is So Popular
    50. 50 A Design Study on YourStack
    51. 51 Relationships, Careers, and Investments
    52. 52 The Fall of GAFA
    53. 53 Wartime CEO
    54. 54 Mass Shooting Cycles
    55. 55 But Who’s Right?
    56. 56 What Matters in the End
    57. 57 Is AR/VR too “weird”?
    58. 58 Convenience + Efficiency
    59. 59 Vicious Cycles
    60. 60 With Smart People
    61. 61 The Third Moat
    62. 62 The Secret to Success
    63. 63 The End of Education
    64. 64 The Ben Carson Effect
    65. 65 Biological Evolution, Cultural Evolution
    66. 66 Suicide and the Self
    67. 67 The Life Cycle of a Business
    68. 68 How to Get into Harvard
    69. 69 Managing Your Personal KPIs
    70. 70 Fog of War
    71. 71 Make-or-Break vs. Nice-to-Have
    72. 72 Identity and Political Groups
    73. 73 Nationalism’s Pros and Cons
    74. 74 How to Win an Election
    75. 75 Why You Shouldn’t Network
    76. 76 On Relative Morality
    77. 77 Death in the Age of the Internet
    78. 78 Audience of One
    79. 79 The Rise of Anxiety
    80. 80 Stop Getting Scared By AI
    81. 81 If You Build It, They Will Come
    82. 82 Building Knowledge Advantage
    83. 83 The Quest to be Kinder
    84. 84 Everything You Do Sucks
    85. 85 Niche Monopolies
    86. 86 Up Shit Creek
    87. 87 Emotional Equivalent Salary
    88. 88 Uncertainty Principle
    89. 89 Building a Startup with Anxiety
    90. 90 Punishment and Forgiveness
    91. 91 Favorites of 2020
    92. 92 State of the Union 2021
    93. 93 Random Factor Social Suicide
    94. 94 Old World Business, New World Business
    95. 95 The Customer Takes Care of Itself
    96. 96 How to Use Reddit as a Business
    97. 97 Genetic Obsolescence
    98. 98 A Social Capital Theory of Management
    99. 99 If I Could Do My Education Over Again
    100. 100 Aggregation and Decoupling
    101. 101 The Crazies, Part II
    102. 102 Art as Inspiration, Art as Education
    103. 103 Free to Play Models for SaaS Services
    104. 104 A Recap on my 2020 Politics Posts
    105. 105 Making Mistakes, Post-Internet
    106. 106 Taste is Not Tech
    107. 107 Immortality
    108. 108 Biases Against Consistency
    109. 109 The Mythical Self
    110. 110 Faulty Mental Models
    111. 111 Honor and Glory
    112. 112 Momentum, and the Value of Cold Showers
    113. 113 The Prestige Myth: Providing Structure to Structureless Forms
    114. 114 The Unluckiest Man Alive
    115. 115 Those Who Make, And Those Who Work
    116. 116 Post-Truth, Revisited
    117. 117 Conscience Questions, Unchristianized Edition
    118. 118 On Being You
    119. 119 Effective versus Efficient
    120. 120 The Experimental Method of Learning
    121. 121 A New Model for Teaching Humanities
    122. 122 A Dialogue on Mindset
    123. 123 I Am Not Your Friend
    124. 124 My 5 Tips Towards Reading Books
    125. 125 Buying Immortality
    126. 126 Do this, but…
    127. 127 Leverage and Wealth: A Simplification
    128. 128 Mind’s Eye and Creative Application
    129. 129 Intersection of Technology and Art
    130. 130 Blog Writing for Robots
    131. 131 10 Tips on Writing
    132. 132 The Cost of Staying Alive in America
    133. 133 Taste on the Margin
    134. 134 Top 10 Tools I Use in 2021
    135. 135 Income Generators 101
    136. 136 Strategies for David, Versus Goliath
    137. 137 Progress in the Long Term
    138. 138 The End of Journalism
    139. 139 Myths of People
    140. 140 Heroism
    141. 141 External Mentors
    142. 142 Favorites of 2021
    143. 143 State of the Union 2022
    144. 144 The End of Gender
    145. 145 The Work After Work
    146. 146 There Is No Metaverse
    147. 147 On Dealing With Chaos
    148. 148 Risk/Return Bets in Entrepreneurship
    149. 149 The Weird, Magical World of Modern Day Quora
    150. 150 On Business
    151. 151 What’s “Hivemind”?
    152. 152 Life as a War
    153. 153 On Culture
    154. 154 My Approach to Investing
    155. 155 Stop Caring About Your Reputation
    156. 156 On Humility
    157. 157 Is Relative Morality Helpful?
    158. 158 Achievement-Based Thinking
    159. 159 On Success
    160. 160 Moral Implications in Maximums and Minimums
    161. 161 Sentience and Suffering
    162. 162 On Improving
    163. 163 The Suicide Disease
    164. 164 Social Diversification
    165. 165 On Meaning
    166. 166 Convenience and Pareto Optimality
    167. 167 Mental Illness is Not a Permanent State
    168. 168 Biased Science
    169. 169 People as Means
    170. 170 Being Wise and Being Smart
    171. 171 The New Working Class
    172. 172 Fitness, Suffering, Self
    173. 173 Political Theory and Evolution
    174. 174 Explaining my Rating System
    175. 175 Why You Might Want A Trading Account Over An IRA
    176. 176 Why NFTs Don’t Work For Games
    177. 177 A Few Thoughts on MMOs
    178. 178 A Review of MyMind
    179. 179 Read, Think, Make, Share, Monetize
    180. 180 Is Science Perfect?
    181. 181 A Guide to Fiction Writing
    182. 182 The Productivity Tool World Tour 2022
    183. 183 My Daily Routine
    184. 184 The Curse of The Conqueror
    185. 185 Some Brief Notes on Programming
    186. 186 My Life Principles
    187. 187 Cyberpunk in 2022
    188. 188 What Decides Good Art?
    189. 189 Trends in Mass Murder Techniques
    190. 190 Changing to Fit the Times
    191. 191 Why Did Photographers Embrace NFTs, but Artists Didn’t?
    192. 192 Favorites of 2022
    193. 193 State of the Union 2023
    194. 194 New Blog Post Schedule
    195. 195 Was Lex Fridman’s “Reading List” Controversy Caused by Mass Psychosis?
    196. 196 Is Notion “Too Good”?
    197. 197 Mastodon is Social Media PvP with Permadeath Enabled
    198. 198 Arc, Vivaldi, and the Next Generation of Browsers
    199. 199 Is the Four-Hour Workweek Actually Possible?
    200. 200 Do Not Obsess Over Your Customer. Obsess Over Your Community.
    201. 201 Why Big Entertainment Failed to Bring in Diversity
    202. 202 My All-Time Favorite Tools
    203. 203 Museums on the Computer
    204. 204 How to use AI to get you motivated to work
    205. 205 The Best Way to Focus: Do Nothing
    206. 206 Can Buddhism and Ego Intertwine?
    207. 207 How I Write My Blog Posts Now
    208. 208 Practice and Mysticism
    209. 209 Automate the Boring Stuff with ChatGPT
    210. 210 State of the Union 2024
    211. 211 The Secret to Great Products You Aren’t Doing
    212. 212 How to build a $5 billion dollar business in 5 minutes
    213. 213 How To Easily Automate Investments, Beat Big Tech, and more!
    214. 214 Can I Use AI To Solve The World’s Unsolvable Problems?
    215. 215 How A “Taste Matrix” Can Improve Your Art And Kill Writer’s Block
    216. 216 My Bizarre, Counterintuitive, and Lifechanging Approach to Work
    217. 217 Top 3 Recommended Browsers (They Aren’t What You Think)
    218. 218 How Hard Is It To Make Money On Steam?
    219. 219 What To Do If You’re Socially Anxious
    220. 220 My New New Way of Reading Books (I Abandoned AI)
    221. 221 References Are The Best Way To Learn Anything
    222. 222 How To Quit Planning Goals And Start Achieving Them
    223. 223 The Only Good Self-Help Book
    224. 224 Be Nice To Everyone. It Just Might Save Your Life.
    225. 225 I Learned An Ancient 3-Step Process And Became A Master Meditator
    226. 226 How Canva Became The Home of Everything
    227. 227 Has The Internet Made More People Mentally Ill?
    228. 228 The Biology Behind Why We Like Stories
    229. 229 This Tape Haunted An Entire Nation. But Was It Even Real?
    230. 230 We Are All Most Certainly Going To Make It
    231. 231 What Makes Good Taste?
    232. 232 In a World of Cancel Culture, Reputation Doesn’t Matter Anymore
    233. 233 Why You Hate Diversity In Media
    234. 234 I Used Readwise for 200 days. Here’s what happened.
    235. 235 Lessons on Bootstrapping Solo with Little Income
    236. 236 This Ancient Story Was Drowned In Lies. The Truth is Much More Interesting.
    237. 237 How To Get Actionable, Memorable Advice Out Of Anything You Read
    238. 238 You Are A Coward. And That’s Not So Bad, Really.
    239. 239 Time Travel is Real. Here’s How to Use It.
    240. 240 How I Got Into Old Games
    241. 241 Is Every Story Political?
    242. 242 Your Culture Decides Your Fate
    243. 243 In Defense of the Jumpscare
    244. 244 Final Fantasy XIV’s Story Is A Mess
    245. 245 Markov Chains and You!
    246. 246 I used Obsidian for 90 days. Here’s what happened.
    247. 247 Ideas That Can Kill You
    248. 248 Porting A 250k+ Item Database from Notion to Obsidian
    249. 249 Why the Fast Inverse Square Root Matters
    250. 250 A Deep Dive on Networking
    251. 251 A Deep Dive on Cybersecurity
    252. 252 Converting a $100k+ Trading Algorithm from Excel to Python
    253. 253 How To Make A (Super) Hard Drive From Scratch
    254. 254 So, How Do Computer Graphics Work?
    255. 255 The Quantum Physics Rabbit Hole, Part I
    256. 256 The Quantum Physics Rabbit Hole, Part II
    257. 257 The Story (and Meaning) of Humanity’s Greatest Achievement
    258. 258 Explaining Tonal Indicators
    259. 259 AI should have you thinking MORE, not less
    260. 260 Forbidden Knowledge: The Vela Incident
    261. 261 All success in life comes back to pain tolerance
    262. 262 The AI Revolution rewards the curious
    263. 263 Forbidden Knowledge: Marco Evaristti
    264. 264 Forbidden Knowledge: Casket Failure
    265. 265 Forbidden Knowledge: Coriander Theory

    Share

    Make it your own before sharing.

    Page link

    Send to

    EmailWhatsAppBlueskyFacebookX

    Facebook shares the link only; add your message there.