Quant / Algo trading

rojito · 08-10-2020 10:20am #1

Howdi,

Is anyone here into quantitative or algorithmic trading? I have been playing with Quantopian for backtesting for about 18 months now, and created an Interactive Brokers account to use their API for live trading. I got more serious about it during the corona lockdown and I went live with 2 algos in June when the SPY 10 day SMA crossed over the 200 day SMA.

The markets have obviously been volatile since then, and I had a few coding errors that I didn't catch in testing, but I now have my algos where I think they should be and I'm a small bit in the black. I don't have a life changing sum of money invested ATM, but I'm thinking if I can ride out this current storm without losing my shirt it'll be a good lesson and stand me in good stead in the future.

I have a strong development background, but not as much trading experience. I have found that the forum users on Quantopian tend to be the opposite of that and I have a decent amount of success from taking their shared algos and improving them.

What I would really like at this point is some insight into what the pros use, or how to take my algos to the next level. I have been looking for books, but so many of them have negative reviews or complaints that the gains they claim aren't achievable, anyone here with any background in this have any tips?

Jim2007 · 08-10-2020 11:02am

What I would really like at this point is some insight into what the pros use, or how to take my algos to the next level. I have been looking for books, but so many of them have negative reviews or complaints that the gains they claim aren't achievable, anyone here with any background in this have any tips?

The reviews are correct! All types of math based approaches are doomed to failure because they fail to model the human factor. The most dangerous feature is that they all appear to work for short periods of time.

The best approach is to sell your software to the punters that think it works. From experience I can say that works very well

. It is not terribly difficult to find a 1000 punters willing to pay say a $150 a year.

rojito · 08-10-2020 12:05pm

Thanks Jim,

Yeah, I'd love to get there some day, but at the moment I feel I need to continue educating myself and building up a track record before I have something at least demonstrable that I can actually sell.

In terms of trying to build a reputation, I have gone on Upwork looking for a couple of contracts and there are quite a few people looking to go this route (Quantopian backtesting -> IB live trading), but there doesn't seem to be a lot of people out there with the skills I feel I have picked up. My problem is that I have a decently paid day job I'm not willing to give up yet so I'm not going to do contract work cheaply if I would have to give up my free time. I would be competing with contractors from developing countries who might not have the skills but are willing to work fulltime on the project for $10 an hour. My wife wouldn't be happy if I couldn't help out with the kids over the weekend if I was working a $10 an hour contract. In saying that I should go back in for another look around - I am sure there are people who value quality and experience if I look hard enough.

I do backtest my algorithms back to 1/1/2005, so in that sense I feel they are somewhat robust and future proof - even if there are somewhat conservative and not excessively elaborate. I guess that's a decent compromise instead of eking out short-term stellar performance at the expense of longevity of the algorithm.

Mach 3 · 08-10-2020 1:42pm

rojito wrote: »

Thanks Jim,

Yeah, I'd love to get there some day, but at the moment I feel I need to continue educating myself and building up a track recordbefore I have something at least demonstrable that I can actually sell.

In terms of trying to build a reputation, I have gone on Upwork looking for a couple of contracts and there are quite a few people looking to go this route (Quantopian backtesting -> IB live trading), but there doesn't seem to be a lot of people out there with the skills I feel I have picked up. My problem is that I have a decently paid day job I'm not willing to give up yet so I'm not going to do contract work cheaply if I would have to give up my free time. I would be competing with contractors from developing countries who might not have the skills but are willing to work fulltime on the project for $10 an hour. My wife wouldn't be happy if I couldn't help out with the kids over the weekend if I was working a $10 an hour contract. In saying that I should go back in for another look around - I am sure there are people who value quality and experience if I look hard enough.

I do backtest my algorithms back to 1/1/2005, so in that sense I feel they are somewhat robust and future proof - even if there are somewhat conservative and not excessively elaborate. I guess that's a decent compromise instead of eking out short-term stellar performance at the expense of longevity of the algorithm.

If it is working, you dont sell it! You sell it when it stops working but can demonstrate how it did work and show the P and L.:cool:

What was the biggest monthly drawdown?
How long was the biggest drawdown?

Algo or not some people can't handle the drawdowns, even in a high positive expectancy.

Best of luck with it.

Jimbobjoeyman · 08-10-2020 1:44pm

Jim2007 wrote: »

The reviews are correct! All types of math based approaches are doomed to failure because they fail to model the human factor. The most dangerous feature is that they all appear to work for short periods of time.

The best approach is to sell your software to the punters that think it works. From experience I can say that works very well . It is not terribly difficult to find a 1000 punters willing to pay say a $150 a year.

I always find the people that buy these to be very interesting, especially the forex ones.

A few minutes of research will tell you that there is mathematically zero info in forex pairs pricing as to where they will be in the future.
But people seem to love forex bots that trade on charts

Timmaay · 08-10-2020 2:21pm

rojito wrote: »

Thanks Jim,

Yeah, I'd love to get there some day, but at the moment I feel I need to continue educating myself and building up a track record before I have something at least demonstrable that I can actually sell.

In terms of trying to build a reputation, I have gone on Upwork looking for a couple of contracts and there are quite a few people looking to go this route (Quantopian backtesting -> IB live trading), but there doesn't seem to be a lot of people out there with the skills I feel I have picked up. My problem is that I have a decently paid day job I'm not willing to give up yet so I'm not going to do contract work cheaply if I would have to give up my free time. I would be competing with contractors from developing countries who might not have the skills but are willing to work fulltime on the project for $10 an hour. My wife wouldn't be happy if I couldn't help out with the kids over the weekend if I was working a $10 an hour contract. In saying that I should go back in for another look around - I am sure there are people who value quality and experience if I look hard enough.

I do backtest my algorithms back to 1/1/2005, so in that sense I feel they are somewhat robust and future proof - even if there are somewhat conservative and not excessively elaborate. I guess that's a decent compromise instead of eking out short-term stellar performance at the expense of longevity of the algorithm.

I think Jims sarcasm went over your head there...

Anyways I messed around with an interesting free app called Proquant for a while there, copying several experienced traders patterns, traders who had spent hours doing loads of backtesting etc, and all I can say about the whole experience is its a great way to slowly bleed off all of your capital. I gave that app 2months, I started out with 2k in it and pulled the plug at 1500e. Most of the time the time the trading patterns I was copying had good back test results, and would be steadily making money, then one moderate market shock would result in a big loss that would more than wipe out that previous wks profits. As Jim said your trying to put an algorithm on an insanely large and complex set of human emotions, stick to the dayjob is my vote, dollar average across time if you want to make money off the stockmarket.

rojito · 09-10-2020 11:04am

Mach 3 wrote: »

If it is working, you dont sell it! You sell it when it stops working but can demonstrate how it did work and show the P and L.:cool:

What was the biggest monthly drawdown?
How long was the biggest drawdown?

Algo or not some people can't handle the drawdowns, even in a high positive expectancy.

Best of luck with it.

Yeah, I'm finding the psychological aspect of watching this program buy and sell interesting, but I've entered into this with only a relatively small amount of capital, and it is money I can afford to lose. At the moment I am happy to trust the program.

Hard to measure monthly drawdown in Quantopian but the biggest overall was 40% between July and November 2008. But I reckon everyone lost money in that period! It had bounced right back by Feb 2009 however. I will work on trying to reduce this, but given the amount of money in play I am happy to take on higher risk. I currently only open max 5 positions for example, high risk but potential high reward. With more funds I would probably increase this number to mitigate risk.

I'm under no illusions here, and am treating it as a hobby. I forced myself to write everything in Python which was not my strongest programming language but something I was keen to improve. I'm also gaining knowledge in the an area I have a personal interest in, and am gaining a little bit of a foothold in something that could possibly be an option for future employment. For the money I could potentially lose there are a lot easier ways to spend it without gaining any benefits. As it happens though I am up a little so all good.

rojito · 09-10-2020 11:18am

Timmaay wrote: »

I think Jims sarcasm went over your head there...

Maybe it did haha. I've been looking at it and still can't find the "sarcasm". He made a comment in jest along the lines of the old adage "better to sell books/software/services to traders than trade oneself". I responded in agreement but felt I still needed to build up knowledge, experience and a track record.

Timmaay wrote: »

Anyways I messed around with an interesting free app called Proquant for a while there, copying several experienced traders patterns, traders who had spent hours doing loads of backtesting etc, and all I can say about the whole experience is its a great way to slowly bleed off all of your capital. I gave that app 2months, I started out with 2k in it and pulled the plug at 1500e. Most of the time the time the trading patterns I was copying had good back test results, and would be steadily making money, then one moderate market shock would result in a big loss that would more than wipe out that previous wks profits.

Ouch sounds like you got stung there, but there were plenty of red flags. I don't know if any decent traders would put their algos on a free app. It also sounds like maybe you didn't have the risk appetite to match the algorithm you invested in. I'm assuming you had no visibility of the actual algorithm. I also wouldn't invest in anyone else based on backtesting results of a blackbox algo, they would need to have a record of actual live trading. It could have just been bad luck or bad timing, but I'm not sure it was either.

I guess you're still feeling the hurt, but I am wondering why you would try to discourage someone from learning about something (as was the original intent of this thread) because you previously lost money in the same area... seemingly due to failing to learn enough before investing?

Timmaay wrote: »

As Jim said your trying to put an algorithm on an insanely large and complex set of human emotions, stick to the dayjob is my vote, dollar average across time if you want to make money off the stockmarket.

Currently, I am well able to manage the dayjob and this pet project on the side! If it happens to work it will just be an alternative income stream until I make enough to retire! If it doesn't work then I am learning, testing myself and having some fun, so I am happy enough to continue.

Mach 3 · 09-10-2020 1:59pm

rojito wrote: »

Yeah, I'm finding the psychological aspect of watching this program buy and sell interesting, but I've entered into this with only a relatively small amount of capital, and it is money I can afford to lose. At the moment I am happy to trust the program.

Hard to measure monthly drawdown in Quantopian but the biggest overall was 40% between July and November 2008. But I reckon everyone lost money in that period! It had bounced right back by Feb 2009 however. I will work on trying to reduce this, but given the amount of money in play I am happy to take on higher risk. I currently only open max 5 positions for example, high risk but potential high reward. With more funds I would probably increase this number to mitigate risk.

I'm under no illusions here, and am treating it as a hobby. I forced myself to write everything in Python which was not my strongest programming language but something I was keen to improve. I'm also gaining knowledge in the an area I have a personal interest in, and am gaining a little bit of a foothold in something that could possibly be an option for future employment. For the money I could potentially lose there are a lot easier ways to spend it without gaining any benefits. As it happens though I am up a little so all good.

I don't know how much experience you have, so take with a pinch of salt if you already understand, but do a bit of research on CTA's and how they operate(d). It might give you a few ideas.

You could also try identifying if/when the market is in a up trend, downtrend or sideways range.
Backtest in each case and see which is your algo optimal for and can you develop one for each.

rojito · 09-10-2020 4:46pm

Mach 3 wrote: »

I don't know how much experience you have, so take with a pinch of salt if you already understand, but do a bit of research on CTA's and how they operate(d). It might give you a few ideas.

You could also try identifying if/when the market is in a up trend, downtrend or sideways range.
Backtest in each case and see which is your algo optimal for and can you develop one for each.

Cheers, I'll have a look into CTAs but I'm not sure I could get one to talk to me, I'm trading pocket change in their eyes I'm sure.

I look for a 200:10 SMA crossover on SPY to gauge if the market is trending up or down. 200:50 is more standard, but I found 200:10 lets my code react quicker.
In backtests my code flips on downturns to go short on weak stocks, I have that present in my live code but have made it toggleable with a simple setting. Currently it's turned off, I'm not sure I'm ready for that if we went on an extended downturn.

Mach 3 · 09-10-2020 7:54pm

rojito wrote: »

Cheers, I'll have a look into CTAs but I'm not sure I could get one to talk to me, I'm trading pocket change in their eyes I'm sure.

I look for a 200:10 SMA crossover on SPY to gauge if the market is trending up or down. 200:50 is more standard, but I found 200:10 lets my code react quicker.
In backtests my code flips on downturns to go short on weak stocks, I have that present in my live code but have made it toggleable with a simple setting. Currently it's turned off, I'm not sure I'm ready for that if we went on an extended downturn.

https://www.cmegroup.com/education/courses/managed-futures/evaluating-ctas-quantitative-and-qualitative-factors.html

rojito · 30-05-2021 12:53pm

Hopefully this doesn't come across as just being smug but I feel like I can at least give my self a slight pat on the back: my trading algorithm has just crossed the 100% profit line since the time I started this thread back in October last year.

It kissed the line back in March, but didn't quite cross it, and then fell away like everyone else at that time (drawdown 16.8%).

I can't lie, I do find it very satisfying that I have managed to double my initial investment with about 300 lines of code (*) and zero manual intervention in terms of trade execution.

Since October, Quantopian closed so I had to migrate to QuantConnect as my backtesting platform. It has some restrictions on it's free tier, and I have my own question marks over some of it's data, but it lets me validate the behaviour of the live algorithm as well as test out new ideas.

I have been playing around with some machine learning techniques such as portfolio optimization according to Modern Portfolio Theory, and price prediction with LSTM models but this is a totally new area for me so I am definitely learning as I go and haven't come up with anything I would want to run live yet.

Overall though, I'd just like to repeat that I think it definitely is possible to come up with a profitable algorithm and there are free or cheap tools out there to help anyone who is interested research, design, test and run trading algorithms.

* Full disclosure - originally it was about 300 lines of code, it has since grown with some re-architecture to make it more maintainable, increased logging, error handling, logging metrics to a Grafana dashboard, code to push notifications to my phone etc.

Static M.e. · 31-05-2021 10:21am

I have zero experience in this and \ or coding but I love the idea of it and wish you a lot of success. I imagine it's a lot of fun to watch your own little trading bot create 100% profit. If you take out the initial investment it is pure profit from now on. I would love to hear how the story continues to develop over time. We could setup a Boards Bot trading pool!

rojito · 31-05-2021 2:33pm

Static M.e. wrote: »

I have zero experience in this and \ or coding but I love the idea of it and wish you a lot of success. I imagine it's a lot of fun to watch your own little trading bot create 100% profit. If you take out the initial investment it is pure profit from now on. I would love to hear how the story continues to develop over time. We could setup a Boards Bot trading pool!

It definitely has been a lot of fun. I went from calculating how long it would take me to be a millionaire based on the gains in Jan and Feb to wondering at what stage I would pull the plug in March! (It didn't turn out too badly though I have to say)

But the bad days teach me not to get too carried away by the good days, so every cloud and all that.

I have learned though that it is not a case of writing some code and running it and moving on to do something else with my life. I think it has a become a long-term hobby that I see myself tinkering with for years to come.

I think the market will continue in a generally upward "covid recovery" trend for now so I want to leave my original investment in compounding for a while yet. Maybe if I can get to 50% more from where I am now (so a 200% overall increase) I'll take it out!

RainInSummer · 31-05-2021 9:00pm

Fair play. Profits are profits.

I'm currently ahead of you in strict percentage terms but the trade off is I've put several hundred hours into reading about the sector I'm into.

If you were to put a time value on what you've sunk into the project what do you think it would come to hours wise?

rojito · 01-06-2021 10:08am

RainInSummer wrote: »

Fair play. Profits are profits.

I'm currently ahead of you in strict percentage terms but the trade off is I've put several hundred hours into reading about the sector I'm into.

If you were to put a time value on what you've sunk into the project what do you think it would come to hours wise?

If I total everything, I would say it comes to hundreds of hours also.

I started in Feb/March 2019 having never bought or sold a share in my life. I spent a lot of time in the early days researching on Quantopian but there was a big learning curve.

I also put in a lot of hours during the first lock down last year as I wanted to port from the testing platform to something that could run live with Interactive Brokers once the covid recovery kicked in.

I also spent a lot of time over the Christmas break this year reworking a few things and just making the whole system more robust.

Since Christmas though, I would say I have been spending 3-5 hours a week and that is mainly playing around with new ideas.

I admire thorough research and due diligence a lot, some of the posts on the share picks thread just blow me away. Knowing myself though I think I could do that as a once off maybe but for me it wouldn't be repeatable. I couldn't motivate myself to keep the standards high enough on a continual basis. Whatever works for each individual I guess though, we are all working with the same goal, congrats on your own successes also!

RainInSummer · 01-06-2021 3:40pm

Cheers for the reply!

So a similar time sunk into it, but the advantage you have is that your tooling will continue to work whereas I'm back into several hundred more hours if I want to go in on a new sector. I know software has a lifecycle like anything else and will need to be maintained and altered to reflect any changes in the markets but it's not a bad spend of time!

Do IB charge for use of their APIs? I am with them myself and recall reading something about that.

rojito · 01-06-2021 5:52pm

RainInSummer wrote: »

Cheers for the reply!

So a similar time sunk into it, but the advantage you have is that your tooling will continue to work whereas I'm back into several hundred more hours if I want to go in on a new sector. I know software has a lifecycle like anything else and will need to be maintained and altered to reflect any changes in the markets but it's not a bad spend of time!

Do IB charge for use of their APIs? I am with them myself and recall reading something about that.

They don't have a direct fee for using the API but I pay USD$3 (so 2 and a bit euro) a month for live market data access for NYSE and NASDAQ stocks.

There's also their usual 10 euro monthly fee minus commissions.

I think there might be certain restrictions if you have less than 5k in your trading account also but I forget what exactly those were.

RainInSummer · 01-06-2021 9:20pm

Cool. That's not bad so.
Best of luck in your investing and your coding!

DL90 · 25-07-2022 12:48pm

https://www.boards.ie/discussion/2058120221/quant-algo-trading

Hi Rojito, I'm trying to send you a message but your account is private. Can you send me one - wanted to ask you a few q's.

Cheers!

HerrKuehn · 26-07-2022 2:11pm

https://www.boards.ie/discussion/2058120221/quant-algo-trading

Interesting topic, I didn't see the thread when it was originally posted.

In my experience (15 years developing trading strategies professionally across several companies), I haven't seen these type of moving average signals or pairs spread trading strategies working very well for long. I have been involved in several attempts at it, but never found much success to be honest.

I would agree with some of the above, if someone has a working strategy, they would not sell it. I mean I think that should be fairly obvious but perhaps it isn't. From what I have seen of the types of strategies people sell is that they are quite simple things that someone with a bit of cop-on could build themselves.

In terms of what pros do, I have to say, unfortunately, a lot of the strategies are developed based on some particular insight that may not have been seen by other people. The kernel of the strategy is often a simple enough idea, but it has so many additional "features" based on fairly specific things like exchange behavior etc to get it working successfully. Often even a great idea can be difficult to get working well. It might take several iterations to get something working the way I want.

rojito · 26-07-2022 2:53pm

https://www.boards.ie/discussion/comment/119389379#Comment_119389379

Thanks HerrKuehn!

I have been reading about the likes of Navinder Sarao and as you say it really does seem to way to huge success is to identify that kink or gap in the market that noone else has seen yet - the trading edge. One has to keep it legal though!

In saying that I still feel it is possible to make modest profits even with a seemingly simple but well-coded and well-tested algorithm. I've moved away from typical trend following or mean reversion strategies at this stage, but my algorithms I feel have a solid and straightforward premise, I could probably put together a half decent elevator pitch if I was that way inclined.

This year I have done a few online courses and read some books around AI and machine learning. I have some ideas of some things I think might work (hint: not typical price movement prediction) and hopefully I have some freetime opening up now where I can sit down and work through them. They might work, they might not, but as I have said all along for me it is about the opportunity to learn and gain some experience in a new field.

HerrKuehn · 26-07-2022 3:24pm

https://www.boards.ie/discussion/comment/119389745#Comment_119389745

Well what he was doing was spoofing/layering the book, this is completely illegal and highly regulated, especially in US markets. If you work for a trading firm you would receive training on all this, but if not, you need to inform yourself. You are criminal liable for any market manipulation like this and they can and will come after you (as you have seen in the case mentioned above) even years later.

So, that aside, yes, you need to find something with an "edge". You might be able to make modest profits with simple strategies, the main thing is making sure that a spread blowout or something else doesn't wipe out all your profits every so often. I don't want to put you off trying, markets are predominantly algos trading against each other nowadays, so obviously it is possible to make money (and sometimes a lot of money) doing it. I would say over the last 15 years I have seen increased competitiveness, with strategies generally getting more complex over time. A lot of things you might read in books would be from the early days of electronic trading when simpler ideas could work. I haven't read anything in the public domain that would be particularly enlightening, as people are, for obvious reasons, unwilling to share ideas that work.

rojito · 31-07-2022 12:50pm

https://www.boards.ie/discussion/comment/119389908#Comment_119389908

Yes, just using Sarao as an extreme example. I wouldn't be anywhere near his level in terms of intelligence. Plus I have found I'm actually quite risk averse when it comes down to it, so it's unlikely I will find myself in legal grey areas.

It makes sense that the more sophisticated models would never be shared publicly. I do find the community on QuantConnect is quite open and collaborative but for the most part I feel that is made up of beginners and amateurs like myself.

Is this an area you play around with by yourself in your free time or do you save it for work?

HerrKuehn · 08-08-2022 8:55am

https://www.boards.ie/discussion/comment/119411236#Comment_119411236

It wouldn't be something I work on as a hobby, no. I specialise in low latency co-located trading. I have a kind of profit share and obviously we trade much larger size than would be possible at home, so it is best to focus my efforts on work. For my own investments, to be honest, I just have professionals/wealth managers look after it. It wouldn't make much sense for me to spend a significant amount of time building something to maybe make 10-20k a year.

Donald Trump · 08-08-2022 9:04am

https://www.boards.ie/discussion/comment/119389908#Comment_119389908

Sarao was done for spoofing and layering alright but he also made tens of millions on "legitimate" strategies.

Some say he was scapegoated to a certain extent for the Flash Crash but he was definitely involved in illegal practices. He was later conned out of all his money.

HerrKuehn · 08-08-2022 9:15am

https://www.boards.ie/discussion/comment/119443641#Comment_119443641

I have no idea what proportion of his earnings were legal. The fact is you are not allowed to place large orders on one side of the book to encourage others to cross the market on the other side. It is not allowed to enter orders without the intention to trade. Orders entered need to take market conditions into account, so you can't send large orders into thin books. If larger firms were doing this there would be absolute carnage in markets.

Donald Trump · 08-08-2022 9:28am

https://www.boards.ie/discussion/comment/119443677#Comment_119443677

Other poster mentioned Sarao in relation to quant trading. From memory, Sarao was found to have made in the order of 10m from his market manipulating strategies, but had made something like 30 or 40m overall. When convicted and fined he could not pay as he had been since scammed out of all his money.

Poster is talking about quant trading. Not market manipulation. Pointing out that he made tens of millions on legitimate activities in not disproved by saying he also made money using illegal tactics.

BTW, spoofing and layering does not need thin books to work. The just need the market to shift a few ticks one way or the other to make money. In fact, Sarao did his against the S&P E-mini if I remember correctly. (Or at least that is the one that caused the Flash Crash) And large firms do indeed make money by getting as close to the line as possible. Sarao was not the only one involved in such tactics. He was identified years later, almost by accident. Someone kinda just came across his activity when looking back. And that was after all the major investigations. That is why there is some merit to the view that he was scapegoated as being basically the sole cause the Flash Crash. That he was at that craic is not in dispute

HerrKuehn · 08-08-2022 9:41am

https://www.boards.ie/discussion/comment/119443717#Comment_119443717

Well they say he was involved in various market manipulations from 2005-2010. I have no idea how you would calculate what was legit and what was not, unless they went through every one of his trades. I know what spoofing and layering is, I have regular training on this stuff. He may have been blamed for the flash crash but he was given a very, very lenient punishment for it.

Donald Trump · 08-08-2022 10:01am

https://www.boards.ie/discussion/comment/119443768#Comment_119443768

You can read the court judgments. There are a few related to his extradition and then his ones from the US. They are not that long. If I remember correctly, those give a breakdown of the profits and trades. His fine was not small however he could not pay it due to having being scammed himself. Any potential imprisonment would have been affected by his autism. He also assisted the US government in other cases - including acting as witness in one if I remember correctly.

I read the original CFTC Flash Crash Reports when they were published (preliminary and final a few months later). Market manipulation was not even suspected, never mind blamed at the time.

There is a line between layering and legitimately cancelling an order. The extremes are easy to discern, but somewhere in the middle there is a grey area.

HerrKuehn · 08-08-2022 10:09am

https://www.boards.ie/discussion/comment/119443845#Comment_119443845

Layering would be showing large size on one side with the intention to encourage others to trade into the other side. Obviously intention could be subjective but traders or algos don't generally send large orders in without reason. If the large order is a hedging order, so the intention to trade is there, this can still be an issue if it is outsized with respect to the market, so you can't send a hedge of 5k lots if there are only 100 lots on the other side.

Quant / Algo trading

Comments