Tuesday, 2 August 2011

Planning the Development of a Bot

Time to get down to writing some bots. As a software developer by trade, that means we need to do some planning.

A bot is simply an automated way to do something you would do manually. It has several advantages we wish to exploit, and if anything we do negates one of these advantages we must reconsider whether it's suitable for a bot to be doing it.


  1. Speed is of the essence. A bot can obviously act and react much faster than a human. Because of how the APIs work a bot can monitor many markets simultaneously and strike/cancel bets much faster than any human. This has obvious advantages as the earliest mover in a market to make the right decision typically will make the most money.
  2. It will behave consistently according to design. We have all found systems, techniques and tricks that appear to be profitable. Those with strong discipline might even be able to stick to them. For the rest of us though, there is always a desire to have a quick punt or to break our system based on a hunch. It makes the statistical analysis of what we're doing borderline ridiculous. That's when losses occur. Bots will do what they're programmed to do, and only what they're programmed to do.
  3. They free you up to do other things. Back in 2002 when I was first trading I was spending 10-12 hours/day at screen. It was demoralising and boring work grinding out a meagre existence. I would have rather spent my days enjoying watching the sport without needing to keep an exchange screen open constantly, or doing something entirely different instead. A bot can - and probably should - work completely unaccompanied and perhaps running on a server in a data centre so I don't even need my computer to be turned on if necessary. It should be able to work on markets whilst I am enjoying the fruits of its labour.
  4. The nature of building a bot forces you to be more systematic and analytical. Rather than going with "it's a hunch" style of betting/trading, you must justify every line of code. Even better, following industry standard programming methods, a bot can be "iteratively" developed, that is to say tweaked and enhanced bit by bit based on data and results. This requires us to go back and revisit our assumptions regularly, and make new ones. It's a healthier way to think about what we're doing here.
So we want speed, consistency, complete automation and iterative development based on systematic and analytical thought processes. What could be easier?

A bot has a few things it needs to be able to to meet these objective.

It must be able to login to a betting exchange or bookmaker website (the latter important for arbing bots), select an event or events, determine which selection or selections it will bet on, strike the bets thereby committing capital according to some sort of staking method, monitor those bets and impose a stop loss if necessary, green up when optimal if we believe greening up is the best way to behave, and log all the data about what it's done somewhere so we can go back and check what it was doing in a spreadsheet later.

There's a couple of bits of this which are relatively straight-forward, and others where we need to make some decisions. The logging into an exchange at first appears trivial (use the "login" function of the API), but do we want to use the API or do we want to screen scrape? Why have we made that choice? I'll be exploring that at some point and showing what the code for each looks like. If you've never written/read any code before, don't panic! It'll be understandable.

However the other jobs all give us much more to think about. We need to justify every line of code, and to meet the "systematic and analytical" test that a bot should pass, we need to handle event selection, runner selection, staking, stop lossing and greening up based on some sort of scientific analysis.

Next job then, it's time to do some data gathering to test some hypotheses.

The first idea for a bot I am going to test is one I expect to fail at the analysis stage (which is good! If it's going to fail, that's the point in time I want for it to fail, not after the bot is built and running out onto the exchanges with hard cash), but will be relatively simple to implement and therefore a good "starting project" for this blog. It is very simple to state:

Find non-handicap UK/IRE horse races which will go in-play, and back the favourite for 5% of my betting bank in the place market 5-10 minutes before the official off. Then lay it at a price giving a 10% RoI greened up with the "keep in play" option on the bet selected.

It's a very naive system based on the assumption that favourites in non-handicaps are priced fairly, and I currently have no evidence of it being profitable. Each component needs to be tested and verified before I build any code. I'm happy to modify my thoughts based on the results I get back, and I have plenty of other ideas for bots if this one doesn't stand up to analysis: the purpose of this idea is primarily to go through the analysis stages.

In the next few posts I'll be grabbing data from multiple sources and building/using tools to analyse whether this system is viable. I'll be using statistical analysis, of course, but also some funkier analysis tools like decision trees to refine event selection and the like.

Feel free to ask any questions if you have them.

11 comments:

  1. Absolutely intrigued, didn't program for 20 years (do you remember C+) but have been thinking about something similar, I hope my logical reasoning still stands and I can understand. What language or programming code will you be using ? Will you be giving detailed updates ? This is kinda like a personal Market Feeder Pro is it not ?

    ReplyDelete
  2. Average Guy: C++ is still a very popular language, and a new standard for it is due out in the next year or so!

    I'm most likely to choose Ruby for this for a few reasons. Primarily, it's easy for even novices to read code snippets and understand what is going on. Secondly, there are some *brilliant* web scraping libraries for Ruby which will make some aspects of this much easier. It's also borderline trivial to write REST and SOAP API clients in Ruby as well. No memory management to worry about, no weirdness going on that will confuse most people following along.

    I will be giving details. Mostly snippets, but also full tools people can deploy. A few tools around data collection I'll open source under an MIT license. Some of the bot code might remain proprietary, but I'll make my mind up closer to the time: I'm minded to make as much of it as open source as possible.

    Market Feeder Pro for me has a couple of disadvantages:

    1. It's Windows only. I know the majority of people out there are on Windows, but I want something that will work on OS X and Linux, and critically is controllable and deployable on a unix command line. I want to be able to throw it onto an Amazon EC2 image, and for it to just work without me having to babysit it or leave a window open. That doesn't mean GUIs aren't possible, just not a high priority for me right now.

    2. MFP is a commercial program there to make the owner money. No problem with that, but my objective is to have an open conversation about bot design and to also help those interested in bots to work out how to write their own. I don't expect to make any cash from sharing this knowledge, but in a year or two I might have enough material and knowledge to write a book on it that is better than the (in my opinion, quite flawed), current only text on writing bots on the market.

    3. It's currently Betfair only and the triggers are based entirely on the betfair market data. What if I want to trigger an action based on the weather forecast near a racetrack or cricket stadium? What if I want to write an arbing bot?

    My long term goal might be to write a tool that develops what we call a "Domain Specific Language" for a bot. That means I can write in a file a description of a series of triggers based on a wide range of data sources, and describe a betting strategy. It could eventually mean we get to write things like this:

    market = BetfairMarket.find("England vs India > Match Odds")
    market.back("Draw", BettingBank.amount("5%")) if market.location.weather_forecast contains "Heavy Rain" and market.selections("Draw").price > 3.0

    I.e. in a couple of lines we've found a way to back the draw above a certain price if heavy rain is forecast. I don't know how you would do that in any of the tools currently available.

    Hope that helps you get an idea of where I'm coming from. I'm open to feedback and taking other directions with it all though!

    ReplyDelete
  3. Hi Paul, these days I've been thinking about the things you write here. Creating my own bot. But I would'n even know where to start.
    I used to love programming and was also pretty good at it too. But that was 10 years ago. :( Now i forgot almost everything. And I'm pretty sure what i knew is now outdated. All a got left with is my analytically way of thinking. I was wondering if there would be anything I could help you with and in exchange you could teach me what you do ( by pointing me to what I should learn )? And how much time would it take me to learn all I'd need to know to be able to create my own bots?

    ReplyDelete
  4. Vlad, two things:

    1. If you have learned how to program at some point in the past, you're fine. The same languages will still work and you can crack on if you want to, you might just need to pick up the odd book and refresh your memory.

    That said, there's always room to learn more. I'll be using Ruby which is a beautiful language to catch up with if you want to pick up a book on programming again. Seriously: its creator has as the main design goal the desire for it to be joyful to program in. As a professional developer, it changed my work life.

    2. I'll be releasing code for all to see, so of course you can learn. Everybody who reads this blog will get to see code, and if you don't understand what I'm posting you can ask questions in the comments and I'll answer everything I can. I'm not a "buy my system for £200/month, I'll not be accepting new members soon!" type of a guy: I'm transparent and want everybody to understand what I'm doing.

    I want a conversation about bot creation, with newbies and old hands alike, so feel free to join in.

    I hope that as I start building things here you and others will be able to learn more about what I'm doing and understand how it all plugs together. I think the first bots will be deployed this year, so there is no reason why your bots won't be there in a similar time frame. That might seem a long time, but I think to get it right, that's not so awful.

    After the first bot or two is deployed, I expect I'll be tinkering and building bots for the next few years and blogging here, so there will be improvements for a while to come.

    Welcome to the blog, and I hope with time you gradually get a profitable bot that you're proud of!

    ReplyDelete
  5. Paul, as a hobbyist programming VB I am in the (painstaking) process of writing a simple bot. I will watch your pages with interest, particularly as you intend using a language which I have never heard of!

    Good luck with it, and your trading.

    Dave

    ReplyDelete
  6. This all looks very interesting. If you need/want any help, I may be useful (or I may not, of course) with my developer/quant Ruby/C#/Excel, er, "skills". Not C++, though - it gives me a rash.

    ReplyDelete
  7. What programming language will you use? It would be a good idea to use some functional language. I know it is not for beginners but it would be a good reading, and you will be the first one writing about using such programming language for betfair bot development.

    You mentioned about data gathering, what publicly available recourses do you want to use?

    I have got my own bot and tested its flexibility on your example, if anyone is interested I posted article here: bitly.com/PaulBot

    I am looking forward to read about your bot development, as many started on blogsphere, but that was all they did, just stated intention to develop betfair bot.

    ReplyDelete
  8. Guys, I'll be coding most of this in Ruby. It's what I use in the day job, and I've used it every day in anger since 2005.

    mikewoodhouse: might be great to get some expertise in from the quant side of things. And don't worry, C++ brings me out in a rash too. :-)

    Stefan: Ruby can do functional coding. It's OO-first, but functional is doable, and I'll definitely be exploring that aspect. I might be tempted to do a little bit of code in Clojure down the line. A sane Lisp syntax on the JVM should be funky enough to get the power of functional code, but accessible enough on the JVM that people can get involved.

    I'll be using any resources I can get my hands on for free. I may be persuaded to pay for data unobtainable anywhere else if I think I need it.

    It's interesting that you managed to get a bot together so quickly - looked really impressive. I should stress though I don't intend to build that exact bot as I suspect it's unprofitable: it's a series of hypotheses that I'll test, adapt and then build the corrected result which might look similar. It may be 5% bank for 10% RoI is sub-optimal by a long way, which is why I deliberately called it a naive bot! I hope you're not running that code for real... ! :-)

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Functional languages have no concept of variable as imperative languages do. You can write code in Ruby, in a "functional" style but Ruby is not functional language like Clojure or F# is. If you do your coding in Clojure then I can do some experiments in F# and we could compare experiences and results.

    I develop for betfair from 2006 so I have got thousands lines of code, so extending my bot to fulfill requirements for your testing bot was just question of minutes, writing my article and making video took me more time. I believe when you build your base bot platform you will be even more productive if you code in functional language, at least they say so.

    I test all my bots in the training/practice mode, so without risking any money on betfair. I learned my lessons, so my bot runs in the training mode automatically if I run it from Visual Studio.

    ReplyDelete