Placing your first Forex trade with Python

Update: I updated the code so it works with Oanda's new API. Get it here

Time to talk about brokers, how to place a trade programmatically and most importantly how not to get scammed.

This is the third part of the series: How to build your own algotrading platform.

A broker is nothing more than a company that lets you trade (buy or sell) assets on a market through their platform. What is very important for algotrading is:

  1. The broker offers an API in order for us to place orders
  2. You can have a demo account to run your staging environment and experiment
  3. The spread is as small as possible

In our case, we don't really care about spread as we won't be doing High Frequency Trading any time soon.

Even though brokers are regulated, there have been incidents in the past couple of years, were brokers folded due to certain conditions. Be very wary if

  1. There are no reviews of the broker on the internet (or most of them are bad)
  2. If the broker offers you some crazy leverage (like 1:200)
  3. If the broker seems to be in a very strange country

What could happen is that you start making some money and you aren't be able to pull them out. Seriously. Super stressful situation.

But let's switch to a happier note which is opening an account and placing our first programmatic trade. Whooha!

I am using Oanda as a broker (I am not affiliated with them) and they offer a pretty decent API, libraries on github and a free demo account.

Go and open a free fxTrade Practice account and then sign in.

After you sign in to your demo account, go to Manage API Access. There you can find your API key which we are going to use in our system to place trades. MAKE SURE YOU DON'T SHARE THIS KEY.

The code for this is and all other posts is on github and you can install it and run it pretty easily.

Update: Oanda released a new (kickass) execution engine called v20 and they have released a new (improved) API. This post has been updated in order to use the new API but if (for any reason) you want to check the old code, it is right here. You lucky you!

Connecting to Oanda needs a conf file - which you can generate using a script that Oanda provides here or you can just create it yourself. Why would you want that? First of all when it comes to credentials (and my money), I prefer to know everything that is going on. And I don't like having to install PyYAML just to read a conf file. Feel free to use either method.

Now, prepare to be amazed. The code is straight-forward. We initialize the API:

import v20

api = v20.Context(
        'api-fxpractice.oanda.com',
        '443',
        token='HERE GOES YOUR API KEY')

and now let's place an order (buy 5000 units of EURUSD)

response = api.order.market(
             'ACCOUNT ID',
             instrument='EUR_USD',
             units=5000)

print("Response: {} ({})".format(response.status, response.reason))

>Response: 201 (Created)

Check the current price is as easy!

from datetime import datetime

latest_price_time = datetime.utcnow().isoformat('T')+'Z',

response = api.pricing.get(
                account_id,
                instruments='EUR_USD',
                since=latest_price_time,
                includeUnitsAvailable=False)

for price in response.get("prices", 200):
        if latest_price_time is None or price.time > latest_price_time:
            print "Buy at", price.bids[0].price

Super easy. Don't worry about what EURUSD is or how many units we are buying or what a market order is. For now, we have placed our first trade from our laptop and we are going to build our own API to place trades. Exciting stuff!

You can read Oanda's documentation here to see what else you can do with their API and find the Python library here.Tons of examples are available from Oanda's github page here.

Coming up next, connecting to a real LIVE algotrading system, running from my RaspberryPI at home.

You'll be able to see the (almost) final program running and we'll talk more about Forex and strategies.

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Trading Ethereum: Making 10% every 20 minutes

This is more of a "How to build your own algotrading strategy - the Ethereum edition" and not a "make money fast" blog post.

It is also a real example with real returns (and real production errors that cost me money) where you can see how to identify opportunities, why algotrading is awesome and why risk management can save your ass.

This is the another post of the series: How to build your own algotrading platform.

I get this question almost on a daily basis. How can I find a good strategy? How can I built my own? Do I need to have a PhD in mathematics? Statistics?

Newsflash: If I can write a strategy, anyone can write a strategy. Trust me on that. The only trick is to look for a simple one.

Cryptocurrencies

Update: This post has been rewritten "at least" five times (as "The DAO drama" escalated) and it is the perfect example of a strategy doing a full circle.

I started getting involved with Ethereum early on as I really liked the "run your algorithms on the blockchain" thing. When TheDAO came out, I read everything about it and loved the idea. You don't need to understand what Ethereum, theDAO, blockchain is at this point (I promise I will ramble on a another post). The same ideas apply to Forex, Stocks even Pokemon balls.

     As I said before, simple ideas turn into (simple?) strategies.
     Complex strategies turn into mayhem that is impossible to backtest
     and deploy without errors.

My four step - captain obvious - system

I personally, have a specific way that I work.

Step one : Identify an idea.

My idea in this case is that there are a couple of exchanges offering Ethereum and DAO tokens. What if there was an arbitrage between those?

Step two : Manually test the idea.

If something "kinda works", I am on to something. All I had to do is execute all the steps manually and write down any fees, conditions or anything that should be documented.

Step three : Automate

This algorithm is not a high frequency trading algorithm. There is a considerable amount of time risk (which you can eliminate as we'll see later on) but what I did could be done manually. The problem is that I would have to spend all time in front of my computer, checking if there is an arbitrage condition and if there was, I had to act fast and without messing up. Oh and I had to recruit five of my friends to scale this up.

Long story short, I spent Presidents' Day writing a simple program that will replay all my manual steps. The program would crash and it was not more that 100 lines of code. This is the data collection stage where I see if there is an advantage that algorithms can give me. Advantages can be:

  1. Something that be automated and run 1000 of times per seconds or 1000 times in parallel
  2. Something that thinks faster than I can
  3. Something that has no feelings to screw up my system

If there is at least one or more conditions met, I will start building and rewriting the algo.

Step four : All in

I am kidding and you'll see in a bit why risk management is super important in this business.

The Arbitrage

Let's discuss a little bit about what this arbitrage was.

The idea is: "I wonder whether Kraken and Shapeshift have different prices for the same assets". This is a classic arbitrage case (Kraken and Shapeshift are "exchanges"). I could exchange DAO for ETH on Kraken, transfer ETH to Shapeshift, exchange ETH for DAO and send them back to Kraken and due to price inconsintencies I would end up with more DAO than I initially started! Risk-free money, the best kind of money.

You can make money as long as ETHDAO from Kraken * DAOETH from ShapeShift > 1 (+ fees + gas). Very simple formula, right?

Every cycle, was a 2% to 10% return of my capital. After a while, I started hitting the limits of Shapeshift and I had to make this run in parallel.

The question is what would you do if you had an algorithm that makes you 10% of your money every 20 minutes? The stupidest thing you could do is put tons of money into it.

Hybris

If you are not familiar with the Greek word hybris, consider yourself lucky. Hybris is when you think that you are invincible, better than gods. And this is the biggest NO NO you can do in trading.

After a couple of weeks, theDAO was hacked. 160 million dollars were stolen (or should I say frozen?) and noone knew what was going to happen. For me, this happened, 10 minutes before boarding a plane to fly to New York. Hybris. Or as people in the US say: Fuck.

I was smart(lucky?) enough to have good risk management habits (thank you Forex). I never, ever, ever risk more that 2% of my capital even if it seems the best kind of deal.

Luckily, the money were "restored" and I could withdraw/convert my DAO to Ethereum (but yes, I bought in-flight WiFi to keep up with what's going on).

This whole experience is a reminder that there are always things that you cannot predict. Things that you cannot control. This was a systematic risk and there was no way I could have seen it coming. Pushing buttons and building algorithms is not enough. Proper risk management and knowing when you need to take a chill pill is what can keep you in the game.

On the next post, I will post the whole algorithm and go line by line. I also plan to discuss a little bit more about theDAO and Ethereum. If you don't want to miss any of these and get some more additional info, feel free to sign up to the newsletter where I talk about fintech, algorithms and the markets.

By the way, if you want to make your own cryptocurrency and learn more about Ethereum, I have a great post with the code posted here.

Coming next: Diving into the ETHDAO algotrading program

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Machine Beats Human: Using Machine Learning in Forex

This is the another post of the series: How to build your own algotrading platform.

Machine learning and trading is a very interesting subject. It is also a subject where you can spend tons of time writing code and reading papers and then a kid can beat you while playing Mario Kart.

In the nexts posts, we are going to talk about:

  1. Optimize entries and exits. This and only this could make a ton of difference in your bank roll.
  2. Calculate position size (in case you don't like Kelly criterion)
  3. Find possible correlation between different pairs (pair trading). I love the EURUSD vs GBPJPY correlation!
  4. Calculate support & resistance lines

But what is Machine Learning?

(image from fastcompany)

Machine learning algorithms are algorithms where a machine can identify patterns in your data. Yeap, it is that simple. For example, find all the animals in this photo and draw a box around them. Also, name that animal. Crazy I know. For trading as you can imagine it is pretty similar:

"Find how can I make money based on this chart and do all the trades.
Then wash my underwear and don't mix the colored with the whites"

In order for a machine to "learn", you need to teach it what is right or wrong (supervised learning) or give it a big dataset and let it got wild (unsupervised). For identifying objects this is straight-forward but what about trading?

Playing with data

I looked around to see if there is any machine learning program that can identify S/R lines but to no avail. So I decided to write the first machine learning program in python that identifies support and resistance lines in Python. Another first! Hooray!

But how can an algorithm identify these areas? Hoooooow? Ladies and gents (and robots), let me introduce you to MeanShift, an unsupervised algorithm that is used mostly for image recognition and is pretty trivial to setup and run (but also very slow).

The idea is that this algorithm will let me partition my data (forex ticks) into areas and then I can use the "edges" as support and resistance lines. Cool idea but does it work?

Spoiler: It works.





We analyse around 12 million datapoints of EURUSD in 2014 and a couple of months of 2015. The resistance lines are placed automagically by a machine learning algorithm.

What is really cool (and spooky) is that the algorithm pretty much nails it. NAILS it hard. It gets really spooky when we are going to use the algorithm to identify micro-structures and start scalping.

The system is able to process any kind of timeseries data (stocks, forex, gold, whatever) and it will render an html interactive chart (like the chart above) with your data and the machine generated S/L. The code is here so go crazy.

Now let's step through the code. After you have your set of data you need to read them and clean them. Prepare for some pandas magic.

# read csv files with daily data per tick
df = pandas.read_csv(filename, parse_dates=[0], index_col=0, names=['Date_Time', 'Buy', 'Sell'],
                     date_parser=lambda x: pandas.to_datetime(x, format="%d/%m/%y %H:%M:%S"))

# group by day and drop NA values (usually weekends)
grouped_data = df.dropna()
ticks_data = grouped_data['Sell'].resample('24H').ohlc()

We drop the empty values (weekends) and then we resample the data to 24 hours candlesticks (ohcl). This makes it MUCH easier to plot. The grouped_data are the data that we will feed into the ml algorithm.

Then we prepare the data that we are going to use in the algo.

# use 'ask'
sell_data = grouped_data.as_matrix(columns=['Sell'])

# calculate bandwidth (expirement with quantile and samples)
bandwidth = estimate_bandwidth(sell_data, quantile=0.1, n_samples=12000000)
ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)

# fit the data
ms.fit(sell_data)

On the next post, we'll discuss how to make this work even better, discuss some very interesting results (can the algorithm actually predict about the future?) and start using it in our own trading. If you want to check the next article and read more about trading and investing using algorithms, signup to the newsletter.

Coming up next: Machine Learning Gone Wild - Using the code!

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Forex and Taxes for new traders

This is the another post of the series: How to build your own algotrading platform.

Update: The Machine Learning post is going to be epic but it takes so much time to make the code presentable. Bear with me, cool things are coming (as you've read at the newsletter)

Disclaimer: THIS IS NOT TAX ADVICE.

What is really surprising is that the majority of the requests in the newsletter are:

  1. Use tools to assist Trading
  2. Machine Learning to optimize trades
  3. Taxes

The first two, I can understand. Everybody wants to be a better trader. I get it. But taxes? TAXES?

This is your lucky day. Forex taxes are super easy. Seriously. By default (this is called Section 988), all your losses are going to offset your income taxes without the 3k limit per year. This is much better than stock trading where losses offset your capital gains.

But what happens to gains? WHY DO YOU CARE? WHY? The majority of the Forex traders lose money (I call it "paying tuition") the first year(s), so you are better off keeping it simple until you have a proven and consistent strategy.

Having said that, if you actually make a profit, you are taxed at the short-term capital gain (sometimes up to 40%). The solutions to when you start making money are:

  1. Opting out of Section 988 and get taxed under Section 1256 where 60% of profits are taxed as long term capital gains and 40% as short term (but now losses cannot offset your income). This is very good when you make money, very bad when you don't.
  2. Start an LLC

For people that just started experimenting with Forex and algotrading, I always suggest them to stay with Section 988 (the default) and when they start making some money (consistently) or they want to go full time, talk to me :) Seriously, there are so many things that you will start doing differently when you go from the "hobby" stage to "second income" to "full-time job" that there is no reason to over-optimize this.

Coming up next: Machine Learning Gone Wild!

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Building the simplest backtesting system in Python

This is the another post of the series: How to build your own algotrading platform.

Last time we talked about The "for-looper" backtester (as I love to call them). Now it's time to see some code!

We said that we have something like that:

:::python for each element of readhistoricaldata(): apply_strategy() how_our_strategy_did()

Sweet, let's load our strategy, load some historical data, run our algorithm and print some results!

I prefer to have my strategies in a JSON format that contains the name of the strategy and some specs (like how many pips for stop_loss or take_profit etc). That way, when we'll start using an event-based backtester, we can pass the strategy through a machine learning algorithm and try to optimize it.

Next line is loading our data in. I know people don't like pickle and there other ways to load data (and we are going to talk about BColz at some point) but for now, just bare with me.

The next line is self-explanatory. We pass the historical data to our algo and we get back some stats to print.

Let's focus on the algorithm a little bit and we can discuss plotting etc at a later point.

The magic of the simple backtesting system

Prepare to be amazed by how ridiculously easy to do this.

Short comings What happens with this type of backtesting is that 1. you'll probably make mistakes when you want to using the exact same algorithm 2. You won't be able to write a very complex strategy (at least that easy) 3. Very difficult to scale (compared to event-based) 4. You need to have your simulation and execution in the same language

BUT remember that this is the BEST and fastest way to start out and figure out how all these things work.

Coming up next, using other well-known backtesters in Python and adding graphs to our own!

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Strategies for Forex and trading

This is the another post of the series: How to build your own algotrading platform.

What is a good algotrading system without some neat strategies to deploy?

Below there is a list of strategies that I found online (or sent to me by traders that are on the newsletter). I plan to update the list as I keep coming across to new ideas.

The concept is that as we keep diving more and more into our algotrading system, I will show you how to code and deploy these strategies. I know for sure that most of them work with minimal changes. Worst case scenario, you'll have a system to test out your assumptions.

Here is the list (and please send me any other strategy that you think it should be included):

Coming up next, sharing and discussing my simplest (but most successful) backtester!

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Building a backtesting system in Python: or how I lost $3400 in two hours

This is the another post of the series: How to build your own algotrading platform.

Building a backtest system is actually pretty easy. Easy to screw up I mean. Even though there are tons of excellent libraries out there (and we'll go through them at some point), I always like doing this on my own in order to fine-tune it.

From all the backtesting systems I have seen, we can assume that there are two categories:

  1. The "for-loopers"
  2. The Event generators

Today, we'll talk about for-loopers.

The "for-loopers" are my favorite type of backtesters. They are trivial to write and super fun to expand but they have some vital flows and sadly the majority of backtesters out there is "for-loopers" (ps: I need to find a better name for this!).

How for-loopers work? Using a for loop (as you might have guessed). It is something like this:

for each element of readhistoricaldata(): 
    apply_strategy() 
    how_our_strategy_did()

Very simple right? This is how one backtesting system works, that runs a momentum strategy:

So, what is the problem?

  1. Very difficult to scale (horizontally)
  2. Needs lots of work to keep your apply_strategy() working on backtesting and production
  3. You need to have everything in the same programming language

Let's dive into these, one by one.

Scalability. I was experimenting a couple a weeks ago with a hill-climbing algorithm to optimize one of my strategies. It is still running. After two weeks. And I build uber-scalable systems for a living. Why is it still running? You can use multiprocessing, Disco, producer/consumer (using ZeroMQ) or just threads to speed this up but some problems are not "embarrassing parallel" (yes, this is an actual term, not one of my made-up words). The amount of work to scale a backtester like this (especially when you want to do same machine learning on top of it) is huge. You can do it but it is the wrong way.

Production and backtesting in sync This. The times I have been bitten by this. I can recall the lost trades where I was "hm, why I entered this trade?" or my old time favorite "WHY TRAILING STOP WAS APPLIED NOW?".

Story time: I had an idea in order to optimize my strategy, to run a backtester to see what would happen if I could put a trailing stop AFTER the trade was profitable in order to always secure profits. Backtesting worked like a charm at a 13% increase of earnings and production lost every single trade. I figured it out after my algo lost $3400 in a couple of hours (a very expensive lesson).

Keeping the apply_strategy in sync is very difficult and becomes almost impossible when you want to do it in a distributed fashion. And you don't want to have two version of your strategy that are "almost" identical. Unless you have $3400 to spare.

Using different languages I love Python. And Erlang. And Clojure. And J. And C. And R. And Ruby (no actually I hate Ruby). I want to be able to leverage the strength of other languages in my system. I want to try out strategies in R where there are very well-tested libraries and there is a huge community behind it. I want to have Erlang to scale my code and C to crunch data. If you want to be succesful (not only in trading), you need to be able to use all the available resources without prejudices. I have learnt tons of stuff from hanging out with R developers regarding how you can delta hedge bonds and visualize them or why Sharpe ratio can be a lie. Every language has a different crowd and you want as many people pouring ideas into your system. If you try to have apply_strategy in different language then good luck with (2).

Are you convinced now? Well, I am not trying to convince you as for-loopers is a great way to run your initial tests. It is how I started and for many strategies I don't send them down to the pipeline. A "better" way (so you can sleep at night) is the event generators.

Coming up next, sharing and discussing my simplest (but most successful) backtester!

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

Downloading historical Forex tick data and importing them in to Python using Pandas

This is the another post of the series: How to build your own algotrading platform.

Before running any live algotrading system, it is a good practice to backtest (that means run a simulation) our algorithms. Have in mind that this doesn't mean that if your system is killing it for the last 5 years/months/days it will make a profit but it is a good indicator that you might be on to something.

There are four things that we need to take into consideration when we do our backtesting:

  1. The quality of the data
  2. How to load them efficiently
  3. How to built our backtesting system
  4. Try to have our backtesting and our live system share as much code as we can

Today, we are going to focus on (1) and (2).

For Forex data, I am using GainCapital. Their data are in the form of ticks. For a free source it is good enough. I used to use Oanda's historical data service but it seems that they moved it to a premium product. Too bad. Make sure that you use GainCapital's data only for experimentation.

For any other kind of paid historical data (ETFs, stocks, options stc), I am using eoddata.com (they also have some forex historical data but I haven't used them).

Let's download data for a week and experiment a little bit. The link to the data is http://ratedata.gaincapital.com/2015/11%20November/EUR_USD_Week1.zip for the first week of November 2015.

First we need to unzip the file :::python >unzip EUR_USD_Week1.zip

and you'll get a 25MB file named EUR_USD_Week1.csv. These are data for one week for one currency pair. You can imagine the amount of data you need to process for all currencies for the last five years (hint: a lot!). But don't worry, we are going optimize this. For now, let's open the file and inspect.

>head EUR_USD_Week1.csv
lTid cDealable CurrencyPair RateDateTime RateBid RateAsk
4464650058 D EUR/USD 2015-11-01 17:00:06.490000000 1.103380 1.103770
4464650061 D EUR/USD 2015-11-01 17:00:06.740000000 1.103400 1.103760
4464650072 D EUR/USD 2015-11-01 17:00:07.990000000 1.103390 1.103750
4464650083 D EUR/USD 2015-11-01 17:00:08.990000000 1.103400 1.103750

the things that we care about is the RateDateTime, RateBid and RateAsk. As you can understade each line has a timestamp and the how much was the price to buy or sell. Formats downloaded by other services are pretty similar.

There are many ways to load these data into Python but the most preferable when it comes to data slicing and manipulating is using Pandas. We can always use the csv library to load data (and it might be faster) but we need to do some optimizations and processing first that as you will see it is pretty easy with pandas.

Another great tool to load TONS of GBs pretty efficiently and very fast is using Bcolz, covered in a much later post (or you can read a preview if you have signed up in the newsletter.

Manipulating data using Pandas The data we downloaded are in ticks. Unless we are building an UHFT (ultra high frequency trading) algorithm, it is much more efficient (memory, storage and processing-wise) to "group" these ticks into seconds (or minutes or hours depending on your strategy). This will make our download scale down from 25MB to just 35KB which translate to HUGE performance and memory benefits.

Let's group all these data in 15 minutes. How? Time to fall in love with resample.

df = pandas.read_csv(filename, parse_dates={'DateTime'}, index_col='DateTime', names=['Tid', 'Dealable', 'Pair', 'DateTime', 'Buy', 'Sell'], date_parser=parse)

# let's erase the columns we don't need
del df['Tid'] 
del df['Dealable']
del df['Pair']

# group every 15 minutes and create an OHLC bar
grouped_data = df.resample('15Min', how='ohlc')

The resampled the dataset looks like this:

DateTime Buy Open Buy High Buy Low Buy Close
2015-08-02 17:00:00 1.09706 1.09821 1.09706 1.09735 1.09752 1.09853 1.09740 1.09762
2015-08-02 17:15:00 1.09731 1.09825 1.09731 1.09769 1.09762 1.09843 1.09758 1.09808

This is called OHLC (Open High Low Close) bar for every 15 minutes. You can see now that the ticks are grouped in 15 minute segments and you have the highest and lowest point that the price reached during these 15 minutes and also the open/close for buy and sell. Pure gold! Not only you have all the information you need but now it is extremely fast to load it. You just need to save the data:

:: python
# save to file
grouped_data.to_pickle(filename+'-OHLC.pkl')

and then you can reuse this 35kb file.

We can write a simple momentum algorithm that checks if there was a huge movement the last 15 minutes and if that was the case, let's buy. We will dive into this in a later post.

You can see the code as always on github.

Coming up next, building a backtesting system from scratch!

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

What is Forex

This is the second part of the series: How to build your own algotrading platform.

Before building any algotrading systems, you need to know how to trade manually. What that actually means is that you need to lose money on your own before blaming the machine. As simple as that.

So, let's talk about Foreign Exchange or Forex as the cool guys call it.

First of all, why do we choose Forex for algotrading? Why don't we become millionaires trading like everybody else? Why not just buy Tesla, Amazon, Google, Facebook, Twitter and hope for the best (PS: please read the legal outro at the end of this blog post before buying any stocks).

Easy answer. You cannot win (or lose) money fast enough by buying stocks.

Forex has a nice (or terrible, depending on which side of the coin you are) thing called leverage.

Leverage can be 1:10, 1:50, 1:100, 1:200, 1:1000 depending on how suicidal you are or how sketchy your broker is (don't worry, we'll talk about brokers in the next post). Let's see an example.

We want to place a trade of $1k. The logical thing is that in order to buy something that costs $1k, you need to have in your account $1k, right? Nope. You can always get a loan. Oooooooor: Enter leverage.

If we have a 1:100 leverage, we can place a trade and "control" $1k with just $10. For those that failed at math (don't be ashamed, I am one of you), $10 x 100 (leverage) = $1,000. That means you can trade big and win big! Actually 100 times more big! The catch is that you can actually go 100 times more small. Let's have another example.

I just got my bonus ($1k) and I want to play on Forex. Without any leverage, I can buy 1,000 Forex units that cost $1 (by the way, there are no Forex units but we'll talk about this later). How many units can I buy with 1:50 leverage?

If you answer if 100,000, you did something wrong. The answer is 50,000 ($1,000 x 50 leverage).

There you have it. One of the reasons that we are doing Forex, is because you can lose/win big. We'll get back to leverage when we start placing trades.

There are three more exciting reasons actually that are even more awesome (dare to say awesomer?).

  1. Forex (almost) never sleeps. The markets are open ALL DAY, six days per week. To be more exact, there is not one market but four and they are overlapping providing the "all day" effect.
  2. Forex is very volatile and there are tons of money moving around (more than 5 trillion per day). Crazy right?
  3. No fees on trades. You don't pay $10 per trade like you are doing with stocks. Here you pay the spread which is just a fraction of a cent (again, we'll talk about this in another post).

All these reasons (leverage, all-day, volatility, fees) make Forex the most exciting platform to build and deploy your algorithms.

Coming up next, Forex brokers. How not to be scammed before even writing a line of code.

If you have more feedback, ping me at jonromero or signup to the newsletter.

Legal outro. This is an engineering tutorial on how to build an algotrading platform for experimentation and FUN. Any suggestions here are not financial advices. If you lose any (or all) you money because you followed any trading advices or deployed this system in production, you cannot blame this random blog (and/or me). Enjoy at your own risk.

How to build your own algorithmic trading platform - Part One

That's it. Every week, I get at least 10 DMs on twitter asking on how to experiment with algotrading, Forex and portfolio analysis and I've decided that it's time to do something about it.

Update: Find the posts here.

So, I am planning to cover the basics of how to build your own trading platform, write your own strategies and go on vacations while electrons are making you money. Or zeroing your account. Either way, it is going to be fun!

The majority of the examples are going to be in Python even though there might be parts in Erlang and I'll try to keep it as easy as it can be.

Let's talk now about what the final product will look like.

We are building a system where you will be able to:

  1. Simulate your strategy (this is called backtesting)
  2. Execute your strategy without supervision
  3. Alert you via sms/email for trades and errors
  4. Be scalable and trivial to deploy new updates
  5. Being able to run even from your home (from a raspberrypi for example)

I assume this is going to be a total of 20 chapters, give or take.

That is a system similar to the one that I am running the last year and includes a UI, sms alerts, backtesting pipeline, continuous delivery and all the cool stuff that us geeks love.

All code will be on github and if everything goes well, I'll wrap it up in a book for everyone to enjoy.

I have like three chapters almost done, so if you want early access just ping me at - jonromero.

How to build your own algorithmic trading platform

These are all the post that have been written up until now. I may add/remove post as I get more feedback.

Get early access and free insights for Algotrading and Forex!

indicates required

Post One: Building your own algotrading platform

Post Two: What is Forex

Post Three: Placing your first trade

Post Four: Downloading historical Forex tick data and importing them in to Python

Post Five: Building a backtesting system in Python: or how I lost $3400 in two hours

Post Six: Building the simplest backtesting system in Python

Post Seven: Forex and Taxes for new traders

Post Eight: Machine Learning and Trading

Post Eight+: Trading Ethereum: Making 10% every 20 minutes

Post Nine: Forex Psychology

Post Ten: Deploy and automate

Post Eleven: Notifications, SMS and remote control

Post Twelve: Deep Learning and kicking (human) ass

Post Thirteen: Forex advanced

Port Fourteen: Into to HFT

Mapping and processing data in Clojure

I have to admit that I have a thing for DSLs. You can see it at music-as-data were notes and rhythm/beat is "mapped" to data and you can apply data transformations.

The same thing I want to do with data at-rest.

Here is a scenario: I have lots of data sitting as CSV on my hard-drive and I want to process them. Not query them. Process them.

What would be really interesting is to be able to define (dynamically) a schema like that:

(defschema "EURUSD" 
    (tokenizer #(.split % ":"))  
    ;; the mapping is done here
    (columns |time| |open| |high| |low| |close| |volume|))

Let me explain. First of all, a "tokenize" function. Each dataline is tokenized based on a function. Do you want regex? Something more complex? You are free to write anything you like. I really hate frameworks that you must write a complex regular expression or use a compicated system just to tokenize a line.

As you can imagine, tokenize returns a list of data that are mapped to "columns".

Now, the interesting stuff.

You can write scripts like the following:

(if (> |close| 1.45)
    (place-order :buy)
    (place-order :sell))

Thoughts?

ping me here -> JR

HybrisDB

I love Forex because:

  1. It has enormous amount of data (volume)
  2. These data are coming extremely fast (velocity)
  3. You need to consider multiple resources when you are building your strategy (variety)

My definition of BigData is that you have volume-velocity-variety information and you need to react on it right now (realtime). It is one of the main reasons why I don't like Hadoop (ok, the other is because I don't like Java:).

Forex is the best place if you want to start playing with BigData. You have (at least) one data channel, hitting you with data, you need to keep running algorithms on this stream (sometimes doing correlations up to a week) and you need be able to respond very fast. If a garbage collector kicks in or if you need to grab data from a database (even if this DB is in memory - long live Redis) then you will have issues.

That's the reason why most of the "trading" databases have all their data in the same memory space and have custom languages doing the analysis (like Kdb).

That was the inspiration for LDB.

Millions of data sources (mobile phones), hitting your database and calculating/updating for each one of the requests thousands of counters and running all sorts of algorithms. Per request. In realtime.

But let's face it. The vast majority of users/companies will never have millions (or even thousands) of requests hitting their servers. That's why I started a new opensource database with codename: HybrisDB.

HDB has the following characteristics:

  1. Simple to install (no moving parts)
  2. Simple to use (pre-defined dashboards)
  3. It will be perfect for the 99% of users/companies but not for the 1% like Facebook or Google (sacrificing Enterprise features)

The concept is to have a dashboard, to watch indicators going on/off and then (maybe) connect to a system to place an order.

Sounds like an interesting cool hobby project and I still try to decide between using Erlang or Clojure for this.

Ping me on twitter if you have any ideas!

Cheers!

JR

How to have 7 more days every month

One of my new year's resolution was "study one of your habits each month" and I decided to focus on what activities I am spending my online time. So, I installed RescueTime which is a very cool app that sits on the background and creates reports about which apps and sites you are using the most. And no, it is not sending that info to NSA (you are not that important!).

Even though I spent most of my time working, writing and communicating, I also spent around 2 hours on average on Facebook (gasp!). No biggie. What I hated was that for the majority of times I was scrolling/refreshing for new updates. Like opening your refrigirator every two minutes even though you know it is empty! Damn you habits!

So, spending 2 hours every day (sometimes more) on Facebook, means 60 hours per month or 7 working days. 7 freaking working days each month. 7 days, each day with 8 full hours, to sleep, go out, exercise, write a book, learn something new.

What I did was installing StayFocused, a free plugin for your browser that doesn't let you spent more than 10 minutes on specific sites everyday. And guess what. The "I am just using facebook to communicate" is such a lame excuse as I had no problem communicating with my friends, even by using facebook for 10 minutes.

Now, give it a shot and tell me what you built in your spare time!