Dukascopy Forex Tick Data

Abstract

In my exploration of world of big data and I became curious about tick data. Tick data is extremely granular and provides a great challenge for those looking to work on their optimization skills due to its size. Unfortunately, market data is almost always behind a pay wall or de-sampled to the point of uselessness. After discovering the Dukascopy api, I knew I wanted to make this data available for all in a more accessible format.

I have made this date set available via a torrent here :) UPDATE: The torrent is now also available via academictorrents.com

File Formats

The data was collected from https://www.dukascopy.com/ via a public api that allows for the download of tick data on the hour level. These files come in the form of a .bi5 file. Th

These files where decompressed, then merged into yearly CSV’s named in the following convention. “AUDCHF_tick_UTC+0_00_2011.csv” or ‘Pair_Resolution_Timezone_Year.csv’

These CSV’s are split into 3 categories “Majors”, “Crosses”, “Commodities”

Majors, Crosses, and Commodities have had their timestamps modified so that they are in the official UTC ISO standard. This was originally done for a Postgresql database that quickly became obsolesced. Any files that have been modified are appended with a “-Parse”. These timestamps have been modified in the following format.

Millisecond timestamps to UTC +00:00 time
[2017.01.01 22:37:08.014] ----> [2017-01-01T22:37:08.014+00:00]

User Resources

For those looking to use this data in a live context or update it frequently, I have included a number of tools for both Windows and Linux that will be useful.

Windows

The ~/dukascopy/resources/windows contains a third party tool written in java that can download and convert Dukascopy’s .bi5 files. I have also included the latest zstd binaries from Zstandard Github page.

Linux

Linux is my daily driver in 99% of cases, so I have developed all my scraping tools using linux only tools. In the ~/dukascopy/resources/linux folder you will find a number of shell script and pyhton3 files that I used to collect this data. There are quite a few files in this directory but I will cover the core ones below.

download-day.py

This file is used to download a single symbol for a single day and then convert and merge all 24 .bi5 files it into a single csv.

#!/usr/bin/env python3

import argparse
import dukascopy

parser = argparse.ArgumentParser(description="This file is used to download a single symbol for a single day and then convert and merge all 24 .bi5 files it into a single csv.")
parser.add_argument("-d", "--dir", default=False, help="Set a directory for downloads")
parser.add_argument("-dt", "--date", help="Set a date to download and convert")
parser.add_argument("-sym", "--symbol", help="Spesify symbol to download")
args = parser.parse_args()

if args.symbol and args.date:
    if args.dir is False:
        out_dir = '/tmp/dukascopy/working/' + args.symbol.upper() + '/' + str(args.date)
    else:
        out_dir = args.dir + '/' + str(args.date)

    dukascopy.fetch_day(args.symbol, args.date, out_dir)

download-year.py

This file is used to download a single symbol for a full year and then convert and merge all .bi5 files it into a single csv.

#!/usr/bin/env python3

import argparse
import dukascopy

parser = argparse.ArgumentParser(description="This file is used to download a single symbol for a full year and then convert and merge all .bi5 files it into a single csv.")
parser.add_argument("-y", "--year", help="Year to download and convert")
parser.add_argument("-sym", "--symbol", help="Spesify symbol to download")
parser.add_argument("-d", "--dir", default=False, help="Set a directory for downloads")
parser.add_argument("-o", "--out_dir", default=False, help="Set a directory for file output")
args = parser.parse_args()


if args.symbol and args.year:
    if args.dir is False:
        working_dir = '/tmp/dukascopy/working/' + args.symbol.upper()
    else:
        working_dir = args.dir

    if args.out_dir is False:
        out_dir = '/tmp/dukascopy/'
    else:
        out_dir = args.out_dir + '/' + str(args.year)

    dukascopy.download_sym(args.symbol, args.year, working_dir)
    dukascopy.merge_to_year(working_dir, out_dir, args.symbol, args.year)

dukascopy.py

This file contains all the core logic for downloading and converting data from dukascopy.

#!/usr/bin/env python3

import os
import lzma
import time
import struct
import shutil
import pandas as pd
from datetime import timedelta, date


def scan_dir(path):
    files = set()
    for x in os.scandir(path):
        if x.is_file():
            files.add(x.path)

    return files


def bi5_to_csv(date_ts, out_dir, files):
    print('Starting Coversion of All .bi5 Files...')

    sort = sorted(files)
    chunk_size = struct.calcsize('>3i2f')
    data = []

    for bi5 in sort:

        try:
            size = os.path.getsize(bi5)
        except (IOError, OSError):
            break

        if size > 0:
            with lzma.open(bi5) as f:
                while True:
                    chunk = f.read(chunk_size)
                    if chunk:
                        data.append(struct.unpack('>3i2f', chunk))
                    else:
                        break

        os.remove(bi5)

    if not data:
        print('All Downloaded Files Where Empty!')
        return 1

    df = pd.DataFrame(data)
    df.columns = ['UTC', 'AskPrice', 'BidPrice', 'AskVolume', 'BidVolume']
    df.AskPrice = df.AskPrice / 100000
    df.BidPrice = df.BidPrice / 100000
    df.UTC = pd.TimedeltaIndex(df.UTC, 'ms')
    df.UTC = df.UTC.astype(str)
    df.UTC = df.UTC.replace(regex=['0 days'], value=[str(date_ts)])
    df.UTC = df.UTC.str[:-3]

    df.to_csv(out_dir + '/daily.csv', index=False)
    print('Finished Converting Files!')
    return 0


def download(url, out_dir):
    try:
        os.system('wget -q ' + url + ' -P ' + out_dir)
    except Exception:
        pass


def fetch_day(pair, date, out_dir):
    baseurl = "http://datafeed.dukascopy.com/datafeed/"
    timestamps = {date + "/" + h for h in {str(n).zfill(2) for n in range(0, 24)}}

    if os.path.exists(out_dir):
        shutil.rmtree(out_dir)
    os.makedirs(out_dir)

    print('Downloading Source Files for ' + pair.upper() + ' on ' + date + '...')
    for t in timestamps:
        url = baseurl + pair + "/" + t + "h_ticks.bi5"
        download(url, out_dir)
        time.sleep(.1)

    if scan_dir(out_dir):
        print('Finished Download!')
    else:
        print('No Files Were Downloaded')
        shutil.rmtree(out_dir)
        return

    dt = date.split('/')
    date_ts = dt[0] + '.' + str(int(dt[1]) + 1).zfill(2) + '.' + dt[2]
    status = bi5_to_csv(date_ts, out_dir, scan_dir(out_dir))
    if status == 1:
        shutil.rmtree(out_dir)
        return
    os.rename(out_dir + '/daily.csv', os.path.dirname(out_dir) + '/' + date_ts + '.csv')
    shutil.rmtree(out_dir)


def create_date_list(year):
    dates = set()

    def daterange(date1, date2):
        for n in range(int((date2 - date1).days)+1):
            yield date1 + timedelta(n)

    start_dt = date(int(year), 1, 1)
    end_dt = date(int(year), 12, 31)

    for dt in daterange(start_dt, end_dt):
        tmp_dt = dt.strftime("%Y/%m/%d")
        tmp_dt = tmp_dt.split('/')
        fdate = str(str(tmp_dt[0]).zfill(2) + '/' + str(int(tmp_dt[1]) - 1).zfill(2) + '/' + str(tmp_dt[2]).zfill(2))
        dates.add(fdate)

    return sorted(dates)


def download_sym(ticker, year, working_dir):
    dl_dates = create_date_list(year)
    for d in dl_dates:
        fetch_day(ticker, d, working_dir + '/' + d)
        print('----------------------------------------------------')


def merge_to_year(working_dir, out_dir, ticker, year):
    files = sorted({os.path.join(dp, f) for dp, dn, fn in os.walk(os.path.expanduser(working_dir)) for f in fn})
    print('Starting to Merge ' + str(len(files)) + ' Files...')
    df = pd.DataFrame(columns=['UTC', 'AskPrice', 'BidPrice', 'AskVolume', 'BidVolume'])

    for f in files:
        tmp_df = pd.read_csv(f, error_bad_lines=False)
        df = df.append(tmp_df)

    fname = ticker.upper() + '_tick_UTC+0_00_' + year + '.csv'
    print('Done Merging Files!')
    print('Exporting Data Now...')
    df.to_csv(out_dir + '/' + fname, index=False)
    print('Exported Data to ' + fname)
    shutil.rmtree(working_dir)

utc-timestamp-convert.py

This tad slow but works well enough. It requires the pandas project and parses timestamps into the UTC ISO standard. This is useful for those looking to maintain the format of new files with the those in this repo, or those looking to use this in a SQL database.

#!/usr/bin/env python3

# utc_timestamp_convert.py
#
# Convert Dukascopy milisecond timestamps to UTC +00:00 time
# I.E. [2017.01.01 22:37:08.014] ----> [2017-01-01T22:37:08.014+00:00]
#
# 1. Add headers to each csv. UTCtime,AskPrice,BidPrice,AskVolume,BidVolume
# 2. Convert First 3 "." with "-" in column 1
# 3. Repalce space with "T" in column 1
# 4. Append "+00:00" on the end of column 1
# 5. Export New CSV
# 6. Remove Old CSV

import os
import argparse
import pandas as pd


def parse_tick_downloader_csv(source):
    if os.path.exists(source) is False:
        print(source + ' Was NOT Found!')
        return

    if source.endswith('.gz') or source.endswith('.zst') or source.endswith('.zip'):
        print('Cannot Convert Compressed Files!')
        return

    if source.endswith('-Parse.csv'):
        print(source + ' Was Already Parsed!')
        return

    export_path = source[:-4] + '-Parse.csv'
    print('Reading Source CSV ' + source.split('/')[-1] + '...')
    df = pd.read_csv(source, error_bad_lines=False)

    print('Parsing Source CSV...')
    df.columns = ['UTC', 'AskPrice', 'BidPrice', 'AskVolume', 'BidVolume']
    df.UTC = df.UTC + '+00:00'
    df[['UTC', 'timetemp']] = df['UTC'].str.split(' ', expand=True)
    df.UTC.replace('\.', '-', inplace=True, regex=True)
    df.UTC = df.UTC + 'T' + df.timetemp
    df.drop(['timetemp'], axis=1, inplace=True)

    print('Exporting Parsed CSV to ' + export_path.split('/')[-1] + '...')
    df.to_csv(export_path, index=False)
    del df
    print('Finished Parsing CSV!')
    print('Removing Source File!')
    os.remove(source)
    print('')


parser = argparse.ArgumentParser(description="Convert Dukascopy milisecond timestamps to UTC +00:00 time.")
parser.add_argument("-f", "--file", metavar=('/full/file/path'), help="Full file path of CSV to convert.")
args = parser.parse_args()

if args.file:
    parse_tick_downloader_csv(args.file)

Full Stats