Intra-day data from Quandl and a new tick database in town – party time !

(with permission from Quandl)

Quandl will soon be offering intra-day data (1 min bars). Rock on !

I was kindly given some data to test out (see below). I can’t say much more than this but keep an eye out for an official announcement soon 🙂

With both QuantGo and Quandl offering reasonably priced intra-day data, smaller trading shops have never had it so good.

I’ve been involved in the integration (i.e. messaging) of trading systems since my days working at IBM nearly 10 years ago. My current research is centred around the development of predictive models for trading systems, looking at different data sources to feed into my models. In today’s world, data is streaming from all directions and so the successful integration of data is vitally important.

Kerf is a new columnar tick database and time-series language designed especially for large volumes of numeric data. It is written in C and natively speaks JSON and SQL. Kerf has support for both real-time and historical databases and it provides data loaders for all the common Quandl formats, so you can run queries against Quandl data right away.

I’ve taken Kerf for a quick spin and via it’s Foreign Function Interface (FFI), which makes it possible to call out to C libraries from Kerf and call into Kerf from C, I can query the data using R (very easily). With ZeroMQ for my messaging, it could stack up to be a seriously good trading or analytics system.

I really like what I’ve seen so far. Is this a true competitor to Kx‘s kdb+ database ? Possibly. You should check it out for sure !


Here is the output from a simple request/response using S&P 500 One Minute Bars from Quandl. A C application acts as the handler for Kerf requests. It calls into Kerf and responds back to the client with the data. A R script requests the closing prices for AAPL, but just as easily C++, Java or Python could be used. The messages are in JSON format and these are then transformed to xts in R. Messaging uses ZeroMQ.

R (rzmq) -> C (libzmq) -> kerf (FFI) -> C (libzmq) -> R (rzmq)

I’ve decided to further develop the R API and to start integrating my real-time Interactive Brokers feed to Kerf (e.g. Java API and ZeroMq). If anyone is interested in this, please get in touch.




7 thoughts on “Intra-day data from Quandl and a new tick database in town – party time !”

  1. Interesting read. kdb+ is certainly the #1 time-series db but thanks for mentioning Kerf – we’ll be checking it out.

      1. PortfolioAnalytics does not offer any meaningful intraday analysis – it does not model microstructure nose, jumps, outliers, fat tails, fractality or quick changes in price dynamics. They use a version of RiskMetrics methodology built in the 90’s for EOD data. Thus, intraday volatility & risk estimates would be severely biased, same goes for other metrics like Sharpe ratio. I would not use PortfolioAnalytics except for EOD/low-frequency portfolio analysis. PortfolioEffect model pipeline handles those microstructure effects automatically, producing tick-by-tick resolution of position and portfolio metrics. Also, tick-level market data access is built into the R/Matlab/Python packages, so that you could request historical data for most US symbols by their ticker.

  2. Nice ! 🙂
    I’ve just created an account and have started looking at the tutorial “Introduction to Intraday Portfolio Metrics”.

    Thanks for commenting Aleksey.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s