Category Archives: Big Data

Intra-day data from Quandl and a new tick database in town – party time !

(with permission from Quandl)

Quandl will soon be offering intra-day data (1 min bars). Rock on !

I was kindly given¬†some data to test out (see below). I can’t say much more than this but keep an eye out for an official announcement soon ūüôā

With both QuantGo and Quandl offering reasonably priced intra-day data, smaller trading shops have never had it so good.

I’ve been involved in the integration (i.e. messaging) of trading systems since my days working at IBM nearly 10 years ago. My current research is centred around the development of predictive models for trading systems, looking at different data sources to feed into my models. In today’s world, data is streaming from all directions and so the successful integration of data is vitally important.

Kerf is a new columnar tick database and time-series language designed especially for large volumes of numeric data. It is written in C and natively speaks JSON and SQL. Kerf has support for both real-time and historical databases and it provides data loaders for all the common Quandl formats, so you can run queries against Quandl data right away.

I’ve taken Kerf for a quick spin and via¬†it’s Foreign Function Interface (FFI), which makes it possible to call out to C libraries from Kerf and call into Kerf from C, I can query the data using R (very easily). With¬†ZeroMQ for my messaging, it could stack up to be a seriously good trading or analytics system.

I really like what I’ve seen so far. Is this a true competitor to Kx‘s kdb+ database ? Possibly. You should check it out for sure !

repl-header

Here is the output from a simple request/response using S&P 500 One Minute Bars from Quandl. A C application acts as the handler for Kerf requests. It calls into Kerf and responds back to the client with the data. A R script requests the closing prices for AAPL, but just as easily C++, Java or Python could be used. The messages are in JSON format and these are then transformed to xts in R. Messaging uses ZeroMQ.

R (rzmq) -> C (libzmq) -> kerf (FFI) -> C (libzmq) -> R (rzmq)

I’ve decided to further develop the R API and to start integrating my real-time Interactive Brokers feed to Kerf (e.g. Java API and ZeroMq). If anyone is interested in this, please get in touch.

kerf_handler_jpg

rclient_kerf_jpg