Quantlane tech blog

ETL pipelines are at the core of every data-driven company. They process the data, transform it, extracting valuable information that is used on a day-to-day basis. At Quantlane we write these ETL pipelines using Python's asynchronous generators. Read on how we do it.

Python generators are a powerful extension of the language allowing users to generate large sequences one element at a time. Also allowing somewhat functional approach to programming in Python.

Static typing is a great tool for discovering bugs long before the code reaches production. Using it in new code is an obvious choice for us. However, introducing it in older codebases that predated mypy's maturity was not an easy journey for us. If you're in the same boat, read on about the strategy we successfully used to cover a lot of critical code with mypy checks.

Approaching Data as a Product is helping us in Quantlane, by guiding the crucial steps when identifying and addressing needs and data requirements, making the role of data engineer more collaborative. It leads to the creation of a better data structure.

Orange is not only our favorite color, but more importantly our everyday tool that helps us with code readability & consistency and takes off a little bit of mental strain when programming. It however all starts with the code style.

When developing applications that are long-running, processing big data, or just developing software for memory tight systems, memory usage can become a real issue. This is where memory profilers step in, to help you debug and optimize your code.

Quantlane, like every quantitative trading company, relies heavily on data. As we grow, we enter new markets, look further back in history and expand the data sets we use day to day. Join us on a journey through our data storage and warehousing.

Python's enum type is a useful building block for creating semantic types and constants in your programs. The problem? It's kind of slow. We were using enums a lot in our code, until we noticed that enum overhead takes up single digit percentages of our CPU time! Luckily, it only took a few hours to write a much faster implementation with almost the same functionality. Enter fastenum.

Optimization of an often-used tool leads to more frequent usage and thus to saving more time than previously anticipated. This claim is supported by an anecdotal story we experienced when gathering profiles from our production instances.

When creating a machine learning model, we have to choose a good model complexity if we want it to perform well. So, we tune model complexity and regularization to reach an optimal performance on out-of-sample data. What are the tradeoffs of such settings and how can we effectively reduce overfitting, ideally without the loss of generalization ability?

We use a messaging broker to run independent applications that can quickly integrate with our platform. Over the years we've learned a few useful tips and we want to share them with you. This article discusses the format of the data sent and the way we transport them from producers to consumers.

It is easy to train a machine learning model on historical data and make it seemingly profitable. This might however be just a result of overfit, a state where the model can perform flawlessly on data it trained on, but is much worse on any other data. Let's explore the details of this well-known phenomenon in the context of trading.

When building big applications, separation of concerns is a great way to keep things manageable. In messaging systems, the publish-subscribe pattern is often used to decouple data producers and data consumers. We went a step ahead and designed even the internals of our applications around this pattern.

Sequel to SSL Lessons learned: Part 1. In this article we uncover how different tools handle SSL and what are the lessons learned.

We take security seriously, that’s why even on local intranet almost all of our services use SSL. In this article we will explain how we use SSL with other tools transparently, with little overhead. Despite our prudence, surprises occur, like this year (2020) when a simple misconception taught us a lesson.

We make heavy use of asyncio and pytest, and thus are big fans of the pytest-asyncio plugin. It allows one to write async def test_... functions as well as asynchronous fixtures just as easily as normal synchronous code. Well, almost...

At Quantlane, we write a lot of asyncio applications. We often find it useful to schedule tasks to run concurrently in the event loop. Asyncio provides the create_task function for that. But what happens to exceptions raised in tasks?