Hello everybody. I'm Olimpiu Pop, technologist and probably I can call myself a tech journalist as well. And I have in front of me Andrew Lamb, another technologist that knows at least two very complicated things on my side.

And those are databases or data systems, probably to be more correct, and the other one is C language.

Well, Andrew, I think it's better for you to introduce yourself, please.

>> Hello. Yeah, thank you. I'm very excited to be here.

I am a staff engineer currently at FlexData but my and my in that context I work a lot on databases, specifically the analytic database stack and I'm sure a lot of stuff we'll talk about later.

My background was basically as a low-level systems programmer, so that means like databases and compilers kind of stuff. I worked for many years at a company called Vertica where I worked at Oracle.

Vertica was one of the sort of first wave of commercializations of these sort of shared-nothing distributed database architectures.

And then I've worked in various other sort of engineering and data platform teams. And now I work on FlexData and the core engine that drives that, including the SQL engine, which is built on top of the big pile of open source, including Apache DataFusion, Apache Arrow, where I'm the PMC.

I'm on the PMC for both of those projects and I am down to the guts sort of on a day-to-day basis.

So I really love that stuff. >> Thank you. Thank you for loving it.

because then we don't have to love it and just play with it. On top of that, we had other conversations and one of them was together with Felix, which unfortunately cannot join us.

And what we spoke is that on other parts of the programming universe, we can take bits and pieces.

We have multiple frameworks, we have libraries that we can just place them together and have and have a final system. But until not long ago, that wasn't the case for data systems.

And that sort of was the conversation that we started as a trio, that Kafka played a role as one of the parts, at least for ingestion systems for data systems.

And then we had different engines like RocksDB or DB.

But when you rebuilt InfluxDB with its third version, you made a couple of changes.

One of them was using another programming language, and that's Rust, as we spoke about. And the other one, you coined the term FDAP.

Can you please elaborate about what's FDAP, or should we add the I as well or not just yet?

Yeah. Yeah. I, I for Iceberg probably. Yeah. So the FDAP stack, I want to say I wrote the blog post, but like really it was the CEO and CTO of Influx that came up with that, right?

So Paul Dix is the CTO of Influx, and he saw early on that perhaps partly because he wanted to go write yet another version of the database, right?

And the CEO was like, "How are we ever going to possibly afford that?

" Because it's a, it's a big expensive undertaking, which they knew firsthand because they'd just gone through it, right?

And now they want to sort of re-rework everything from the ground up.

Paul's vision at that point in the pitch was, we don't have to rebuild everything from the ground up because at this by this point, uh, people have done sort of built these analytic systems and done it enough that the common patterns are emerging.

And so at that point, he identified the technologies specifically Flight Arrow, Apache Arrow Flight, which is part of Arrow, DataFusion, and Parquet as technologies that you could build a whole new sort of time series database on top of that without having to reinvent everything, right?

And Paul was definitely on the earlier side of that trend, but the idea was rather than have to build a database from scratch, which by the way, I did it vertically, I saw it was like a five-year endeavor with like 20 engineers, right?

Like it's a it's a major investment, like good engineers too, like PhDs database things.

But since that pattern has now been implemented so many times, the thesis was we'd be able to we understood where the boundaries were and we can really focus people's efforts together to not have to reinvent the same thing over and over again.

>> Okay. So because you made a mistake and told me that you already have the experience of building everything from scratch, let's take a step backwards then and just maybe think about it.

What are the conceptual pieces that usually will come and we'll just say that these are the building blocks for a normal data system.

And then, yeah, go back and map them and see how it actually moves from, I don't know, ingestion via the memory model and then on the file system and then back again.

So, yeah, yeah, yeah, you know, and so I'm going to talk from the perspective of a database implementer, right?

Like someone who's building one of these components that you then build an actual data platform on top of, right? So I'm going to talk about how you would implement a database and depending on you as a user, when you're actually trying to assemble this into some either data platform or higher level application, there's often you would use pre-assembled, um, components, right?

So I'm going to talk about how you would implement stuff like InfluxDB, not how you application on top of, or likewise with Cascade, whatever.

And actually, some, you know, the line there is always a little blurry and I think it's getting more blurry as we go in the sense that it's easier to decide where exactly where you want to work on.

Maybe we'll talk about that later in the call.

But at a high level, like when you're going to build one of these high-performance analytic systems, sort of the best practice in the last probably 20 years came out of the research world is a very simple concept, which is you store it in columns.

Right? Like which basically means for this is mostly in the context of storing tabular data, right?

But it the same concepts apply to other forms.

So storing relational tables, you can either, like, you can either store all the row, uh, the values in each row next to each other, if you think about it, or you can store all the values in each column next to each other in memory.

And it turns out for analytic processing, for a lot of reasons, doing the column store is a much better choice, specifically because it can take advantage of modern hardware like vectorized execution, right, which does phenomenally well when you just have the same uh operations apply to a lot of different data items versus row-oriented, you kind of like, I would have to be continually changing what application.

So there there's basically a shift started probably 20, 30 years ago, starting in academia and moving into like really multi-layer technology databases.

And so these days, if you're going to go build a new database system, to your point, you would typically need to have some way to get the data from your client and you have to represent it in memory somehow.

And in particular, you have to be able to do operations on it quickly. The other thing you need to do is, of course, so that's what Apache Arrow is basically designed for.

And I wouldn't say Apache Arrow is really like earth-shattering in terms of its technical content, right?

Like those ideas have been around. There's really not that many ways to represent an array of 64-bit integers right in memory.

But what's important is that everyone agrees on how, you know, how it looks, how to pass the metadata around, how you like represent nulls, a null mask or ability mask doesn't really matter, but it matters a lot that everyone agrees on what the what the state mask means.

So having like a standardized what I think Arrow's real value was it standardized best practice across sort of column representations.

>> Okay. So let me let me just stop you here to just translate it for a 5-year-old and see if I got it correctly.

So no worries, no worries. That's why we we're here. So what you're saying is that in the last 20 to 30 years we had a transition from the normal way how we thought about databases to looking especially in terms of analytics to just look at them in terms of columnar data because that made sense also in terms of extractions but also in the way how we can store it and how the new hardware behaves lately as it uh it would be much much more efficient.

Right. So after that what you mentioned is that Apache Arrow takes care of that memory model nothing earth-shattering but it just takes care of a couple of it depends because we love to use it depends on multiple levels but from that point of view what it actually does it just comes sets a standard and then all the all the things that you normally want to think what would be the the sequence of data data types and probably the way how we store it, how we represent null or other other aspects are now standard and then we can build on top of this attraction and that will be easy to do.

So with each iteration, it will just improve and then everybody will use the same standard.

Yeah. Yeah. Yeah. Yeah. That's that's a very good summary.

Great. Thank you, Andrew. And I just wanted to, like, maybe I can like take a step back just a little higher to try to describe why the transition to column stores happened, right?

Like at a really high level, like philosophically, when computers and databases were first coming around in the '60s and '70s, there wasn't that much data in them, right?

Like, not a whole lot of the world had been digitized.

We we didn't weren't creating massive amounts of digital information at that point.

And so the speed of getting data in and getting data out was was typically happening at, you know, individual like was based on human actions, right?

Like you did something to your bank account, and so that fundamentally sort of, you know, like, that's as human scale both reading and writing.

Then what happened in the, you know, '90s, 2000s, is an increasing amount of both our lives and also all the computing infrastructure became digitized.

And so it generated huge, huge amounts of data, right? That that didn't just come from human scale systems, it came from machine scale systems.

So the volumes of data were much, much higher and we needed new techniques to keep up with the increasing data lines.

So that's around the time when columnar technology really took off because we needed better ways to basically handle the ever-increasing data lines.

And now, of course, there's some theories we talk about to move into AI, but that's that's maybe a different, uh, a different conversation.

So anyway, so Arrow is the in-memory representation for any database. You need a way to actually manipulate the data once you get in there, right?

To actually compute the calculations people ask for. And you kind of touched on it, but I wanted to highlight again.

Arrow really has two things that are going for it. One is that it's standard. So, so what that means is you can pass data without having to translate the format from things like data frames to a database to another database to, you know, like basically it lets you easily pass data between systems without paying the serialization cost of having to translate it back and forth to different formats, which is actually I think the original reason why Arrow was was founded.

But I think a hugely important other aspect that we talked about that I that that I think is really critical for system design is you now you when you're designing this no longer have to go through the trouble of like deciding exactly how you're going to represent timestamps and where the time zones come in and how you're going to convert it and what kind of date you have, like a bunch of this stuff is not impossible to solve, but I'll tell you from experience, it takes a huge amount of time and effort to actually work out all the details.

And so the fact that someone else has done that already just basically accelerates your development cycle a lot.

There's a whole bunch of infrastructural work you just don't have to do because you can just take advantage of what's already there.

And I think that that's pretty huge. >> Okay. So this would be the things that we'll have in the memory.

But usually we call them persistent because they stay somewhere where we can find them afterwards.

>> That's right. >> And that's probably right.

How do they interconnect? How do you make the connection between the two of them?

>> Yeah. So they're different technologies.

They're both columnar technology. I would say Parquet is designed for persisting data. In particular, it has a bunch of encodings and compression so that the data volume on disk that you have to transfer the network is significantly lower.

So in order to process things in memory, right, you have a representation that's convenient to process, but if you want to save that for long-term archival storage or share it with people or store on S3 or whatever, you end up with Parquet.

So it's a columnar file format and it's designed for storing large volumes of data efficiently on disk.

And I think it has two has a couple of benefits over the alternative, right? Like, well, one option is you could build your own whole file format, which you have ultimate freedom, but it's very expensive.

Another classic alternative was CSV files, right?

You just dump a bunch of CSV files. There's a couple of problems with CSV files or JSON files.

Similarly, they're not particularly well compressed. So, they're very space inefficient, both for like your storage cost, but also that the amount of time it takes to transfer.

If you want to read a CSV, it actually takes a lot of effort to extract the data out of the text form into the form that you want to process, right?

Because you have to like parse strings into numbers, which is a very, very expensive operation. And the final thing is it's actually they they don't have type information attached to them, right?

So, so you basically have to when you get a CSV file, you have to figure out is this column full of dates, or maybe it just kind of looks like dates, or maybe they're UUIDs or like so and every system sort of has to do that schema interpretation, schema inference itself.

And of course, they're a little incompatible and so there ends up being a bunch of pain when you when you have to get systems to go.

So, Parquet has its own type system. It's got a very efficient encoding and compression scheme in there.

And now it's a standard also sort of governed by Apache Software Foundation. So you have lots of vendors that support the the file format.

>> Sorry to be tricky, but CSV and JSON are not exactly earth chattering again.

So comparing them to Parquet, I I cannot say that I'm wow, but maybe we can underline one other thing that Parquet has that allows it to to be that that important in the new data data landscape.

What I think it's probably very important is the fact that a lot of other projects have already adopted it.

So that means that most probably you can use it also in terms of interconnectivity across the board in multiple data projects.

Did that make a difference for you while implementing it? What what would be a couple of benefits as you as a provider of databases have with that?

>> That's right. Yeah. So there's there's two primary benefits to to Parquet.

The first is that it's compatible, that you can write a Parquet file and read it with other tools.

It both helps if you want to transfer data, you know, like how do your customers migrate their data in bulk either into or out of your system.

It's very helpful for that, especially so if you don't have to rewrite it internally, you just give them the internal Parquet files was was the theory.

The other major benefit, which goes back to the original discussion, is the amount of effort that's gone into these open source Parquet readers and writers is actually quite high.

And so while if you had your own custom file format, which is what you would typically do actually in a database to your point, right?

Building Modern Databases with the FDAP Stack • Andrew Lamb & Olimpiu Pop • GOTO 2025

GOTO gathers the brightest minds in the software community to help developers tackle projects today, plan for tomorrow and create a better future.

Learn from thought leaders and innovators with top-rated videos released daily, and at our year-round conferences, masterclasses and meetups.

Get started: https://gotopia.tech

GOTO is produced by Trifork, a leading full-service supplier of high-quality, custom-built applications and end-to-end solutions.

GOTO YouTube Channel Code of Conduct: https://blog.gotocon.com/yt-coc


Building Modern Databases with the FDAP Stack • Andrew Lamb & Olimpiu Pop • GOTO 2025

Scripts