Data Engineering Story by a Pragmatist
Prolog: As I look at the rainy Christmas morning through the window of my Brooklyn work-from-home-for-almost-two-years office, I am still a little lethargic and short on coffee from waking up early to watch James Webb space telescope lift off. My two cats watched it with me (or maybe it was them letting me watch? Cats…) Right now in my mind I am still playing back last night’s Christmas dinner…. My sons telling me (I have two, neither watched Webb’s liftoff but surely will want to hear all about it, in the most minute detail as they unpack their presents) — “Dad, you love what you do but stop telling us “when I retire, I’ll open a school in Costa Rica to teach kids how to write software programs…write a blog instead, and do it tomorrow!”
They make good points — while I love Costa Rica (6 visits and counting), I am not planning to retire any time soon — to the utter bemusement of my wife. But I do love to talk about what I do, and I don’t think of it as software programming. Those Costa Ricans kids — no chance to hear me talk then?
What exactly do I do for a living? In short, I work for a large US bank and design computer related “stuff”. To most, computer programming jobs are not considered your typical engineering field, and in most US schools likely to be aligned with math departments. But I think of it as an engineering job. Back in the day, I graduated with a “real” engineering degree from a university in the USSR, learning how to design nuclear plants, and one thing for certain — I could not have done it without first falling in love with the art of “software engineering”. I spent more time on our mainframe than I did drawing blueprints. I simulated hydro and thermodynamics scenarios and ran through thousands of variables in my first Pascal programs to find the most efficient curve of the turbine blades. My first paying job was designing computer simulators for refueling of nuclear reactors. I already knew back then that I code to solve engineering problems, not because I wanted to find a better algorithm.
Don’t get me wrong, there was a real computer science school at the university, and I took classes there. But one thing struck me — while I had real engineering challenges to solve and I used computers to do it, the compsci major folks just liked tweaking their algorithms to be “even more efficient”. Pragmatism was nowhere in sight. The concepts of “Time to market” or “Can this be used in real life?” haven’t been on their minds. Well, except maybe during the finals, but for all the different reasons.
My Masters thesis was the first in the history of the department with presentation boards holding block diagrams of C++ simulation of a nuclear reactor losing its cooling following an earthquake breaching its containment. That idea of “applied programming” stuck with me. And while life took me into a different direction away from designing nuclear power plants, that principle of pragmatism is what I “preach” in my daily job.
OK, so I think I am still an engineer. But just like doctors have specializations, so do engineers. I don’t calculate loads for a new bridge across Hudson river (that will never happen anyway), or thickness of concrete for a highway bed, or maximum rotational speed of an aircraft engine fan, or size of elevator shafts in skyscrapers. I deal with “data” problems.
My engineering specialty is “Data Engineer” (no relation to the famous character in Star Trek Next Generation TV show). While I hope this blog is read by my peers, other data engineers who obviously understand what this job entails, I want to offer how I personally understand it. To me, it is the amalgam of knowledge, technology, software, hardware aiming to solve challenges of capturing, storing, processing and analyzing the vastness of the information (a.k.a. “data”) surrounding us. I won’t bore you with the infamous “3Vs of Big Data” (feel free to google that, I hear there are even “7Vs” now), but the job essentially is battle all those Vs.
With 25 plus years on the job, I will attempt to provide my somewhat biased opinions deeply rooted in pragmatism rather than an academic approach. I’ll push against the conventional wisdom or marketing hype. I’ll advocate to bear “technical debt” to deliver results to the business, use “strings” instead of “integers” for star schema joins, and all such other nonsense.
In my first post-prolog blog I will be discussing the choices of databases to store your favorite data. Marketing departments need not apply, this is going to get real! See you soon!
Edit: Link to the second blog in the series Data Engineering by a Pragmatist — Tales of Choosing a Database | by Boris Serebrinskiy | Jan, 2022 | Medium
P.S You can also find me on LinkedIn https://www.linkedin.com/in/silverboris
