When I was first starting work in the data space, we had an architect at the large company that employed me. This individual reviewed new application proposals, decided on platforms and technologies, and mostly went to a lot of conferences. I think this was a smart person, but they never seemed to do much work, mostly just talk about technology and ideas. As a young person in a career, I both coveted the job (good pay and hours), and thought it ought to be eliminated.
These days I think architecture is important, at least at a coordinating level. I don't know that it needs to be a full time job, but we need a person not too distracted by daily work examining how we build our systems. Perhaps this is the type of position that someone with a lot of experience that wants to work part time ought to hold. If you have such a position, call me in about 15 years.
In this blog from Stitch Fix, who gets way too much money from my family, an engineer talks about the data science world, with scientists, data engineers, and infrastructure engineers. This is written as an ideal way to structure the data science/analysis part of your company, though not necessarily what is happening at Stitch Fix. The idea here is that the work of writing ETL pipelines isn't something that anyone really wants to do, at least not as a full time job.
I saw a note on Twitter, asking what Andy Leonard thinks of this. If you don't know Andy, he's one of the #sqlfamily data professionals that focuses on ETL work for clients. Andy disagrees with the piece, noting that he likes writing ETL flows. I hope so, because Andy has made a living producing data import and export routines for clients, and he's written a number of Stairways at SQLServerCentral (SSIS, Biml, ADF). Andy really likes this stuff, though perhaps he's an outlier.
What I think is that most people like a little variety at work. Even if you are an expert at a very niche skill, I'm sure you look for ways to express some creativity and change your job just a bit on a regular basis. Most of the very talented craftspeople I know, whether in technology, construction, medicine, etc., like a bit of variation in routine. Even as a developer or DBA, I've enjoyed breaking up my job on occasion to handle some other task, from firewalls to email to project management.
Where I see people not enjoying a change is when they are focused on a solution and their skills are not up to the task, or perhaps their skills are rusty. I think SSIS is a great platform at times, but it's also not very intuitive to use and easy to get frustrated with the interface when trying to set up some data movement. I suspect that many data engineers that really want to focus on problem solving do hate writing ETL pipelines, if their focus and interest is in some other area.
If you don't enjoy most of your daily work, then perhaps you need to change. While the article has good points about engineers improving and providing ways to efficiently move data in a scalable manner, I'm not sure I want them trying to produce too many "Lego blocks" for data movement. Rather, I think they ought to be building those pipelines with patterns and best practices, providing ways for others to consume their work, or even adapt it where needed. Just avoid building bespoke work for every new request. That's tedious, it certainly might make an engineer dread their work, and it's a poor way of working.
All of us in technology need to learn to be the thinkers. We need to find ways to improve processes and adopt better ways of working as computers and technology become more and more embedded in our organizations. This isn't chasing the shiny new technologies every year, but it is ensuring that you are not working the same way year after year if there are better techniques out there.
Steve Jones from SQLServerCentral.comJoin the debate, and respond to today's editorial on the forums
