https://ift.tt/31bXqjI Purist vs. Pragmatist — Why We Can Both Agree on Automation We all strive for solutions that are sustainable yet we...
Purist vs. Pragmatist — Why We Can Both Agree on Automation
We all strive for solutions that are sustainable yet we sit on spaghetti code, with our future inevitable … spaghetti code in the cloud.
We have met both personas. The purist who loves theorizing on the “future proof data pipelines”, “modern data architecture”, “best practices” all neatly put into Death by PowerPoint. He is a real stickler for the rules. Let’s call him stickler Steve. You wait in anticipation for the next architecture meeting where you will be thrilled and amazed by what we are going to implement next. And on the other side, the hero, the doer, the guy (or gal) who runs jobs at 2 am to get data into production. Let’s call her pragmatic Prudence. It is a much different (and scarier) slideshow that she can compile. Put what she knows into PowerPoint, flick through it and you might get something that resembles a Wes Craven movie.
These two don’t speak fondly of each other, at least not in private.
Prudence blames Steve for not being able to implement pragmatic solutions and Steve blames Prudence for the horrible unsustainable mess that has been created.
I have often found myself being the conduit between these two personas. As a consultant I have worked with many Steves talking about sustainability and well architected solutions, but I have also sat in the trenches with Prudence, accountable for delivery, getting the solution into production. There seems to be this twilight zone somewhere between halfway through dev and production where we bring out the duct tape. The “best practice” just doesn’t get the job done. The conversation morphs slowly from “we need to stick to good conceptual designs” into “we need to get this into production by next week or business will pull the plug”. Prior to this moment hammer and nails are used to carefully construct the solution that Steve has put forward. After this inflection point is when Prudence brings out her trusted companion, Mr. Duct Tape.
So, how do we all pull in the same direction? I really appreciated an ex-colleagues’ article on how data architects (the purists) can get it wrong and how being accountable for delivery is that critical prerequisite. ¹ This is what gets the two personas sitting around the same table. Making sure the purist is accountable for getting the well architected solution into production, and thus adding a bit of pragmatism to Steve’s vocabulary.
Now that we are sitting around the same table, let’s start talking the same language. I will focus on one of the aspects which I believe both will agree on and appreciate … automation.
Steve will agree that building patterns, finding repeatable components and automating the repeatable components definitely fits within the purist’s mindset. Prudence knows by experience that she is often repeating certain tests, certain processes ad infinitum in order to refine, revise and get the final solution ready.
First, we need to answer the core question — what can be automated? And if the answer is “nothing, because everything is so bespoke” I really hope you get the same shivers as I do. Just flick through those Wes Craven slides again to make sure you are thoroughly frightened.
Only repeatable tasks can be automated. I advocate for an environment where patterns are constantly found, templatized, or modularized and improved. Continuous improvement. It is the purist role to drive this culture, and it is the pragmatist role to test any new patterns to make sure it is feasible. Just as being accountable for delivery will allow Steve to be more hands on, so a bit of overall accountability will force Prudence to be prudent and become an advocate for automation.
At this point Steve is convinced. The only thing we still need to convince Prudence of is that even though everything is bespoke at the moment, there are repeatable patterns hiding in there. Prudence will require hard facts and examples, something that I will not elaborate on now but probably a good topic for a follow up.
The antagonists to automation will often share this picture. They are not entirely wrong. That is the reality of memes. They are only funny because there is some truth to it.
Automation requires a lot of discipline. If you don’t have the discipline you will always fall back to old ways once time pressures appear. You must hate inefficiencies and doing repetitive tasks more than you hate not being the hero.
Pause and take a moment of introspection — ask yourself do I hate inefficiencies? The top of mind answer I suppose will always be “yeah sure” but really think about it. For myself, I would rephrase the question like this: Do I continuously find myself in a situation saying “I really think we solved this problem a couple of months ago, why are we thinking about this again?” That feeling bothers me to the core, it agitates, irritates and any other “ates” me. Those that would have worked with me know I become extremely pedantic and almost robotic at this stage. Must. Solve. Inefficiency. Now.
Setting up the groundwork can take longer than expected and the identification and templatization of repeatable patterns will take longer than manually doing it the first time. Maybe even the 2nd, maybe even the 3rd, and maybe … ok ok, you get my point. Somewhere along the line automation will surpass manual, but only if you have the discipline to stick to the process.
Steve definitely hates inefficiencies, and I believe Prudence does too, she might just enjoy being a hero too much. (Topic for a different day) . Once we are aligned in our hatred toward inefficiencies, we need to believe that there are enough inefficiencies that we can solve, hidden patterns (or blatantly obvious ones) that we can extract, make repeatable and finally automate. Don’t be like the guys in the picture below, make time now in order to free up time later.
On the other hand, automation is inevitable. I believe there is enough momentum created by the demand from both business and IT. Data engineers are burnt out and calling for a better approach. ² Instead of trying to fight it we should strive towards it and how to make it better.
In my conversations with clients, the terms Data Ops, Dev Ops, CI/CD comes up every time without fail. Automation is a key pillar to Data Ops and Dev Ops. I can’t always ask my clients whether it is something they really need or something their Steve suggested, but the demand is clear. Tools need to be able to integrate with Jenkins, GitHub etc. Everything should be scriptable. We want the nice pretty front-ends to design in, but we want the command line interfaces to operate and deploy.
Our methods and our toolkits need to complement our culture, not the other way around. We need a disciplined yet pragmatic culture that is complimented by our methods and software. Let people do what we are good at— coming up with creative solutions to interesting problems. Let’s make computers do what they are good at — repetition and parallel processing.
Spaghetti code does not need to be our destiny. Let’s have the discipline and purist naivety to start untangling it in order to find hidden patterns. Then be pragmatic to iterate and continuously improve knowing that 100% is not the goal.
At the end of the day both Steve and Prudence have families that they would like to spend more time with. Automation will help make this a reality. That’s the second thing they can agree on.
1. Thomas, J., 2021. The rise and fall of data architecture. [online] ITWeb. Available at: <https://www.itweb.co.za/content/o1Jr5Mx9AxPqKdWL> [Accessed 24 November 2021].
2.WIRE, B., 2021. Data Engineers Are Burned Out and Calling for DataOps. [online] Businesswire.com. Available at: <https://www.businesswire.com/news/home/20211019005858/en/Data-Engineers-Are-Burned-Out-and-Calling-for-DataOps> [Accessed 24 November 2021].
Purist v Pragmatist — why we can both agree on automation was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
from Towards Data Science - Medium https://ift.tt/32OaoFn
via RiYo Analytics
ليست هناك تعليقات