https://ift.tt/3qUMbqX When starting out as a Data Engineer, I didn’t know what I didn’t know. This is a list of 6 things I learnt early on...
When starting out as a Data Engineer, I didn’t know what I didn’t know. This is a list of 6 things I learnt early on in my data career.
1. Keep communicating with stakeholders
For my first couple of data engineering projects, I scoped out the work, gathered the initial requirements and went off to crack on with it. I made good progress and felt like I was achieving something. Then I got stuck. I was sure I was finished, but thought there had to be more to do.
After delaying approaching the stakeholders for too long, I reached out to have a check in. Turns out I had done exactly what was needed initially, but requirements had changed so I needed to re-do a large chunk of the work. Had I kept touching base with them I would have saved myself a heap of time!
This taught me to always keep in contact with the stakeholders to stay aligned and be efficient with my time. Projects change and better solutions will inevitably be found during the process, so it’s best to keep communication lines open.
It’s also a good idea to regularly update the stakeholder when working on a project, instead of waiting until you’ve perfected your work, which is a common error new programmers make.
2. Learn to prioritise work
There’s no doubt that you’ll be an asset to your team and many people will want their piece of the pie (your time).
You will need to prioritise work in order of urgency and importance, and not be afraid to say no to people. It’s better to say that a piece of work is not possible yet, but you’ll let them know when there is time available, than to say that you can do the work but then under deliver.
The 80–20 principle comes into play here, where 80% of your results come from 20% of your efforts. You need to prioritise the parts of your work that bring the most value, and not spend lots of time perfecting that final 20% of a piece of work. This way you will get done what is important and spend most your time on this.
Once again, don’t be afraid to say no if needed.
3. Focus on the core skills
SQL and Python are the bread and butter of data engineering. Learn the core skills and then you can transfer them into the more complex tech that you consequently use.
Trying to learn everything all at once will get you nowhere, but having a solid foundation will get you far. No matter what new tools come out, you need to understand the fundamentals to fully unlock the value of the latest software.
The latest tech often changes rapidly as new, more optimised tools are released. So at the start of your career don’t focus on chasing the hottest new tool and instead build up a strong knowledge of topics that are transferable. It’s better to be able to pick up new tech quickly due to a solid foundation, than to know one specific tool which is likely not to be popular in a couple of years.
Recently, many companies are pushing for a simpler tech stack that works efficiently. For example, moving away from building complex Hadoop systems and instead using Databricks as a ‘one size fits all’ solution.
It goes without saying however, learn about the tech stack that you’re using at your work!
4. Stack Overflow, Stack Overflow, Stack Overflow
During my four months training to be a Data Engineer, some people tried to memorise parts of code rather than working on the ability to be able to apply code and find solutions.
Focusing on being able to code from memory is wrong. Even though this is pretty impressive, it’s just not practical. Each project you work on is likely to be vastly different from the last and you will need to use new techniques.
The key here is to develop critical thinking and problem solving skills. We are paid partly for our ability to adapt to problems and find a solution. If the problem could be solved by some code you’ve done before, then you’re not as valuable to the team.
Being able to use Google efficiently and knowing how to apply what you find to your particular problem is a key ability of a Data Engineer. Use Stack Overflow.
5. Get a mentor
An important part of working as a Data Engineer is being open to learning. This includes building both your technical and soft skills.
Getting a mentor allows you to learn from those that have already walked the path you are on. Many mishaps and mistakes can be prevented if you spend time learning from a mentor.
I personally think it’s good to have one mentor closely linked to your work (likely your manager) and then one that is more removed. This should be someone in the same sort of profession, but in a different company.
This solution gives you someone to critique your work, and develop you within the business you are working for. It also allows you to get advice about how to handle the company you’re working in, and to think about your career progression outside of your current company.
6. Enjoy the work
You spend a large chunk of your waking hours working, and you probably don’t want to spend it unhappy. Focus on noticing the good things about your work, working towards making it better and enjoying it.
Data engineering is really rewarding work. You struggle through difficult problems and can have some real impact for the company. The moment of relief when your code stops producing errors and for the first time runs to completion should be cherished.
You may be hidden behind a Data Scientist or Analyst but you are the backbone to the work. It’s always nice to solve a problem.
These are just a few tips I’ve picked up whilst working as a Data Engineer. I wish I knew these when I started and so hopefully this can help you too.
Some advice I’ve received and would like to pass on is don’t stress too much, be selective in your brainpower so as not to get burnt out, and talk to your team. Especially with working from home, you don’t want to be like this guy all the time:
I’m Rory Middleton — a Data Engineer with a passion for personal development, behavioural economics and gaming. Previously, I’ve worked as an alpaca shearer and a zipline instructor. I aim to write articles on a variety of topics and hope you subscribe to join me along the way.
6 lessons I learnt early as a Data Engineer was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
from Towards Data Science - Medium https://ift.tt/3ng4NPQ
via RiYo Analytics
No comments