Data Engineer

The Company's Equal Opportunities policy applies equally to the recruitment process and must be complied with at every stage of the recruitment process. This means that prospective applicants should not be discriminated against either directly or indirectly on the grounds of race, nationality, ethnic origin, gender, marital status, sexual orientation, cultural or religious beliefs, disability and age. In addition, candidates should not be discriminated against based on pregnancy. Preparation of job descriptions, person specifications, advertisements, short-listing, interviewing and selection of applicants should reflect a commitment to achieving and maintaining equal opportunities within the workplace. Interviewers should only ask job-related questions.

Role and Responsibilities:

1. Programming and Scripting: Utilize programming languages such as Python, R, and Java to develop scripts for data processing tasks.

2. Database Management: Work with various database systems including SQL and NoSQL databases for storing and managing data effectively.

3. Container Management: Experience with container management frameworks like Docker, Kubernetes, and ECR to deploy and manage data processing applications efficiently.

4. Data Processing Technologies: Proficiency in scalable and distributed data processing technologies/frameworks such as Spark/PySpark, DBT, Kafka, and Hadoop to handle large volumes of data effectively.

5. SQL Skills: Strong SQL skills across a variety of relational data warehousing technologies, especially in cloud data warehousing platforms such as Amazon Redshift, Google BigQuery, and Snowflake.

6. Software Development Lifecycle (SDLC): Understanding of SDLC, particularly Agile methodologies, to ensure efficient project delivery.

7. Big Data Principles: Thorough understanding of big data principles, techniques, and best practices.

8. Data Pipeline Architecture: Design and develop scalable, reliable, and efficient data pipelines for collecting, ingesting, and processing diverse data from multiple sources.

9. Data Modeling and Warehousing: Create and optimize data models to enable seamless integration of data from various sources into centralized data warehouses or data lakes.

10. Data Governance and Security: Establish data governance frameworks and implement robust data security measures to ensure compliance with privacy regulations and protect sensitive information.

11. Performance Optimization: Continuously monitor and fine-tune data platform performance to meet SLAs and optimize resource utilization.

12. Technology Evaluation: Stay updated with industry trends, emerging technologies, and best practices in data engineering. Assess and recommend new tools and frameworks to improve data processing capabilities.

13. Cross-Functional Collaboration: Collaborate with data analysts, data scientists, and software engineers to understand their data requirements, offer data engineering support, and contribute to the development of data-driven products and solutions.

14. Data Quality and Monitoring: Implement robust data quality checks and monitoring systems to ensure the accuracy, consistency, and reliability of data.

15. Automation and Fault Tolerance: Possess an "automation-first" mindset when building solutions, incorporating self-healing and fault-tolerant methods to minimize manual intervention and downtime.

16. Documentation and Knowledge Sharing: Create and maintain comprehensive documentation of data engineering processes, best practices, and technical standards. Facilitate knowledge sharing sessions with the broader team.

Qualifications Requirements:

• 3 years of experience in data engineering or a related field.

• Strong proficiency in programming languages such as Python, R, and Java.

• Experience with database management systems, including SQL and NoSQL databases.

• Familiarity with container management frameworks like Docker, Kubernetes, and ECR.

• Experience with scalable and distributed data processing technologies/frameworks such as Spark/PySpark, DBT, Kafka, and Hadoop.

• Solid understanding of big data principles, techniques, and best practices.

• Experience with cloud data technology stacks such as Azure, AWS, or GCP is preferred.

• Excellent communication and collaboration skills.

• Ability to work effectively in a fast-paced, dynamic environment.

Education Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field.

Advanced degree preferred.