Data Engineer

Role and Responsibilities:

1. Programming and Scripting: Utilize programming languages such as Python, R, and Java to develop scripts for data processing tasks.
2. Database Management: Work with various database systems including SQL and NoSQL databases for storing and managing data effectively.
3. Container Management: Experience with container management frameworks like Docker, Kubernetes, and ECR to deploy and manage data processing applications efficiently.
4. Data Processing Technologies: Proficiency in scalable and distributed data processing technologies/frameworks such as Spark/PySpark, DBT, Kafka, and Hadoop to handle large volumes of data effectively.
5. SQL Skills: Strong SQL skills across a variety of relational data warehousing technologies, especially in cloud data warehousing platforms such as Amazon Redshift, Google BigQuery, and Snowflake.
6. Software Development Lifecycle (SDLC): Understanding of SDLC, particularly Agile methodologies, to ensure efficient project delivery.
7. Big Data Principles: Thorough understanding of big data principles, techniques, and best practices.
8. Data Pipeline Architecture: Design and develop scalable, reliable, and efficient data pipelines for collecting, ingesting, and processing diverse data from multiple sources.
9. Data Modeling and Warehousing: Create and optimize data models to enable seamless integration of data from various sources into centralized data warehouses or data lakes.
10. Data Governance and Security: Establish data governance frameworks and implement robust data security measures to ensure compliance with privacy regulations and protect sensitive information.
11. Performance Optimization: Continuously monitor and fine-tune data platform performance to meet SLAs and optimize resource utilization.
12. Technology Evaluation: Stay updated with industry trends, emerging technologies, and best practices in data engineering. Assess and recommend new tools and frameworks to improve data processing capabilities.
13. Cross-Functional Collaboration: Collaborate with data analysts, data scientists, and software engineers to understand their data requirements, offer data engineering support, and contribute to the development of data-driven products and solutions.
14. Data Quality and Monitoring: Implement robust data quality checks and monitoring systems to ensure the accuracy, consistency, and reliability of data.
15. Automation and Fault Tolerance: Possess an "automation-first" mindset when building solutions, incorporating self-healing and fault-tolerant methods to minimize manual intervention and downtime.
16. Documentation and Knowledge Sharing: Create and maintain comprehensive documentation of data engineering processes, best practices, and technical standards. Facilitate knowledge sharing sessions with the broader team.

Qualifications Requirements:

• 3 years of experience in data engineering or a related field.
• Strong proficiency in programming languages such as Python, R, and Java.
• Experience with database management systems, including SQL and NoSQL databases.
• Familiarity with container management frameworks like Docker, Kubernetes, and ECR.
• Experience with scalable and distributed data processing technologies/frameworks such as Spark/PySpark, DBT, Kafka, and Hadoop.
• Solid understanding of big data principles, techniques, and best practices.
• Experience with cloud data technology stacks such as Azure, AWS, or GCP is preferred.
• Excellent communication and collaboration skills.
• Ability to work effectively in a fast-paced, dynamic environment.

Education Requirements:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Advanced degree preferred.

Entry level

Full-time

Engineering, Analyst, and Other

Telecommunications, Construction, and Services for Renewable Energy