Senior Data Engineer

We are looking for an experienced data engineer to join our team. As a leading provider of AI strategy, talent, and education services, we strive to bring global expertise in AI to transform companies worldwide.

This remote full-time position requires designing, building, testing, optimizing, and maintaining data infrastructure and code for integration, storage, processing, pipelines, and analytics from ingestion to consumption. Implementing data flow controls and ensuring high data quality and accessibility are key responsibilities.

We seek a senior person who can quickly ramp up, contribute immediately, and lead data & analytics work with minimal oversight. This includes backlog definition, architecture decisions, and technical leadership of the team.

• Design highly scalable and efficient data architectures, defining standards and best practices for data management independently.
  
• Ensure data system scalability, reliability, quality, and performance.
  
• Mentor junior/mid-level data engineers.
  
• Collaborate with Product, Engineering, Data Scientists, and Analysts to understand data requirements and develop data solutions.
  
• Evaluate and implement new technologies and tools to improve data integration, processing, and analysis.
  
• Design architecture, observability, and testing strategies, and build reliable infrastructure and data pipelines.
  
• Own storage layer, data management tasks, including schema design, indexing, and performance tuning.
  
• Address and resolve complex data engineering issues, incidents, and bottlenecks in SQL queries and database operations.
  
• Conduct discovery on existing data infrastructure and proposed architecture.
  
• Evaluate and implement cutting-edge technologies and methodologies, continuously learning and expanding skills in data engineering and cloud platforms.
  
• Evaluate, design, and implement data governance solutions: cataloging, lineage, quality, and data governance frameworks suitable for modern analytics solutions.
  
• Define and document data engineering architectures, processes, and data flows.
  
• Assess best practices and design schemas matching business needs for delivering a modern analytics solution.
  
• Be an active member of our Agile team, participating in all ceremonies and continuous improvement activities.

• Full-time Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
  
• 5+ years of real-world data engineering experience in AWS and GCP, with strong expertise in Python, SQL, PySpark, and AWS in an Agile environment.
  
• Strong programming skills in one or more languages, such as Python, Scala, and proficiency in writing efficient and optimized code for data integration, storage, processing, and manipulation.
  
• Strong knowledge of SDLC tools and technologies, including project management software, source code management, CI/CD systems, and binary repository managers.
  
• Good understanding of Data Modeling and Database Design Principles, able to design and implement efficient database schemas meeting data architecture requirements.
  
• Strong SQL skills and experience working with complex data sets, Enterprise Data Warehouse, and writing advanced SQL queries.
  
• Skilled in Data Integration from different sources, such as APIs, databases, flat files, and event streaming.
  
• Strong experience in implementing data pipelines and efficient ELT/ETL processes, batch and real-time, in AWS and using open-source solutions.
  
• Strong experience with scalable and distributed Data Technologies, such as Spark/PySpark, DBT, and Kafka, to handle large volumes of data.
  
• Experience with stream-processing systems, such as Storm and Spark-Streaming, is a plus.
  
• Strong experience in designing and implementing Data Warehousing solutions in AWS with Redshift, demonstrated experience in designing and implementing efficient ELT/ETL processes.
  
• Strong experience in Orchestration using Apache Airflow.
  
• Expert in Cloud Computing in AWS, including deep knowledge of various AWS services.
  
• Good understanding of Data Quality and Governance, including implementation of data quality checks and monitoring processes.
  
• Good understanding of BI solutions, including Looker and LookML.
  
• Strong knowledge and hands-on experience of DevOps principles, tools, and technologies, including continuous integration, delivery, infrastructure as code, configuration management, automated testing, performance tuning, and cost management.
  
• Good problem-solving skills, able to troubleshoot data processing pipelines and identify performance bottlenecks and other issues.
  
• Possesses strong leadership skills, willingness to lead, create ideas, and be assertive.
  
• Strong project management and organizational skills.
  
• Excellent communication skills to collaborate with cross-functional teams, conveying complex technical concepts and insights to non-technical stakeholders effectively.

If you have experience with emerging technologies like cloud-native applications, microservices architecture, or serverless computing, please highlight them in your resume.