Senior Cloud Data Architect

Job Description:

This is a remote contract position responsible for designing, building and maintaining the infrastructure required for data integration, storage, processing and analytics. We are looking for a skilled Senior Data Engineer with strong background in Python, SQL, PySpark, Azure, Databricks, Synapse, Azure Data Lake, DevOps and cloud-based large scale data applications.

The ideal candidate will develop in an Agile environment, contributing to the architecture, design and implementation of Data products in the Aviation Industry, including migration from Synapse to Azure Data Lake. This role involves hands-on coding, mentoring junior staff and collaboration with multi-disciplined teams to achieve project objectives.

• Must have a full-time Bachelor's degree in Computer Science or similar
• At least 5 years of experience as a data engineer with strong expertise in Databricks, Azure, DevOps, or other hyperscalers
• 5+ years of experience with Azure DevOps, GitHub
• Proven experience delivering large scale projects and products for Data and Analytics, as a data engineer, including migrations

Required Skills and Qualifications:

• Strong programming skills in one or more languages such as Python, Scala, and proficiency in writing efficient and optimized code for data integration, migration, storage, processing and manipulation
• Strong understanding and experience with SQL and writing advanced SQL queries
• Thorough understanding of big data principles, techniques, and best practices
• Strong experience with scalable and distributed data processing technologies such as Spark/PySpark, DBT and Kafka, to be able to handle large volumes of data
• Solid Databricks development experience with significant Python, PySpark, Spark SQL, Pandas, NumPy in Azure environment
• Strong experience in designing and implementing efficient ELT/ETL processes in Azure and Databricks and using open source solutions being able to develop custom integration solutions as needed
• Skilled in data integration from different sources such as APIs, databases, flat files, event streaming
• Expertise in data cleansing, transformation, and validation
• Proficiency with relational databases (Oracle, SQL Server, MySQL, Postgres, or similar) and non-relational databases (MongoDB or Table)
• Good understanding of data modeling and database design principles, being able to design and implement efficient database schemas that meet the requirements of the data architecture to support data solutions
• Strong experience in designing and implementing data warehousing, data lake and data lake house solutions in Azure and Databricks
• Good experience with Delta Lake, Unity Catalog, Delta Sharing, Delta Live Tables (DLT)
• Strong understanding of the software development lifecycle (SDLC), especially Agile methodologies
• Strong knowledge of SDLC tools and technologies Azure DevOps and GitHub, including project management software (Jira, Azure Boards or similar), source code management (GitHub, Azure Repos or similar), CI/CD system (GitHub actions, Azure Pipelines, Jenkins or similar) and binary repository manager (Azure Artifacts or similar)
• Strong understanding of DevOps principles, including continuous integration, continuous delivery (CI/CD), infrastructure as code (IaC – Terraform, ARM including hands-on experience), configuration management, automated testing, performance tuning and cost management and optimization
• Strong knowledge in cloud computing specifically in Microsoft Azure services related to data and analytics, such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake, Azure Stream Analytics, SQL Server, Azure Blob Storage, Azure Data Lake Storage, Azure SQL Database, etc.
• Experience in orchestration using technologies like Databricks workflows and Apache Airflow
• Strong knowledge of data structures and algorithms and good software engineering practices
• Proven experience migrating from Azure Synapse to Azure Data Lake, or other technologies
• Strong analytical skills to identify and address technical issues, performance bottlenecks, and system failures
• Proficiency in debugging and troubleshooting issues in complex data and analytics environments and pipelines
• Good understanding of data quality and governance, including implementation of data quality checks and monitoring processes to ensure that data is accurate, complete, and consistent
• Experience with BI solutions including PowerBI is a plus
• Strong written and verbal communication skills to collaborate and articulate complex situations concisely with cross-functional teams, including business users, data architects, DevOps engineers, data analysts, data scientists, developers, and operations teams
• Ability to document processes, procedures, and deployment configurations
• Promote and enforce best practices in data engineering, data governance, and data quality
• Ensure data quality and accuracy
• Design, implement and maintain data security and privacy measures
• Be an active member of an Agile team, participating in all ceremonies and continuous improvement activities, being able to work independently as well as collaboratively]