Senior Data Engineer

Client Company Introduction:

It is a leading software engineering company delivering innovative, AI-powered platforms for global clients.

Opening
:
Senior Data Engineer (AWS Integration)

Key Responsibilities:

Data Lake Architecture:

-Design, implement, and maintain a data lake on AWS using Iceberg tables on S3, ensuring data consistency, high availability, and easy querying.

-Work with AWS Glue for ETL processes, creating Glue jobs to process, transform, and load data from SQL Server and other sources into Iceberg tables.

-Implement and optimize Spark-based data transformations in AWS Glue, ensuring efficient and scalable data workflows.

SQL Server to AWS Migration:

-Lead the migration of on-premises SQL Server data to AWS S3 using AWS Glue, DMS, and other relevant tools.

-Design and implement CDC (Change Data Capture) pipelines to enable real-time data replication from SQL Server to AWS services like S3, Iceberg, and Redshift.

Data Integration & Management:

-Design and implement data pipelines for batch and real-time data processing, leveraging AWS DMS, Kinesis, and Glue Streaming.

-Automate data loading, transformation, and archiving processes with Glue and ensure the correct data schema management and data governance.

-Set up and maintain data cataloging systems using AWS Glue Data Catalog to organize and manage metadata for Iceberg tables.

Querying with Athena:

-Use AWS Athena for querying Iceberg tables on S3, enabling high-performance, low-latency SQL querying on large datasets stored in Iceberg format.

-Optimize Athena queries to improve performance and manage large-scale data sets effectively.

-Design efficient partitioning strategies and manage the Iceberg tables using Athena's native support for Iceberg.

Continuous Improvement & Optimization:

-Implement data compaction strategies to reduce small file issues in Iceberg and optimize storage and query performance.

-Continuously monitor, troubleshoot, and optimize data pipelines for performance, cost efficiency, and data quality.

Collaboration & Documentation:

-Work closely with data scientists, analysts, and business teams to understand data needs and develop solutions that support their goals.

-Provide clear documentation on architecture, solutions, and workflows to ensure smooth knowledge transfer and maintainability.

Required Skills & Qualifications:

Experience:

-Proven experience
(3+ years) in data engineering roles with expertise in SQL Server, AWS services, and Iceberg.

-Hands-on experience with AWS Glue, AWS DMS, S3
, and Kinesis in building scalable and reliable data pipelines. -Solid understanding of Change Data Capture (CDC) and its implementation with SQL Server and AWS DMS.

-Experience with Apache Iceberg for building data lakes and managing large-scale, versioned data on S3.

-Strong knowledge of ETL processes, data modeling, and data transformation using Spark and Glue.

-AWS Athena experience for querying and managing Iceberg tables in S3.

Technical Skills:

-SQL Server: Expertise in database administration, SQL queries, stored procedures, and performance tuning.

-AWS Glue: Experience in creating, debugging, and deploying Glue jobs and managing Glue Data Catalog.

-AWS DMS: Experience with migrating and replicating data from SQL Server to AWS cloud services, including CDC.

-Apache Iceberg: Practical knowledge of Iceberg table formats, partitioning, compaction, and schema evolution.

-AWS S3: Experience managing large datasets in S3, implementing partitioning strategies, and optimizing storage.

-Kinesis: Familiarity with real-time streaming data pipelines using Kinesis Data Streams and Glue Streaming.

-AWS Athena: Strong experience querying Iceberg tables and optimizing large-scale data queries.

Tools & Technologies:

-Apache Spark (for Glue and standalone Spark jobs)

-SQL, Python, AWS SDK for automation, and scripting tasks.

-Data governance: Experience with AWS Lake Formation for security and access management in a data lake environment.

Desired Soft Skills:

-Strong problem-solving skills with the ability to troubleshoot complex data issues.

-Ability to collaborate effectively with cross-functional teams (data scientists, analysts, DevOps, and stakeholders).

-Strong communication skills to document and explain technical concepts clearly to both technical and non-technical audiences.

-Detail-oriented with a focus on data quality, integrity, and performance.

Experience: 3 + years

Work Timings: (Monday to Fri:11:00 AM - 7:30 PM)

Work Mode: (Hybrid - Karachi, Pakistan) & (Hybrid - Kolkata and Ahmadabad, India)

Salary: Tax-free (USD)

About HR Ways:
HR Ways is an award-winning Technical Recruitment Firm helping software houses and IT Product companies internationally and locally to find IT Talent. HR Ways is engaged by 300+ Employers worldwide ranging from the world's biggest SaaS Companies to the most competitive Startups. We have entities in Dubai, Canada, the US, the UK, Pakistan, India, Saudi Arabia, Portugal, Brazil, and other parts of the world.