Data Engineering Lead

3 weeks ago

Islamabad, Islamabad, Pakistan Confiz Full time

We are seeking a Data Engineering Lead with 8+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.

Key Responsibilities

  • Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.
  • Develop batch processing workflows with Apache Spark (PySpark)
  • Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory
  • Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes
  • Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg
  • Build high-performance analytics engines using tools like Trino or Presto
  • Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps
  • Ensure data quality and consistency using tools like Great Expectations
  • Write modular, well-tested, and maintainable Python and SQL code
  • Develop an observability layer to monitor and optimize performance across data pipelines
  • Participate in agile ceremonies and contribute to sprint planning and reviews

Required Skills & Experience
  • Advanced Python programming with a strong focus on modular and testable code
  • Strong knowledge of SQL and experience working with large-scale datasets
  • Hands-on experience with at least one major cloud platform (Azure preferred)
  • Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)
  • Expertise in Apache Spark (PySpark) for batch processing
  • Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)
  • Proficient in using Azure Data Factory or Apache Airflow for data orchestration
  • Experience in building APIs to expose large datasets
  • Solid experience with Docker, Kubernetes, and Helm
  • Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg
  • Basic experience with Terraform for infrastructure provisioning
  • Practical experience with data quality frameworks (e.g., Great Expectations)
  • Comfortable working in agile development teams
  • Proven ability in debugging and performance tuning of streaming and batch data jobs
  • Experience with AI-driven tools (e.g., text-to-SQL) is a plus

We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.

People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.

To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/
#J-18808-Ljbffr

  • Data Engineering Lead

    4 weeks ago

    Islamabad, Islamabad, Pakistan Confiz Full time

    Join to apply for the Data Engineering Lead role at ConfizContinue with Google Continue with Google2 days ago Be among the first 25 applicantsJoin to apply for the Data Engineering Lead role at ConfizGet AI-powered advice on this job and more exclusive features.Sign in to access AI-powered advicesContinue with Google Continue with GoogleContinue with Google...

  • Lead Data Engineer

    2 weeks ago

    Islamabad, Islamabad, Pakistan beBeeDataEngineer Full time $100,000 - $120,000

    Job OverviewWe are seeking an experienced Data Engineer to join our team. The ideal candidate will have a strong background in designing and building robust data pipelines using orchestration tools.The selected individual will be responsible for leading the development of data engineering solutions that drive business growth and improvement.Minimum 7+ years...

  • Lead Data Engineer

    4 days ago

    Islamabad, Islamabad, Pakistan S&P Global Full time

    About the Role:Grade Level (for internal use):11The Role: Lead Data EngineerThe Location: Islamabad, PakistanThe Team: Our team is responsible for the design, architecture, and development of our Content applications using a variety of tools that are regularly updated as new technologies emerge. You will have the opportunity every day to work with people...

  • Data Engineer

    4 weeks ago

    Islamabad, Islamabad, Pakistan Nisum Full time

    Get AI-powered advice on this job and more exclusive features.Job DescriptionNisum is a leading global digital commerce firm headquartered in California, with services spanning digital strategy and transformation, insights and analytics, blockchain, business agility, and custom software development. Founded in 2000 with the customer-centric mottoJob...

  • Data Engineer

    3 weeks ago

    Islamabad, Islamabad, Pakistan Nisum Full time

    Job DescriptionNisum is a leading global digital commerce firm headquartered in California, with services spanning digital strategy and transformation, insights and analytics, blockchain, business agility, and custom software development. Founded in 2000 with the customer-centric motto "Building Success Together," Nisum has grown to over 1,800 professionals...

  • Data Engineer

    3 weeks ago

    Islamabad, Islamabad, Pakistan AXIAN Group Full time

    Job Title: Data Engineer (AWS + MLOps)Department: Data & PredictionReports to: Head of Data & PredictionType: Full-timeAbout Axian OIFDo you thrive in complex, dynamic environments where data-driven decisions drive significant financial and social impact? Are you passionate about mobile money's potential to revolutionize financial inclusion across Africa? If...

  • Data Engineering Leader

    6 days ago

    Islamabad, Islamabad, Pakistan beBeeData Full time 2,500,000 - 3,500,000

    Job SummaryWe are seeking a skilled and experienced Data Engineering Lead to spearhead our data strategy. The ideal candidate will have 8+ years of hands-on experience in real-time and batch data processing, containerization, and cloud-based data orchestration.Data Pipeline Design and Implementation:Batch Processing Workflows with Apache Spark:Orchestration...

  • Data Engineer

    5 days ago

    Islamabad, Islamabad, Pakistan Codeaza Technologies Full time

    Codeaza Technologies is dedicated to helping individuals and businesses showcase their ideas. Whether it's building a website, launching an online store, or selling services, our team of experienced developers can turn your dreams into reality. We are passionate about providing innovative solutions and exceptional customer experiences.Company...

  • Data Engineer

    3 weeks ago

    Islamabad, Islamabad, Pakistan LMK Resources Ltd. Full time

    Job Summary:The Data Engineer will design, build, and maintain data pipelines and data models to support analytics and reporting needs.Responsibilities:Design, build, and maintain robust and efficient data pipelines using SQL and Oracle Data Integrator.Develop data models to support analytics, reporting, and machine learning use cases.Ensure data pipelines...

  • Team Lead Data Analytics

    4 weeks ago

    Islamabad, Islamabad, Pakistan LMKR Full time

    Join to apply for the Team Lead Data Analytics role at LMKRJoin to apply for the Team Lead Data Analytics role at LMKRThe Team Lead – Data Analytics will lead and oversee the design, delivery, and management of advanced data solutions and machine learning projects. The role ensures high standards of data quality, governance, and compliance while managing...