Data Engineering Leader

6 days ago

Islamabad, Islamabad, Pakistan beBeeData Full time 2,500,000 - 3,500,000
Job Summary

We are seeking a skilled and experienced Data Engineering Lead to spearhead our data strategy. The ideal candidate will have 8+ years of hands-on experience in real-time and batch data processing, containerization, and cloud-based data orchestration.

  • Data Pipeline Design and Implementation:
  • Batch Processing Workflows with Apache Spark:
  • Orchestration and Scheduling with Apache Airflow:
  • Containerization with Docker and Kubernetes:
  • Modern Storage Solutions with Parquet and Delta Lake:
  • High-Performance Analytics Engines with Trino:
  • Collaboration with DevOps for Infrastructure Management:
About the Role

The Data Engineering Lead will be responsible for designing, building, and maintaining robust, scalable, and efficient data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.

They will also develop batch processing workflows with Apache Spark (PySpark) and orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory.

In addition, they will containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes.

The ideal candidate will have expertise in modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg.

They will also be responsible for building high-performance analytics engines using tools like Trino or Presto and collaborating with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps.

Key Responsibilities:

  • Design and Implement Data Pipelines:
  • Develop Batch Processing Workflows:
  • Orchestrate and Schedule Data Workflows:
  • Containerize Applications:
  • Implement Modern Storage Solutions:
  • Build High-Performance Analytics Engines:
  • Collaborate with DevOps:
Required Skills & Experience

Technical Requirements:

  • Advanced Python Programming:
  • Strong Knowledge of SQL:
  • Hands-on Experience with Cloud Platforms:
  • Real-Time Data Processing:
  • Apache Spark (PySpark):
  • Lakehouse Architectures:
  • Data Orchestration Frameworks:
  • Docker, Kubernetes, and Helm:
  • Terraform:
  • Data Quality Frameworks:

Soft Skills:

  • Able to Work in Agile Teams:
  • Proven Ability in Debugging and Performance Tuning:
About Us

We are a leading IT company with a vibrant culture of learning and collaboration.

Company Overview:

We serve Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.

Certifications:

ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS)

],

  • Data Leader

    6 days ago

    Islamabad, Islamabad, Pakistan beBeeDataLeader Full time 1,800,000 - 2,500,000

    Job DescriptionLead Data Engineer at a leading organization, you will spearhead complex data projects and collaborate with cross-functional teams to deliver high-quality results.Key Responsibilities:Design, develop, and implement large-scale data systemsCollaborate with onsite and offshore teams to drive project successLead data-intensive projects from...

  • Data Engineer

    5 days ago

    Islamabad, Islamabad, Pakistan Codeaza Technologies Full time

    Codeaza Technologies is dedicated to helping individuals and businesses showcase their ideas. Whether it's building a website, launching an online store, or selling services, our team of experienced developers can turn your dreams into reality. We are passionate about providing innovative solutions and exceptional customer experiences.Company...

  • Data Engineer

    3 weeks ago

    Islamabad, Islamabad, Pakistan LMK Resources Ltd. Full time

    Job Summary:The Data Engineer will design, build, and maintain data pipelines and data models to support analytics and reporting needs.Responsibilities:Design, build, and maintain robust and efficient data pipelines using SQL and Oracle Data Integrator.Develop data models to support analytics, reporting, and machine learning use cases.Ensure data pipelines...

  • Data Engineer

    2 weeks ago

    Islamabad, Islamabad, Pakistan NorthBay Solutions LLC Full time

    Job Title: Data EngineerLocation:Karachi, Lahore , Islamabad (Hybrid)Experience:5+ YearsJob Type: Full-TimeJob Overview:We are looking for a highly skilled and experienced Data Engineer with a strong foundation in Big Data, distributed computing, and cloud-based data solutions. This role demands a strong understanding of end-to-end Data pipelines, data...

  • Data Engineer

    3 weeks ago

    Islamabad, Islamabad, Pakistan AXIAN Group Full time

    Job Title: Data Engineer (AWS + MLOps)Department: Data & PredictionReports to: Head of Data & PredictionType: Full-timeAbout Axian OIFDo you thrive in complex, dynamic environments where data-driven decisions drive significant financial and social impact? Are you passionate about mobile money's potential to revolutionize financial inclusion across Africa? If...

  • Data Engineer

    4 weeks ago

    Islamabad, Islamabad, Pakistan Nisum Full time

    Get AI-powered advice on this job and more exclusive features.Job DescriptionNisum is a leading global digital commerce firm headquartered in California, with services spanning digital strategy and transformation, insights and analytics, blockchain, business agility, and custom software development. Founded in 2000 with the customer-centric mottoJob...

  • Data Engineer

    3 weeks ago

    Islamabad, Islamabad, Pakistan Nisum Full time

    Job DescriptionNisum is a leading global digital commerce firm headquartered in California, with services spanning digital strategy and transformation, insights and analytics, blockchain, business agility, and custom software development. Founded in 2000 with the customer-centric motto "Building Success Together," Nisum has grown to over 1,800 professionals...

  • Advanced Data Processing Expert

    3 days ago

    Islamabad, Islamabad, Pakistan beBeeDataEngineering Full time 10,000,000 - 12,000,000

    Job Title: Data Engineering LeaderAbout the Role:We are seeking a skilled and experienced Data Engineering Leader to spearhead our data engineering initiatives. As a key member of our team, you will be responsible for designing and developing complex data pipelines and solutions using Big Data and cloud-native technologies.Key Responsibilities:Design and...

  • Big Data Engineer

    1 week ago

    Islamabad, Islamabad, Pakistan Adept Tech Solutions Full time

    Get AI-powered advice on this job and more exclusive features.Direct message the job poster from Adept Tech SolutionsGPU whisperer | Teaching silicon to see and thinkType: Full-time (Onsite)Working Hours: 9:00 AM - 5:30 PMExperience Level: 1-2 yearsAbout the RoleWe're seeking a Big Data Engineer to build sophisticated data processing pipelines that combine...

  • Data Scientist Leader

    2 weeks ago

    Islamabad, Islamabad, Pakistan beBeeMachine Full time 2,000,000 - 2,500,000

    Job Description:As a Senior Machine Learning Engineer, you will be responsible for designing, implementing, and deploying advanced machine learning models and pipelines to drive real-world impact. You will work with cross-functional teams to identify business problems that can be solved using ML and lead the development of scalable machine learning solutions...