Data Engineering Lead

3 days ago

Lahore, Punjab, Pakistan Confiz Full time

We are seeking a passionate and experienced Data Engineering Lead to join our dynamic team, responsible for managing and optimizing our data lake platform. The ideal candidate will have extensive experience in data engineering, particularly in managing complex data pipelines and ensuring data quality, performance, and availability. This role requires a proactive leader who owns end-to-end operations of the platform and can drive solutions and improvements while effectively communicating with stakeholders.

Responsibilities:

  • Diagnose and resolve pipeline issues while implementing effective solutions to minimise outages, enhance performance, and increase platform resilience.
  • Implement and maintain monitoring solutions to assess platform performance, health, and data quality.
  • Establish and track KPIs to ensure timely data availability.
  • Collaborate with an engineering team to ensure the best practices for data ingestion, storage, and processing in Delta Lake.
  • Optimize resource usage and platform costs, ensuring a balance between performance and cost-effectiveness as the platform scales.
  • Drive initiatives for data quality assurance, proactively identifying and resolving issues before they impact stakeholders.
  • Manage deployments of pipeline changes using Azure DevOps, ensuring consistency and reliability through CI/CD best practices.
  • Identify and recommend enhancements to processes, tools, and technologies to improve operational efficiency.
  • Maintain regular updates with stakeholders, ensuring transparent communication of progress and challenges.

Requirements:

  • Bachelor's degree in Computer Science or a related field
  • 8+ years of experience in data engineering, with hands-on experience in:
  • Apache NiFi, Databricks, Delta Lake, Azure Synapse Analytics, Azure Data Lake Storage, Azure Data Factory, and Power BI.
  • Proven track record in managing and troubleshooting complex data pipelines, with a focus on performance improvement and operational efficiency.
  • Experience in setting up and integrating open-source tools for ETL and data visualization/analytics.
  • Experience in setting up and maintaining Kubernetes clusters for deploying and managing data applications.
  • Experience working in Agile/Scrum (SAFe) environments, participating in sprints, stand-ups, and cross-functional team collaboration.
  • Excellent problem-solving skills and the ability to think critically under pressure.
  • Strong communication skills, with a knack for presenting information to both technical and non-technical stakeholders.
  • Self-driven and accountable, with a proactive mindset and strong sense of ownership.
  • Microsoft Azure Data Engineer Associate (DP-203) or Databricks certification is preferred.
  • Prior experience working in the telecom industry or supporting large-scale, data-driven telecom platforms is preferred.

We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast-growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.

People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.

To know more about Confiz Limited, visit: Confiz LinkedIn

#J-18808-Ljbffr

  • Lead Data Engineer

    2 weeks ago

    Lahore, Punjab, Pakistan Lambda Logics Full time

    1 day ago Be among the first 25 applicantsGet AI-powered advice on this job and more exclusive features.Direct message the job poster from Lambda LogicsWe are seeking a Lead Data Engineer to own the design, development, and management of data pipelines, orchestration, and cloud operations for all data initiatives at Revmatics.ai. In this role, you will work...

  • Lead Data Engineer

    1 week ago

    Lahore, Punjab, Pakistan Lambda Logics Full time

    We are seeking a Lead Data Engineer to own the design, development, and management of data pipelines, orchestration, and cloud operations for all data initiatives. In this role, you will work closely with cross-functional teams to ensure the seamless integration of data across platforms, with a strong focus on both traditional analytics and our AI-driven...

  • Lead Data Engineer

    2 weeks ago

    Lahore, Punjab, Pakistan Devflovv Full time

    1 day ago Be among the first 25 applicantsDirect message the job poster from DevflovvRecruitment Specialist @Devflovv | Connecting Top Tech Talent with Opportunities | Leveraging Mass Communication Skills for Exceptional Candidate…We're looking for a Lead Data Engineer to own the design, development, and scaling of data infrastructure across cloud...

  • Lead Data Engineer

    2 weeks ago

    Lahore, Punjab, Pakistan Clinic IT Soultions Full time

    1 day ago Be among the first 25 applicantsWe are seeking a Lead Data Engineer to own the design, development, and management of data pipelines, orchestration, and cloud operations for all data initiatives at Revmatics.ai. In this role, you will work closely with cross-functional teams to ensure the seamless integration of data across platforms, with a strong...

  • Lead Data Scientist

    3 days ago

    Lahore, Punjab, Pakistan beBee Careers Full time

    Data Engineering Lead PositionKey ResponsibilitiesManage and troubleshoot complex data pipelines, focusing on performance improvement and operational efficiency.Implement and maintain monitoring solutions to assess platform performance, health, and data quality.Establish and track KPIs to ensure timely data availability.Collaborate with engineering teams to...

  • Lead Engineer

    1 week ago

    Lahore, Punjab, Pakistan Descon Engineering Limited Full time

    Lead Engineer Descon Engineering Limited, Pakistan Responsibilities:Responsible for monitoring progress and quality of the design work as per project requirements.Monitors and ensures progress and quality of the design work carried out in-house or sub-contracted as per project requirements.Ensures the provision of all necessary inputs for proposal...

  • Data Engineering Lead

    7 days ago

    Lahore, Punjab, Pakistan beBee Careers Full time

    Senior Data EngineerDevsinc is seeking a talented Senior Data Engineer to join our dynamic team. The ideal candidate will possess a deep understanding of data architecture and engineering principles, along with hands-on experience in developing, optimizing, and maintaining data pipelines and databases.This role is crucial to support our data-driven...

  • Data Engineering Lead

    2 days ago

    Lahore, Punjab, Pakistan beBee Careers Full time

    We are looking for a highly skilled Senior Big Data Engineer with expertise in data engineering and a strong passion for leveraging Big Data technologies.The ideal candidate will have at least 4-6 years of experience specifically in Data Engineering, with a proven track record of designing and developing cutting-edge Big Data solutions. They will be part of...

  • Data Engineering Lead

    7 days ago

    Lahore, Punjab, Pakistan beBee Careers Full time

    Job DescriptionWe are seeking a skilled Data Engineer to design, build, and maintain efficient and scalable data pipelines using tools like Airbyte, Airflow, and dbt, with a focus on integrating with Snowflake.The ideal candidate will manage and optimize data warehousing solutions using Snowflake, ensuring data is organized, secure, and accessible for...

  • Data Engineering Lead

    2 days ago

    Lahore, Punjab, Pakistan beBee Careers Full time

    A seasoned Big Data Engineer is needed to design, develop, and maintain complex data systems. The ideal candidate will have expertise in big data processing, cloud-based technologies, and continuous integration/deployment methodologies.Responsibilities:Create and manage scalable data pipelines using cloud-based services.Develop and deploy optimized data...