Data Engineering Lead
5 days ago
Join to apply for the Data Engineering Lead role at Confiz
Continue with Google Continue with Google
2 days ago Be among the first 25 applicants
Join to apply for the Data Engineering Lead role at Confiz
Get AI-powered advice on this job and more exclusive features.
Sign in to access AI-powered advicesContinue with Google Continue with Google
Continue with Google Continue with Google
Continue with Google Continue with Google
Continue with Google Continue with Google
Continue with Google Continue with Google
Continue with Google Continue with Google
We are seeking a Data Engineering Lead with 8+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.
Key Responsibilities
- Design, build, and maintain real-time data pipelines using streaming frameworks such as Kafka, Apache Flink, and Spark Structured Streaming.
- Develop batch processing workflows with Apache Spark (PySpark)
- Orchestrate and schedule data workflows using orchestration frameworks such as Apache Airflow and Azure Data Factory
- Containerize applications using Docker, manage deployments with Helm, and run them on Kubernetes
- Implement modern storage solutions using open formats such as Parquet, Delta Lake, and Apache Iceberg
- Build high-performance analytics engines using tools like Trino or Presto
- Collaborate with DevOps to manage infrastructure with Terraform and integrate with CI/CD pipelines via Azure DevOps
- Ensure data quality and consistency using tools like Great Expectations
- Write modular, well-tested, and maintainable Python and SQL code
- Develop an observability layer to monitor and optimize performance across data pipelines
- Participate in agile ceremonies and contribute to sprint planning and reviews
- Advanced Python programming with a strong focus on modular and testable code
- Strong knowledge of SQL and experience working with large-scale datasets
- Hands-on experience with at least one major cloud platform (Azure preferred)
- Solid experience with real-time data processing (Kafka, Flink, or Spark Streaming)
- Expertise in Apache Spark (PySpark) for batch processing
- Experience implementing lakehouse architectures and working with columnar storage (e.g., ClickHouse)
- Proficient in using Azure Data Factory or Apache Airflow for data orchestration
- Experience in building APIs to expose large datasets
- Solid experience with Docker, Kubernetes, and Helm
- Familiarity with data lake open formats such as Parquet, Delta Lake, and Iceberg
- Basic experience with Terraform for infrastructure provisioning
- Practical experience with data quality frameworks (e.g., Great Expectations)
- Comfortable working in agile development teams
- Proven ability in debugging and performance tuning of streaming and batch data jobs
- Experience with AI-driven tools (e.g., text-to-SQL) is a plus
What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.
People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.
To know more about Confiz Limited, visit: https://www.linkedin.com/company/confiz-pakistan/Seniority level
- Seniority levelMid-Senior level
- Employment typeOther
- Job functionInformation Technology
- IndustriesIT Services and IT Consulting
Referrals increase your chances of interviewing at Confiz by 2x
Associate Software Engineer- Data EngineerL1 Application Support Engineer – Data Science SolutionsData Analytics Specialist – National Fiscal Pact Reform - REMIT ProgrammeJob Alert: Software Engineer Positions for Final Year & Fresh CS GraduatesSenior Data Analyst/Reporting - Microsoft Power BIWe're unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr-
Lead Data Engineer
2 weeks ago
Islamabad, Islamabad, Pakistan Stixor Full time3 days ago Be among the first 25 applicantsExperience: 4+ years of experienceAbout the RoleThe Lead Data Engineer will be responsible for designing, building, and maintaining robust data pipelines and architecture. You'll lead a team of engineers and work with cross-functional stakeholders to ensure data reliability, scalability, and availability. You will...
-
Lead Data Engineer
2 weeks ago
Islamabad, Islamabad, Pakistan Confiz Full timeWe are looking for a strategic and hands-on Operations Lead to ensure the resilience,performance, and cost-effectiveness of our Azure-based data platform. This role combines platform reliability, incident response, SLA ownership, and cost management (FinOps) with deployment oversight and CI/CD ownership. You will act as the single point of contact for...
-
Data Engineering Lead
2 days ago
Islamabad, Islamabad, Pakistan Confiz Full timeWe are seeking a Data Engineering Lead with 8+ years of hands-on experience and a strong background in real-time and batch data processing, containerization, and cloud-based data orchestration. This role is ideal for someone passionate about building robust, scalable, and efficient data pipelines, and who thrives in agile, collaborative environments.Key...
-
Lead Data Scientist
2 weeks ago
Islamabad, Islamabad, Pakistan Stixor Full timeExperience: 5+ years of experienceAbout the RoleAs a Lead Data Scientist, you will define our data science strategy, develop advanced machine learning models, and translate insights into business outcomes. You will lead a team of data scientists and collaborate with engineering, product, and business teams to deliver scalable solutions.Key...
-
Data Engineer
5 days ago
Islamabad, Islamabad, Pakistan Nisum Full timeGet AI-powered advice on this job and more exclusive features.Job DescriptionNisum is a leading global digital commerce firm headquartered in California, with services spanning digital strategy and transformation, insights and analytics, blockchain, business agility, and custom software development. Founded in 2000 with the customer-centric mottoJob...
-
Data Engineer
1 day ago
Islamabad, Islamabad, Pakistan Nisum Full timeJob DescriptionNisum is a leading global digital commerce firm headquartered in California, with services spanning digital strategy and transformation, insights and analytics, blockchain, business agility, and custom software development. Founded in 2000 with the customer-centric motto "Building Success Together," Nisum has grown to over 1,800 professionals...
-
Data Engineer
2 weeks ago
Islamabad, Islamabad, Pakistan Leed | Pakistan Full timeDirect message the job poster from Leed | PakistanCertified Human Resources Professional -CHRPCompany Description:Leed Pakistan is a software development company established in 2020, headquartered in Canada, with development offices in Germany and Pakistan. Despite our rapid growth to 50+ professionals, we prioritize quality and customer care. We work with...
-
Data Engineer
1 day ago
Islamabad, Islamabad, Pakistan AXIAN Group Full timeJob Title: Data Engineer (AWS + MLOps)Department: Data & PredictionReports to: Head of Data & PredictionType: Full-timeAbout Axian OIFDo you thrive in complex, dynamic environments where data-driven decisions drive significant financial and social impact? Are you passionate about mobile money's potential to revolutionize financial inclusion across Africa? If...
-
Data Engineer
2 weeks ago
Islamabad, Islamabad, Pakistan VisionX Technologies, Inc. Full timeAbout us: VisionX works with world-leading brands, Fortune 1000 as their innovation partner, providing product strategy and custom application development leveraging agile methodologies, technology accelerators, and by creating Intellectual Property.VisionX has been listed in the Top 10 Most Innovative Companies of 2020 by Fast Company – ranked alongside...
-
Team Lead Data Analytics
5 days ago
Islamabad, Islamabad, Pakistan LMKR Full timeJoin to apply for the Team Lead Data Analytics role at LMKRJoin to apply for the Team Lead Data Analytics role at LMKRThe Team Lead – Data Analytics will lead and oversee the design, delivery, and management of advanced data solutions and machine learning projects. The role ensures high standards of data quality, governance, and compliance while managing...
