Big Data Developer

3 weeks ago


Islamabad, Islamabad, Pakistan Datamatics Technologies Full time

Direct message the job poster from Datamatics Technologies

Talent Acquisition Lead | Global Technical Recruitment | Technical Recruitment | EMEA & MENA

Job Summary:

We are looking for a highly skilled Big Data Engineer to spearhead the development of a scalable Big Data Platform. The ideal candidate will have extensive experience with Kafka, Flink, Spark, Python, and Talend, along with expertise in Apache Ranger, Prometheus, and strong SQL skills. This role involves designing and implementing data pipelines, ensuring security, monitoring system performance, and collaborating with cross-functional teams to deliver high-performance data solutions.

Key Responsibilities:

  1. Design, develop, and optimize a Big Data Platform leveraging Kafka, Flink, Spark, and Talend.
  2. Architect and implement real-time and batch data processing pipelines.
  3. Ensure secure data access and governance using Apache Ranger.
  4. Monitor and optimize system performance with Prometheus.
  5. Develop and maintain ETL workflows and data ingestion pipelines.
  6. Write efficient and scalable SQL queries for data transformation and analysis.
  7. Collaborate with Data Scientists, Analysts, and DevOps teams to align data infrastructure with business goals.
  8. Implement data security, compliance, and access control policies.
  9. Troubleshoot and resolve issues related to data processing, storage, and system performance.

Required Skills & Qualifications:

  1. 5+ years of experience in Big Data Engineering.
  2. Hands-on expertise in Kafka, Flink, Spark, Python, and Talend.
  3. Strong proficiency in SQL, with experience in query optimization and performance tuning.
  4. Familiarity with Apache Ranger for data security and access control.
  5. Familiarity with Prometheus for monitoring and performance optimization.
  6. Deep understanding of distributed computing, data partitioning, and pipeline orchestration.
  7. Experience with cloud-based big data solutions (AWS, Azure, or GCP) is a plus.
  8. Strong problem-solving and analytical skills.
  9. Excellent communication and leadership abilities.

Preferred Qualifications:

  1. Experience with Docker, Kubernetes, and containerized big data deployments.
  2. Knowledge of workflow orchestration tools like Airflow.
  3. Hands-on experience with NoSQL databases (e.g., Cassandra, HBase).
Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Engineering and Information Technology

Industries

IT Services and IT Consulting

#J-18808-Ljbffr

  • Islamabad, Islamabad, Pakistan Datamatics Technologies Full time

    Job Summary:We are seeking an experienced Big Data Engineer to lead the development of a scalable Big Data Platform. The ideal candidate will have extensive experience with Kafka, Flink, Spark, Python, and Talend, along with expertise in Apache Ranger, Prometheus, and strong SQL skills.This role involves designing and implementing data pipelines, ensuring...


  • Islamabad, Islamabad, Pakistan NorthBay Solutions Full time

    We are NorthBay Solutions, a leading provider of innovative data solutions. We are currently seeking a highly skilled Big Data Professional to join our team.About the Role:We are looking for an individual with extensive knowledge of big data technologies, including AWS Glue, Lambda, Step Functions, and others, to design, develop, and maintain our data...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    Job DescriptionThe purpose of this role is to have a highly skilled QA Automation Expert who can participate in day-to-day QA activities such as sprint calls and providing updates on assigned tickets. The ideal candidate will improve their knowledge in the relevant domain and create and identify the regression testing suite to eliminate manual effort from...


  • Islamabad, Islamabad, Pakistan Fusemachines Full time

    Job SummaryAbout FusemachinesFusemachines is a leading provider of AI solutions and products to various industries. Our company was founded by Sameer Maskey, Ph.D., an Adjunct Associate Professor at Columbia University, who aims to democratize AI and leverage its global impact. With a robust presence in four countries and a team of over 400 employees, we...


  • Islamabad, Islamabad, Pakistan Fusemachines Full time

    **About the Role**Fusemachines is a global AI company with a presence in four countries, committed to democratizing AI and harnessing its power to transform businesses worldwide. We are currently seeking a highly skilled Sr. Data Engineer to join our team and lead the development of big data infrastructure using Microsoft Azure in the Media domain.This...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    Role OverviewThis role is focused on leveraging big data technologies to improve our QA process. We are seeking a highly skilled QA Automation Expert to join our team and contribute to the success of our company.Responsibilities:Developing and automating processes for gathering expected results from data sources and comparing data results from...


  • Islamabad, Islamabad, Pakistan Datamatics Technologies Full time

    About the Job:Datamatics Technologies is looking for a highly skilled Big Data Engineer to join our team. As a key member of our engineering team, you will be responsible for designing and implementing a scalable Big Data Platform.Responsibilities:Design, develop, and optimize Big Data Platform using Kafka, Flink, Spark, and Talend.Architect and implement...


  • Islamabad, Islamabad, Pakistan Fusemachines Full time

    About the RoleData Engineering LeadWe are seeking a highly skilled Senior Data Engineer to lead, design, build, and maintain the infrastructure required for data integration, storage, processing, and analytics (BI, visualization and Advanced Analytics) using Microsoft Azure in the Media domain. The ideal candidate will possess hands-on experience with...

  • Big Data Engineer

    7 days ago


    Islamabad, Islamabad, Pakistan VisionX Technologies, Inc. Full time

    Key QualificationsTo succeed as a Senior Data Engineer at VisionX Technologies, Inc., you should possess:A minimum of a bachelor's degree in Computer Science, Information Technology, or a related field.A deep passion for data and proven ability to create high-quality data products.At least 8+ years of experience in implementing Big Data, Data Lakehouse, and...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    The purpose of this job is to have a QA Automation Expert who can participate in day-to-day QA activities such as sprint calls and providing updates on assigned tickets. You will improve your knowledge in the relevant domain and, with your abilities and skillset, create and identify the regression testing suite to eliminate manual effort from the existing QA...


  • Islamabad, Islamabad, Pakistan Fusemachines Full time

    **Job Description**Fusemachines is a leading AI company that specializes in delivering state-of-the-art AI products and solutions to various industries. As a Sr. Data Engineer, you will be responsible for designing, building, and maintaining the infrastructure required for data integration, storage, processing, and analytics using Microsoft Azure in the...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    About This OpportunityWe are looking for a talented QA Automation Expert to join our team. In this role, you will be responsible for participating in day-to-day QA activities and providing updates on assigned tickets. You will also improve your knowledge in the relevant domain and create and identify the regression testing suite to eliminate manual effort...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    About the RoleWe are seeking an experienced QA Automation Expert to join our team. As a key member of our QA team, you will be responsible for participating in day-to-day QA activities and providing updates on assigned tickets. You will also improve your knowledge in the relevant domain and create and identify the regression testing suite to eliminate manual...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    Confiz is a leading technology company that has been at the forefront of innovation, delivering cutting-edge solutions to businesses across various industries. We are currently seeking an experienced Machine Learning Engineer with expertise in big data processing to join our team.The ideal candidate will have strong programming skills in Python or R,...


  • Islamabad, Islamabad, Pakistan VisionX Technologies, Inc. Full time

    VisionX Technologies, Inc. OverviewVisionX is a global provider of innovation solutions, partnering with Fortune 1000 companies to drive business growth. Our team leverages agile methodologies, technology accelerators, and Intellectual Property creation to deliver high-quality products.We have been recognized as one of the Top 10 Most Innovative Companies of...


  • Islamabad, Islamabad, Pakistan Confiz Full time

    We are undertaking a Data Transformation and as data becomes fundamental to what we are doing, the ability to have the right data in the right place at the right time, that is accurate and consistent is essential.This role will play a really important central role within this evolving landscape, working alongside the Technology Team and the Product Owner to...


  • Islamabad, Islamabad, Pakistan Datamatics Technologies Full time

    About this Opportunity:We are looking for an experienced Big Data Engineer to join our team. As a key member of our engineering team, you will be responsible for designing and implementing a scalable Big Data Platform.This role involves working closely with cross-functional teams to deliver high-performance data solutions. The successful candidate will have...


  • Islamabad, Islamabad, Pakistan DigyCorp Full time

    Job FunctionThe successful candidate will be responsible for implementing data pipelines to feed real-time and batch data to AI/ML models, supporting deployment and monitoring of machine learning models using Azure Machine Learning and MLOps best practices, integrating data from various on-premises and cloud sources, creating and optimizing Extract,...


  • Islamabad, Islamabad, Pakistan Datamatics Technologies Full time

    The Role:We are seeking an experienced Big Data Engineer to design and implement a scalable Big Data Platform. The ideal candidate will have hands-on experience with Kafka, Flink, Spark, Python, and Talend, as well as expertise in Apache Ranger and Prometheus.This role involves working closely with cross-functional teams to deliver high-performance data...

  • Sr. Data Engineer

    3 weeks ago


    Islamabad, Islamabad, Pakistan Fusemachines Full time

    Get AI-powered advice on this job and more exclusive features.About FusemachinesFusemachines is a 10+ year old AI company, dedicated to delivering state-of-the-art AI products and solutions to a diverse range of industries. Founded by Sameer Maskey, Ph.D., an Adjunct Associate Professor at Columbia University, our company is on a steadfast mission to...