Data Engineer

Job Title: Data Engineer

Job Title: Data Engineer

Location: Karachi, Lahore , Islamabad (Hybrid)

Experience: 5+ Years

Job Type: Full-Time

Job Overview

We are looking for a highly skilled and experienced Data Engineer with a strong foundation in Big Data, distributed computing, and cloud-based data solutions. This role demands a strong understanding of end-to-end Data pipelines, data modeling, and advanced data engineering practices across diverse data sources and environments. You will play a pivotal role in building, deploying, and optimizing data infrastructure and pipelines in a scalable cloud-based architecture.

Key Responsibilities

• Design, develop, and maintain large-scale Data pipelines using modern big data technologies and cloud-native tools.
• Build scalable and efficient distributed data processing systems using Hadoop, Spark, Hive, and Kafka.
• Work extensively with cloud platforms (preferably AWS) and services like EMR, Glue, Lambda, Athena, S3.
• Design and implement data integration solutions pulling from multiple sources into a centralized data warehouse or data lake.
• Develop pipelines using DBT (Data Build Tool) and manage workflows with Apache Airflow or Step Functions.
• Write clean, maintainable, and efficient code using Python, PySpark, or Scala for data transformation and processing.
• Build and manage relational and columnar data stores such as PostgreSQL, MySQL, Redshift, Snowflake, HBase, ClickHouse.
• Implement CI/CD pipelines using Docker, Jenkins, and other DevOps tools.
• Collaborate with data scientists, analysts, and other engineering teams to deploy data models into production.
• Drive data quality, integrity, and consistency across systems.
• Participate in Agile/Scrum ceremonies and utilize JIRA for task management.
• Provide mentorship and technical guidance to junior team members.
• Contribute to continuous improvement by making recommendations to enhance data engineering processes and architecture.
  
  

Required Skills & Experience

• 5+ years of hands-on experience as a Data Engineer
• Deep knowledge of Big Data technologies – Hadoop, Spark, Hive, Kafka.
• Expertise in Python, PySpark and/or Scala.
• Proficient with data modeling, SQL scripting, and working with large-scale datasets.
• Experience with distributed storage like HDFS and cloud storage (e.g., AWS S3).
• Hands-on with data orchestration tools like Apache Airflow or StepFunction.
• Experience working in AWS environments with services such as EMR, Glue, Lambda, Athena.
• Familiarity with data warehousing concepts and experience with tools like Redshift, Snowflake (preferred).
• Exposure to tools like Informatica, AbInitio, Apache Iceberg is a plus.
• Knowledge of Docker, Jenkins, and other CI/CD tools.
• Strong problem-solving skills, initiative, and a continuous learning mindset.
  
  

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, or related field.
• Experience with open table formats such as Apache Iceberg.
• Hands-on with AbInitio (GDE, Collect > IT) or Informatica tools.
• Knowledge of Agile methodology, working experience in JIRA.
  
  

Soft Skills

• Self-driven, proactive, and a strong team player.
• Excellent communication and interpersonal skills.
• Passion for data and technology innovation.
• Ability to work independently and manage multiple priorities in a fast-paced environment.

Seniority level

• Seniority levelMid-Senior level

Employment type

• Employment typeFull-time

Job function

• Job functionInformation Technology
• IndustriesIT Services and IT Consulting

Referrals increase your chances of interviewing at NorthBay - Pakistan by 2x

Software Engineering & Development InternIntermediate Full-Stack Software Developer

Karachi Division, Sindh, Pakistan 5 months ago

Karachi Division, Sindh, Pakistan 9 months ago

Karachi Division, Sindh, Pakistan 7 months ago

Karachi Division, Sindh, Pakistan 1 year ago

Karachi Division, Sindh, Pakistan 2 months ago

Karachi Division, Sindh, Pakistan 1 day ago

Karachi Division, Sindh, Pakistan 4 months ago

Karachi Division, Sindh, Pakistan 3 months ago

Karachi Division, Sindh, Pakistan 1 day ago

Karachi East District, Sindh, Pakistan 5 months ago

Backend .Net Core Developer (Wallet - Fintech)

We're unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

#J-18808-Ljbffr

---