Senior Big Data Engineer

3 weeks ago

Lahore, Punjab, Pakistan NorthBay Solutions Full time

We are hiring for Senior Big Data Engineer with solid experience in data engineering and a passion for Big Data. The ideal candidate will have experience of over 4 to 6+ years specifically in Data Engineering.

You will have the chance to join a global, multi-cultural team of Data Engineers, and be part of the growth and development of it, participate in a large variety of exciting and challenging projects and work with our exceptional team of Technology professionals.

Technology Stack Used & Required Knowledge:

  • 4 to 6+ years of experience on ETL Data Engineer, Python, Big Data (Hadoop, spark, hive)
  • Experience with distributed processing technologies and frameworks, such as Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3)
  • Excellent understanding of ETL cycle.
  • Experience in creating and driving large-scale ETL pipelines in AWS-based environment and experience of services such as EMR, Glue, Lambda, Athena
  • Experience with integration of data from multiple data sources.
  • Understanding of data processing pipelines, data management, and data engineering principles, including experience working with large-scale datasets and distributed computing frameworks (e.g., Spark).
  • Good knowledge and understanding of Big data engineering, ETL, Data Modeling, SQL Scripting, and Data Architecture.
  • Strong software development and programming skills with a focus on data using Python, PySpark and/or Scala for data engineering.
  • Experience and understanding of developing and maintaining complex stored procedures
  • Strong proficiency in ETL schedulers such as StepFunction or Apache Airflow
  • Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse)
  • Experience and understanding of various core GCP services such as BigQuery, Cloud Composer, Cloud Storage
  • Knowledge of containerization technologies, CI/CD pipelines and IAC tools like Docker, Jenkins, Terraform for automating software delivery process
  • Experience with the AWS data management tools such as Data lake and Databricks or AWS Snowflake is a plus
  • Understanding of descriptive and exploratory statistics, predictive modeling, evaluation metrics, decision trees, machine learning algorithms is a plus
  • Experience and understanding of AWS data migration services such as SCT, DataSync, Data Extractor will be a plus
  • Should have experience in Jira and Agile development methodologies.
  • Must be open to learning and have the ability to grasp new technologies quickly.
  • Be motivated and passionate about technology.
  • Team player with strong integrity and professionalism who can work well with others in the group.
  • Self-starter with a proven ability to take initiative and strong problem-solving skills.

To keep it short, below are some key responsibilities:

  • Design and develop using Big Data technologies.
  • Design and develop cutting-edge Big Data, and Cloud technologies to build critical, highly complex distributed systems from scratch
  • Design, develop and manage complex ETL jobs and pipelines.
  • Act as a team member, consulting other teammates as your new systems transform the way this entire firm leverages data
  • Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.
  • Support and help them learn and improve their skills, as well as innovate and iterate on best practices.
  • Solve complex issues and provide guidance to team members when needed.
  • Make improvement and process recommendations that have an impact on the business.
#J-18808-Ljbffr

  • Senior Big Data Engineer

    4 weeks ago

    Lahore, Punjab, Pakistan Avanceon Limited Full time

    Join to apply for the Senior Big Data Engineer (Azure) role at Avanceon Middle East & South Asia.As a Big Data engineer, you will be responsible for data-related implementation tasks that include provisioning of data storage services, ingesting, streaming and batch data, transforming data, implementing security requirements, implementing data retention...

  • Full Stack Developer

    1 week ago

    Lahore, Punjab, Pakistan Big Data Inc Full time

    Overview:Big Data Inc is a cutting-edge technology company that offers innovative solutions for big data processing and analysis. We are currently seeking an experienced Full Stack Java Developer to join our team in Lahore, Pakistan.About the Role:The incumbent will be responsible for designing, testing, and implementing large-scale enterprise applications...

  • Senior Big Data Engineer

    4 weeks ago

    Lahore, Punjab, Pakistan NorthBay Solutions LLC Full time

    We are hiring for Senior Big Data Engineer with solid experience in data engineering and a passion for Big Data. The ideal candidate will have experience of over 4 to 6+ years specifically in Data Engineering.You will have the chance to join a global, multi-cultural team of Data Engineers, participate in a large variety of exciting and challenging projects,...

  • Big Data Engineering Expert

    2 weeks ago

    Lahore, Punjab, Pakistan NorthBay Solutions LLC Full time

    We are seeking an experienced Big Data Engineering Expert to join our team at NorthBay Solutions LLC. As a Senior Big Data Engineer, you will have the opportunity to design and develop cutting-edge Big Data technologies, build critical distributed systems from scratch, and lead complex ETL jobs and pipelines.About the Role:Design and develop using Big Data...

  • Big Data Engineer

    1 week ago

    Lahore, Punjab, Pakistan Gtradecenter Full time

    InvoZone is a fast-growing company that offers a range of services to clients worldwide. As a Big Data Engineer, you will play a key role in developing innovative business solutions using Apache Spark.ResponsibilitiesDevelop and implement big data solutions using Apache SparkWork with cross-functional teams to identify business needsDesign and implement data...

  • Big Data Engineering Manager

    6 days ago

    Lahore, Punjab, Pakistan NorthBay Solutions Full time

    NorthBay Solutions is seeking an experienced Big Data Engineering Manager to manage our big data engineering efforts. The ideal candidate will have 4 to 6+ years of experience in data engineering with a focus on big data technologies.About the Position:Main Responsibilities: Lead our data engineering efforts, design and develop large-scale ETL pipelines in...

  • Big Data Technical Lead

    2 weeks ago

    Lahore, Punjab, Pakistan Avanceon Middle East & South Asia Full time

    **About the Role:**We are seeking a highly skilled Senior Big Data Engineer to join our team at Avanceon Middle East & South Asia. As a Senior Big Data Engineer, you will be responsible for developing and implementing big data solutions using cloud-based technologies.Responsibilities:Big Data Solutions: Develop and implement big data solutions using...

  • Senior Big Data Solutions Architect

    3 days ago

    Lahore, Punjab, Pakistan Avanceon Limited Full time

    We are looking for a seasoned Senior Big Data Solutions Architect to lead our team's data engineering efforts.The successful candidate will have extensive experience with big data technologies and a strong understanding of data lake architecture. Key responsibilities include implementing lambda architecture with stream processing and batch processing, as...

  • Cloud Architect for Big Data Solutions

    2 weeks ago

    Lahore, Punjab, Pakistan NorthBay Solutions LLC Full time

    NorthBay Solutions LLC is hiring a skilled Cloud Architect for Big Data Solutions. As a Senior Big Data Engineer, you will be responsible for designing and developing cloud-based Big Data solutions using AWS services such as EMR, Glue, Lambda, and Athena.About the Role:Design and develop cloud-based Big Data solutions using AWS services.Develop and manage...

  • Senior Big Data Engineer

    4 weeks ago

    Lahore, Punjab, Pakistan Avanceon Middle East & South Asia Full time

    As a Big Data engineer, you will be responsible for data-related implementation tasks that include provisioning of data storage services, ingesting, streaming and batch data, transforming data, implementing security requirements, implementing data retention policies, identifying performance bottlenecks, and accessing external data. You will develop and...

  • Big Data Architect

    2 weeks ago

    Lahore, Punjab, Pakistan Devsinc Full time

    Job Title: Senior Data SpecialistDevsinc is seeking a highly skilled Senior Data Specialist to design, implement, and maintain efficient data pipelines. This position requires expertise in data modeling, ETL frameworks, big data tools, and programming languages.Key ResponsibilitiesData Pipeline Development: Design and implement scalable and secure data...

  • Big Data Technology Lead

    3 days ago

    Lahore, Punjab, Pakistan Avanceon Limited Full time

    We are looking for a senior Big Data Technology Lead to drive our data engineering efforts forward.The successful candidate will have extensive experience with big data technologies and a strong understanding of data lake architecture. Key responsibilities include implementing lambda architecture with stream processing and batch processing, as well as...

  • Cloud Big Data Specialist

    6 days ago

    Lahore, Punjab, Pakistan NorthBay Solutions Full time

    We are looking for a talented Cloud Big Data Specialist to join our team at NorthBay Solutions. The ideal candidate will have 4 to 6+ years of experience in cloud-based big data technologies.About the Role:Main Responsibilities: Design and develop cutting-edge big data technologies to build critical, highly complex distributed systems from scratch.Required...

  • Senior Data Engineer for Cloud Platforms

    2 weeks ago

    Lahore, Punjab, Pakistan NorthBay Solutions LLC Full time

    NorthBay Solutions LLC is hiring a skilled Senior Data Engineer for Cloud Platforms. As a Senior Big Data Engineer, you will be responsible for designing and developing cloud-based Big Data solutions using AWS services such as EMR, Glue, Lambda, and Athena.About the Job:Develop and manage complex ETL jobs and pipelines in AWS-based environments.Collaborate...

  • Big Data Specialist

    2 weeks ago

    Lahore, Punjab, Pakistan Devsinc, LLC Full time

    The Senior Data Engineer will play a vital role in shaping our data strategy. We seek an experienced professional who can design, develop, and maintain large-scale data pipelines and databases, as well as collaborate with cross-functional teams to ensure seamless data integration.Key Responsibilities:Data Pipeline Development: Design, develop, and maintain...

  • Backend Software Engineer

    1 week ago

    Lahore, Punjab, Pakistan Big Data Inc Full time

    Company Overview:Big Data Inc is a cutting-edge technology company that specializes in big data processing and analysis. Our mission is to provide innovative solutions to businesses and organizations.About the Role:The incumbent will be responsible for designing, testing, and implementing large-scale enterprise applications using Java back-end...

  • Big Data Infrastructure Specialist

    2 weeks ago

    Lahore, Punjab, Pakistan Devsinc Full time

    As a Big Data Infrastructure Specialist, you will play a vital role in designing, developing, and maintaining our big data infrastructure. You will work closely with our cross-functional teams to ensure that our data systems are scalable, reliable, and efficient.Main Responsibilities:Designing and Developing Big Data Infrastructure for High Performance and...

  • Data Engineer

    2 weeks ago

    Lahore, Punjab, Pakistan Crypthon Technologies Full time

    We're looking for a Data Engineer who thrives on building scalable data solutions, designing robust pipelines, and transforming complex data into actionable insights.Responsibilities:Build and maintain data pipelines and architectureCollaborate with teams to define data requirementsOptimize systems for performance and scalabilityWork with big data tools and...

  • Azure Big Data Solutions Engineer

    2 weeks ago

    Lahore, Punjab, Pakistan Confiz Full time

    Big Data Engineer RoleAs a data engineer at Confiz, you will play a crucial role in designing and implementing big data pipelines that enable seamless data sharing across our systems.Key Responsibilities:Collaborate with solution architects to ensure data pipeline effectiveness.Develop ETL/ELT ingestion pipelines using cloud and on-premises...

  • Data Engineering Leader

    2 weeks ago

    Lahore, Punjab, Pakistan NorthBay Solutions LLC Full time

    We are seeking an experienced Data Engineering Leader to join our team at NorthBay Solutions LLC. As a Senior Big Data Engineer, you will have the opportunity to design and develop cutting-edge Big Data technologies, build critical distributed systems from scratch, and lead complex ETL jobs and pipelines.About the Role:Design and develop using Big Data...