Senior Big Data Engineer

1 week ago

Karachi, Sindh, Pakistan NorthBay Solutions LLC Full time

We are hiring for Senior Big Data Engineer with solid experience in data engineering and a passion for Big Data. The ideal candidate will have experience of over 4 to 6+ years specifically in Data Engineering.

You will have the chance to join a global, multi-cultural team of Data Engineers, and be part of the growth and development of it, participate in a large variety of exciting and challenging projects and work with our exceptional team of Technology professionals.

Technology Stack Used & Required Knowledge:
  • 4 to 6+ years of experience on ETL Data Engineer, Python, Big Data (Hadoop, spark, hive)
  • Experience with distributed processing technologies and frameworks, such as Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3)
  • Excellent understanding of ETL cycle.
  • Experience in creating and driving large-scale ETL pipelines in AWS-based environment and experience of services such as EMR, Glue, Lambda, Athena
  • Experience with integration of data from multiple data sources.
  • Understanding of data processing pipelines, data management, and data engineering principles, including experience working with large-scale datasets and distributed computing frameworks (e.g., Spark).
  • Good knowledge and understanding of Big data engineering, ETL, Data Modeling, SQL Scripting, and Data Architecture.
  • Strong software development and programming skills with a focus on data using Python, PySpark and/or Scala for data engineering.
  • Experience and understanding of developing and maintaining complex stored procedures
  • Strong proficiency in ETL schedulers such as StepFunction or Apache Airflow
  • Solid understanding of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse)
  • Experience and understanding of various core GCP services such as BigQuery, Cloud Composer, Cloud Storage
  • Knowledge of containerization technologies, CI/CD pipelines and IAC tools like Docker, Jenkins, Terraform for automating software delivery process
  • Experience with the AWS data management tools such as Data lake and Databricks or AWS Snowflake is a plus
  • Understanding of descriptive and exploratory statistics, predictive modeling, evaluation metrics, decision trees, machine learning algorithms is a plus
  • Experience and understanding of AWS data migration services such as SCT, DataSync, Data Extractor will be a plus
  • Should have experience in Jira and Agile development methodologies.
  • Must be open to learning and have the ability to grasp new technologies quickly.
  • Be motivated and passionate about technology.
  • Team player with strong integrity and professionalism who can work well with others in the group.
  • Self-starter with a proven ability to take initiative and strong problem-solving skills.

To keep it short, below are some key responsibilities:

  • Design and develop using Big Data technologies.
  • Design and develop cutting edge Big Data, and Cloud technologies to build critical, highly complex distributed systems from scratch
  • Design, develop and manage complex ETL jobs and pipelines.
  • Act as a team member, consulting other teammates as your new systems transform the way this entire firm leverages data
  • Deploying data models into production environments. This entails providing the model with data stored in a warehouse or coming directly from sources, configuring data attributes, managing computing resources, setting up monitoring tools, etc.
  • Support and help them learn and improve their skills, as well as innovate and iterate on best practices.
  • Solve complex issues and provide guidance to team members when needed.
  • Make improvement and process recommendations that have an impact on the business.
#J-18808-Ljbffr

  • Big Data Expert Lead

    4 days ago

    Karachi, Sindh, Pakistan Al Nafi International College Full time

    About this RoleWe are looking for an exceptional Senior Big Data Specialist/Analyst to lead our data strategy at Al Nafi International College. This individual will play a vital role in developing and implementing data-driven solutions that drive business growth and improvement.The successful candidate will have extensive experience in Big Data engineering,...

  • Senior Big Data Specialist/Analyst

    4 days ago

    Karachi, Sindh, Pakistan Al Nafi International College Full time

    Position Overview:We are seeking a highly skilled and experienced Senior Big Data Specialist/Analyst to join our innovative team. This role is crucial for managing complex data pipelines, integrating data from multiple microservices, and creating actionable dashboards for senior management. The position emphasizes leveraging AIOps processes and procedures,...

  • Big Data Specialist

    2 days ago

    Karachi, Sindh, Pakistan Office Field Full time

    About This OpportunityThis role is ideal for individuals who are passionate about data engineering and enjoy working on complex problems. As a data engineer at Office Field, you will have the opportunity to:Design and implement scalable data architectures.Collaborate with cross-functional teams to deliver high-quality solutions.Stay up-to-date with the...

  • Senior Data Engineer

    5 days ago

    Karachi, Sindh, Pakistan BlueBerry Tech inc Full time

    Bachelor's degree in Computer Science, Engineering, or a related field; Master's degree preferred.BlueBerry Tech Inc., a leading player in the Information Technology and Services industry, is dedicated to advancing businesses through innovative digital solutions. We are located in Karachi and serve as digital transformation partners for startups and...

  • Senior Data Engineer

    2 weeks ago

    Karachi, Sindh, Pakistan traxccel Full time

    We are looking for a highly experienced and motivated Azure Data Engineer to join our team. The successful candidate will have a minimum of 5 years of hands-on experience in Azure and Databricks architecture, migration, and development. This role requires excellent communication skills and the ability to own and self-manage work. The candidate must be able...

  • Data Analytics Manager

    4 days ago

    Karachi, Sindh, Pakistan Al Nafi International College Full time

    Why Join Our TeamAt Al Nafi International College, we are passionate about empowering learners through innovative education and cutting-edge technologies. As a Senior Big Data Specialist/Analyst, you will be part of a dynamic team that values collaboration, innovation, and excellence.You will have the opportunity to work on diverse projects, including data...

  • Senior Data Strategist

    4 days ago

    Karachi, Sindh, Pakistan Al Nafi International College Full time

    Job DescriptionWe are seeking an experienced Senior Big Data Specialist/Analyst to join our innovative team at Al Nafi International College. This role is crucial for managing complex data pipelines, integrating data from multiple microservices, and creating actionable dashboards for senior management.This position emphasizes leveraging AIOps processes and...

  • Data Engineer

    2 days ago

    Karachi, Sindh, Pakistan Office Field Full time

    Direct message the job poster from Office FieldHuman Resources Executive at Office Field | HR Policies Implementation, Tech RecruitingOffice Field is looking for a Data Engineer to join our growing team of analytics experts. They will be responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and...

  • Data Engineer

    3 days ago

    Karachi, Sindh, Pakistan NorthBay Solutions Full time

    Job OverviewWe are seeking a highly skilled Data Engineer - L2 Support to join NorthBay Solutions. This role involves monitoring, troubleshooting, and optimizing AWS-based data pipelines while ensuring data integrity and pipeline continuity. The successful candidate will work with various AWS data services, including Glue, Lambda, Step Functions, Athena,...

  • Senior AWS Data Processor

    3 days ago

    Karachi, Sindh, Pakistan NorthBay Solutions Full time

    Job DescriptionAs a Senior AWS Data Processor at NorthBay Solutions, you will play a critical role in the success of our team by designing, building, and maintaining scalable, efficient, and reliable data pipelines using Amazon Web Services (AWS). Your primary responsibilities will include monitoring and troubleshooting data pipelines, investigating job...

  • Senior Data Engineer

    5 days ago

    Karachi, Sindh, Pakistan BlueBerry Tech inc Full time

    As a Senior DevOps Engineer, you will lead the development of internal tools, shape work environments, and seamlessly integrate IT operations. Your focus on managing CI/CD pipelines ensures reliable software delivery while collaborative efforts tackle technical challenges and optimize system performance. This role requires a strong understanding of...

  • Senior Software Engineer

    2 days ago

    Karachi, Sindh, Pakistan Office Field Full time

    About the RoleThis role involves working on high-impact projects that require the design, development, and maintenance of large-scale data systems. As a senior software engineer in our data engineering team, you will be responsible for:Designing and implementing data pipelines and architectures using Python and other relevant programming languages.Developing...

  • Big Data Integration Expert

    3 days ago

    Karachi, Sindh, Pakistan Pakistan Single Window (PSW) Full time

    About the RoleThis is a mid-senior level full-time position within IT Services and IT Consulting at Pakistan Single Window (PSW). The successful candidate will be responsible for staying up-to-date on the latest data technologies and trends, learning new data tools and technologies, and finding innovative ways to use data to solve business problems.

  • Senior ETL Developer

    4 days ago

    Karachi, Sindh, Pakistan NorthBay Solutions LLC Full time

    Career Advancement OpportunitiesWe offer career advancement opportunities for our employees, including:Senior Data Engineer (L2/L3 Support / ETL Development)Big Data Engineer / AWS Data ArchitectData Engineering Lead / ArchitectAbout UsNorthBay Solutions LLC is a leading provider of cloud-based solutions. We are committed to delivering high-quality services...

  • AWS Data Engineer – L2 Support

    2 weeks ago

    Karachi, Sindh, Pakistan NorthBay Solutions Full time

    We are seeking a Level 2 AWS Data Engineer to join our team. This role involves monitoring, troubleshooting, and optimizing AWS-based data pipelines while ensuring data integrity and pipeline continuity. You will work with AWS Glue, Lambda, Step Functions, and other AWS data services to investigate job failures, re-run ETL jobs, and support production...

  • AWS Data Engineer – L2 Support

    4 days ago

    Karachi, Sindh, Pakistan NorthBay Solutions LLC Full time

    Job OverviewWe are seeking a Level 2 AWS Data Engineer to join our team. This role involves monitoring, troubleshooting, and optimizing AWS-based data pipelines while ensuring data integrity and pipeline continuity. You will work with AWS Glue, Lambda, Step Functions, and other AWS data services to investigate job failures, re-run ETL jobs, and support...

  • Senior Data Architect

    2 weeks ago

    Karachi, Sindh, Pakistan Addo AI Full time

    Get AI-powered advice on this job and more exclusive features.Direct message the job poster from Addo AIAVP Data Engineering at ADDO | Seasoned Solutions Architect | Director Data Engineering | Trusted Advisor | Technical Pre-Sales Advisor | Open Source…Company DescriptionAddo AI is an artificial intelligence advisory and incubator based in Singapore, with...

  • Big Data Expert

    1 week ago

    Karachi, Sindh, Pakistan Horizon Technologies Full time

    Job SummaryHiring for the position of Cloud Data Architect at Horizon Technologies!About UsHorizon Technologies is a cutting-edge technology firm that leverages innovative solutions to drive business growth. As a Cloud Data Architect, you will play a key role in shaping our cloud strategy and architecting scalable data platforms.About the RoleThis role...

  • Data Engineering Expert

    5 days ago

    Karachi, Sindh, Pakistan BlueBerry Tech inc Full time

    Job DescriptionWe are seeking a Senior Data Engineer with 7 years of experience to join our dynamic team, collaborating with cross-functional teams to gather requirements and architect data solutions that improve performance and efficiency.

  • Data Science

    4 hours ago

    Karachi, Sindh, Pakistan SCMDOJO Full time

    Direct message the job poster from SCMDOJOYour Supply Chain Success, My MissionThe CompanySCMDOJO now has more than 12,000 members, 45,000 visitors, and 70,000-page views a month. With 28,200 active Newsletter subscribers, and a tribe of over 102,000 social media followers who are mostly supply chain professionals. All these numbers are growing every day and...