Senior Data Engineer

5 hours ago

Hyderabad City Taluka, Pakistan FANATICS INC Full time

Job Overview:
We are looking for a Senior Data Engineer with a deep understanding of Apache Spark (Scala & PySpark), Kafka Streams (Java), AWS services, Snowflake, Apache Iceberg, Tableau, and Data Lake architectures. As a senior member of our team, you will be responsible for leading the design, implementation, and optimization of large-scale data systems, real-time streaming solutions, and cloud-based data platforms. You will work with other engineers to deliver high-quality data solutions, mentor junior team members, and collaborate closely with cross-functional teams to solve complex business problems.

Key Responsibilities:
  • Lead the design and development of scalable, high-performance data architectures on AWS, leveraging services such as S3, EMR, Glue, Redshift, Lambda, and Kinesis. Architect and manage Data Lakes for handling structured, semi-structured, and unstructured data.
  • Design and build complex data pipelines using Apache Spark (Scala & PySpark), Kafka Streams (Java), and cloud-native technologies for batch and real-time data processing. Optimize these pipelines for high performance, scalability, and cost-effectiveness.
  • Develop and optimize real-time data streaming applications using Kafka Streams in Java. Build reliable, low-latency streaming solutions to handle high-throughput data, ensuring smooth data flow from sources to sinks in real-time.
  • Manage Snowflake for cloud data warehousing, ensuring seamless data integration, optimization of queries, and advanced analytics. Implement Apache Iceberg in Data Lakes for managing large-scale datasets with ACID compliance, schema evolution, and versioning.
  • Design and maintain highly scalable Data Lakes on AWS using S3, Glue, and Apache Iceberg. Ensure data is easily accessible, stored in optimal formats, and well-integrated with downstream analytics systems.
  • Work with business stakeholders to create actionable insights using Tableau. Build data models and dashboards that drive key business decisions, ensuring that data is easily accessible and interpretable.
  • Continuously monitor and optimize Spark jobs, Kafka Streams processing, and other cloud-based data systems for performance, scalability, and cost. Implement best practices for stream processing, batch processing, and cloud resource management.
  • Lead and mentor junior engineers, fostering a culture of collaboration, continuous learning, and technical excellence. Ensure high-quality code delivery, adherence to best practices, and optimal use of resources.
  • Work closely with Data Scientists, Product Managers, and DevOps teams to understand business needs and deliver impactful data solutions. Participate in technical discussions, from system design to data governance.
  • Ensure that data pipelines, architectures, and systems are thoroughly documented and follow coding and design best practices. Promote knowledge-sharing across the team to maintain high standards for quality and scalability.
Required Skills & Qualifications:Education:

Bachelor's or Master's degree in Computer Science, Engineering, or related field (or equivalent work experience).

Experience:
  • 5+ years of experience in Data Engineering or a related field, with a proven track record of designing, implementing, and maintaining large-scale distributed data systems.
  • Proficiency in Apache Spark (Scala & PySpark) for distributed data processing and real-time analytics.
  • Hands-on experience with Kafka Streams using Java for real-time data streaming applications.
  • Strong experience in Data Lake architectures on AWS, using services like S3, Glue, EMR, and data management platforms like Apache Iceberg.
  • Proficiency in Snowflake for cloud-based data warehousing, data modeling, and query optimization.
  • Expertise in SQL for querying relational and NoSQL databases, and experience with database design and optimization.
Technical Skills:
  • Strong experience in building ETL pipelines using Spark (Scala & PySpark) and maintain them.
  • Proficiency in Java, particularly in the context of building and optimizing Kafka Streams applications for real-time data processing.
  • Experience with AWS services (e.g., Lambda, Redshift, Athena, Glue, S3) and managing cloud infrastructure.
  • Expertise with Apache Iceberg for handling large-scale, transactional data in Data Lakes, supporting versioning, schema evolution, and partitioning.
  • Experience with Tableau for business intelligence, dashboard creation, and data visualization is a plus.
  • Knowledge of CI/CD tools and practices, particularly in data engineering environments.
  • Familiarity with containerization tools like Docker and Kubernetes for managing cloud-based services.
Soft Skills:
  • Excellent problem-solving skills, with a strong ability to debug and optimize large-scale distributed systems.
  • Strong communication skills to engage with both technical and non-technical stakeholders.
  • Proven leadership ability, including mentoring and guiding junior engineers.
  • A collaborative mindset and the ability to work across teams to deliver integrated solutions.
Preferred Qualifications:
  • Experience with stream processing frameworks like Apache Flink or Apache Beam.
  • Knowledge of machine learning workflows and integration of ML models in data pipelines.
  • Familiarity with data governance, security, and compliance practices in cloud environments.
  • Experience with DevOps practices and infrastructure automation tools such as Terraform or CloudFormation.
#J-18808-Ljbffr

  • Senior Data Engineer

    12 hours ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Job DescriptionWe are seeking a Senior Data Engineer to lead the design, development, and maintenance of our Data as a Service (DaaS) platform on Azure. The ideal candidate will have a strong background in data engineering, cloud data platforms, ETL processes, and data governance.Key ResponsibilitiesData Platform Architecture & Implementation: Design,...

  • Senior Data Engineering Manager

    2 weeks ago

    Hyderabad City Taluka, Pakistan DigitalOcean LLC Full time

    About the RoleWe are seeking a highly skilled Senior BI Engineer to join our data engineering team at DigitalOcean LLC. As a key member of our team, you will be responsible for designing and implementing data architectures and infrastructure that support advanced analytics and reporting needs.Key Responsibilities:Data Architecture & Modeling: Develop and...

  • Senior Data Platform Engineer

    6 days ago

    Hyderabad City Taluka, Pakistan PURVIEW Full time

    About Our TeamPurview's team consists of highly skilled professionals with extensive experience in cloud and data engineering.We are seeking an experienced Senior DBA to join our team and contribute to our success.ResponsibilitiesWork in Sharding Environments.Manage MongoDB backup and recovery.Use OPs Manager.About CompanyPurview offers a wide range of...

  • Data Engineering Leader

    12 hours ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    About the RoleThe Senior Data Engineer will be responsible for leading the design, development, and maintenance of our Data as a Service (DaaS) platform on Azure. This includes designing and implementing secure and scalable data infrastructure, developing and managing data pipelines, and ensuring data consistency, quality, and governance.Requirements5+ years...

  • Engineering Director

    1 day ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Job OverviewThe Engineering Director - Data Platforms will lead the technical direction for our data governance, data quality, and privacy engineering systems. This role requires a strong understanding of data platforms, distributed systems, and cloud architecture.Key Responsibilities:Develop and implement a technical roadmap for data governance, data...

  • Senior Data Science Engineer

    5 days ago

    Hyderabad City Taluka, Pakistan Warner Bros. Discovery, Inc. Full time

    At Warner Bros. Discovery, we're on a mission to push the boundaries of what's possible in the world of data engineering.About the Role:This position is part of our Growth Engineering team, responsible for designing and developing the data platform to efficiently address various data needs across the business. You'll collaborate with cross-functional teams...

  • Software Data Systems Engineer

    7 days ago

    Hyderabad City Taluka, Pakistan Qualcomm Technologies, Inc Full time

    Requirements:We are seeking a highly skilled Senior Big Data Engineer with 2+ years of academic or work experience with programming languages such as C, C++, Java, Python, etc. The ideal candidate will have a bachelor's degree in engineering, information systems, computer science, or a related field. Experience with data modeling, data warehousing, and...

  • Head of Engineering

    1 day ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Job Title: Head of Engineering - Data SolutionsWe are seeking an experienced Head of Engineering - Data Solutions to lead our data engineering team. As a key member of our leadership, you will be responsible for developing and implementing a technical roadmap for data governance, data quality, and privacy engineering systems.Key Responsibilities:Lead a team...

  • Cloud Architect

    5 days ago

    Hyderabad City Taluka, Pakistan Warner Bros. Discovery, Inc. Full time

    Are you ready to take your data engineering skills to the next level? Join our team at Warner Bros. Discovery as a Senior Data Engineer and contribute to shaping the future of data-driven experiences.About the Team:We are a diverse group of data enthusiasts who thrive in a fast-paced, agile environment. Our team consists of data engineers, data scientists,...

  • Senior Software Engineer, Data Ingestion

    4 weeks ago

    Hyderabad City Taluka, Pakistan SingleStore Full time

    Senior Software Engineer, Engine (Data Ingestion) -Position OverviewSingleStore Software Engineers are at the forefront of real-time technology that's revolutionizing how companies run their businesses every day. With release cycles throughout the year, our engineers are accustomed to moving quickly in a rapid production environment. To do this successfully,...

  • Senior Software Engineer, Data Ingestion

    2 weeks ago

    Hyderabad City Taluka, Pakistan SingleStore Full time

    Senior Software Engineer, Engine (Data Ingestion) -Position OverviewSingleStore Software Engineers are at the forefront of real-time technology that's revolutionizing how companies run their businesses every day. With release cycles throughout the year, our engineers are accustomed to moving quickly in a rapid production environment. To do this successfully,...

  • Senior Big Data Engineer

    7 days ago

    Hyderabad City Taluka, Pakistan Qualcomm Technologies, Inc Full time

    Company: Qualcomm India Private LimitedJob Area: Engineering Group, Engineering Group > Software EngineeringGeneral Summary:As a leading technology innovator, Qualcomm pushes the boundaries of what's possible to enable next-generation experiences and drives digital transformation to help create a smarter, connected future for all. As a Qualcomm Software...

  • Data Engineering Lead

    5 days ago

    Hyderabad City Taluka, Pakistan Warner Bros. Discovery, Inc. Full time

    Welcome to Warner Bros. Discovery, where innovation meets storytelling.Job Description:We are seeking a Senior Data Engineer to join our Growth Engineering team, which comprises data engineers, data scientists, ML engineers, and software engineers. Our goal is to enhance customer happiness by quickly creating experiments that improve our main acquisition and...

  • Cloud Data Platform Engineer

    3 days ago

    Hyderabad City Taluka, Pakistan Backbase Full time

    About the RoleWe are seeking a highly skilled Senior System Engineer to join our team and contribute to building an enterprise-grade "Data as a Service" platform from scratch.This role involves designing, implementing, and maintaining the infrastructure of our Data platform on public cloud (Azure). You will work closely with cross-functional teams to ensure...

  • Data Engineering Lead

    2 weeks ago

    Hyderabad City Taluka, Pakistan Warner Bros. Discovery, Inc. Full time

    Welcome to a world of limitless possibilities, where technology and creativity come together to shape the future. At Warner Bros. Discovery, we are on a mission to drive innovation and excellence in all aspects of our business.As a Senior Analytics Engineer, you will play a critical role in leading our data pipeline, strategy, and visualization efforts for...

  • Senior Software Data Architect

    7 days ago

    Hyderabad City Taluka, Pakistan Qualcomm Technologies, Inc Full time

    Job Description:The Senior Big Data Engineer will design, develop, create, modify, and validate embedded and cloud edge software, applications, and/or specialized utility programs that launch cutting-edge, world-class products meeting and exceeding customer needs. As a key member of our engineering team, you will collaborate with systems, hardware,...

  • Analytical Data Engineering Lead

    1 week ago

    Hyderabad City Taluka, Pakistan techcarrot Full time

    About the PositionWe are seeking an experienced Analytical Data Engineering Lead to lead our data engineering team at techcarrot. As a key member of our analytics department, you will be responsible for designing, developing, and maintaining cutting-edge data visualization solutions that empower business-driven decision-making.This role requires a strong...

  • Data Engineer

    1 day ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    About the RoleThe Data and Analytics organisation owns and delivers the Data and AI platform to equip markets with improved decision-making capabilities. The team consists of six key areas: Platform & Architecture, Data Science, Data Engineering, Data Visualisation, Data Governance, and Strategy & Operations.

  • Data Engineer with Cloud Expertise

    2 weeks ago

    Hyderabad City Taluka, Pakistan DigitalOcean LLC Full time

    About the RoleWe are seeking an experienced Senior BI Engineer to join our dynamic team at DigitalOcean LLC. As a key member of our data engineering team, you will play a critical role in designing and implementing cutting-edge data architectures and infrastructure to support advanced analytics and reporting needs.Key Responsibilities:Data Architecture &...

  • Data Engineering Manager

    1 day ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    About the RoleWe are seeking an experienced Data Engineering Manager to lead our analytics engineering team. This is a pivotal role in driving the data and analytics strategy across various streaming and studio data domains, including commerce, product, and content analytics.The successful candidate will have 14+ years of experience in data engineering, data...