Engineer II- Data Engineer

We are seeking an experienced and motivated Data Engineer to join our team. In this role, you will design, build, and maintain scalable data solutions to support critical business needs. You will work with distributed data platforms, cloud infrastructure, and modern data engineering tools to enable efficient data processing, storage, and analytics. The role includes participation in an on-call rotation to ensure the reliability and availability of our systems and pipelines.

Key Responsibilities

1. Data Platform Development: Design, develop, and maintain data pipelines and workflows on distributed data platforms such as BigQuery, Hadoop/EMR/DataProc, or Teradata.
2. Cloud Integration: Build and optimize cloud-based solutions using AWS or GCP to process and store large-scale datasets.
3. Workflow Orchestration: Design and manage workflows and data pipelines using Apache Airflow to ensure scalability, reliability, and maintainability.
4. Containerization and Orchestration: Deploy and manage containerized applications using Kubernetes for efficient scalability and resource management.
5. Event Streaming: Work with Kafka to implement reliable and scalable event streaming systems for real-time data processing.
6. Programming and Automation: Write clean, efficient, and maintainable code in Python and SQL to automate data processing, transformation, and analytics tasks.
7. Database Management: Design and optimize relational and non-relational databases to support high-performance querying and analytics.
8. System Monitoring & Troubleshooting: Participate in the on-call rotation to monitor systems, address incidents, and ensure the reliability of production environments.
9. Collaboration: Work closely with cross-functional teams, including data scientists, analysts, and product managers, to understand data requirements and deliver solutions that meet business objectives.
10. Participate in code reviews, technical discussions, and team collaboration to deliver high-quality software solutions.

This role includes participation in an on-call rotation to ensure the reliability and performance of production systems:

1. Rotation Schedule: Weekly rotation beginning Tuesday at 9:00 PM PST through Monday at 9:00 AM PST.
2. Responsibilities During On-Call:
3. Monitor system health and respond to alerts promptly.
4. Troubleshoot and resolve incidents to minimize downtime.
5. Escalate issues as needed and document resolutions for future reference.

Requirements:

• Primary Technologies: Big Query or other distributed data platform, for example, Big Data (Hadoop/EMR/DataProc), SnowFlake, Teradata, or Netezza, AWS, GCP, Kubernetes, Kafka, Python, SQL.
• Bachelor's degree in computer science, Engineering, or a related field (or equivalent work experience).
• 3+ years of experience in data engineering or related roles.
• Hands-on experience with distributed data platforms such as BigQuery, Hadoop/EMR/DataProc, Snowflake, or Teradata.
• Proficiency in Apache Airflow for building and orchestrating workflows and data pipelines.
• Proficiency in Python and SQL for data processing and analysis.
• Experience with cloud platforms like AWS or GCP, including building scalable solutions.
• Familiarity with Kubernetes for container orchestration.
• Knowledge of Kafka for event streaming and real-time data pipelines.
• Strong problem-solving skills and ability to troubleshoot complex systems.
• Excellent communication and collaboration skills to work effectively in a team environment.

Preferred:

• Familiarity with CI/CD pipelines for automated deployments.
• Knowledge of data governance, security, and compliance best practices.
• Experience with DevOps practices and tools.

We have a global team of amazing individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making the workplace fun.

People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves.

To know more about Confiz Limited, visit our LinkedIn page.