Data Engineer with PySpark and Databricks Expertise

Data Engineer with PySpark and Databricks ExpertiseData Engineer with PySpark and Databricks Expertise

Get AI-powered advice on this job and more exclusive features.

Direct message the job poster from Cloud Enterprise Business Solutions (CEBS)

As a Data Engineer, you will play a pivotal role in designing, building, and maintaining large-scale data pipelines using PySpark and Databricks. You will collaborate closely with stakeholders to clarify requirements and suggest improvements, ensuring our data infrastructure is robust, scalable, and efficient. Your interest in AI will contribute to innovative solutions and data products for our 5 million members.

Key Responsibilities:

• Data Pipeline Development: Deliver efficient, large-scale data pipelines using PySpark to process and analyse big data.
• Data Orchestration in Databricks: Build and maintain data orchestration pipelines within the Databricks platform.
• Stakeholder Collaboration: Communicate with stakeholders to clarify requirements, provide insights, and suggest improvements to existing data processes.
• Data Product Migration: Oversee the migration and integration of existing data products from one company to another, ensuring seamless functionality.
• Customer Journey Analytics: Develop and maintain data pipelines for customer event journeys, supporting analytics for our 5 million members.
• Performance Optimisation: Optimise data processing and storage for efficiency and scalability.
• Monitoring and Logging: Implement monitoring and logging solutions to ensure data pipelines run smoothly and issues are promptly identified.
• Security and Compliance: Ensure all data processing complies with security policies and data protection regulations.
• Innovation and AI Integration: Leverage your interest in AI to incorporate advanced analytics and machine learning where appropriate.
• Documentation: Create and maintain comprehensive documentation for data architectures, processes, and procedures.

Requirements:

• Qualification: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent work experience).
• Experience: 3+ years of experience in data engineering, focusing on building large-scale data pipelines.
• Databricks Expertise: At least 2 years of hands-on experience working with Databricks.
• Big Data Technologies: Proficiency with big data technologies and frameworks (e.g., Hadoop, Spark).
• Cloud Platforms: Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Data Modelling: Solid understanding of data modelling, ETL processes, and data warehousing concepts.
• Programming Skills: Proficiency in Python and SQL; experience with other programming languages is a plus.
• Stakeholder Engagement: Proven ability to communicate effectively with stakeholders to gather requirements and provide solutions.
• AI Interest: Demonstrated interest in AI, machine learning, or data science.
• Problem-Solving Skills: Excellent analytical and troubleshooting abilities.
• Communication: Strong verbal and written communication skills.
• Agile Methodologies: Familiarity with Agile/Scrum development processes.
• Security Awareness: Understanding of data security practices and compliance requirements.
• Continuous Learning: A proactive attitude towards continuous learning and staying updated with industry trends.

What's in it for you

• Employees' Provident Fund, medical and other incentives
• Unique working environment where you communicate and work directly with international clients

Seniority level

• Seniority levelMid-Senior level

Employment type

• Employment typeFull-time

Job function

• Job functionInformation Technology
• IndustriesIT Services and IT Consulting

Referrals increase your chances of interviewing at Cloud Enterprise Business Solutions (CEBS) by 2x

Sign in to set job alerts for "Data Engineer" roles.Associate Software Engineer -React NativeFull Stack Engineer- Node.js, React,js and FirebaseSolutions Engineer (Onsite, Islamabad, USD salary)Associate Software Engineer at FoomotionSoftware Developer at iSmile Dental SoftwareBlockchain Developer - Next.js & Bitcoin EcosystemFull Stack Developer - Python & Angular (Onsite, Islamabad, Remittance Salary)

We're unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

#J-18808-Ljbffr

---