Data engineer- ADF- Business Req

4 weeks ago

Hyderabad City Taluka, Pakistan techcarrot Full time

Job Title: Data
Engineer

Job Summary:

Are you passionate about building scalable data pipelines, optimizing ETL processes, and designing efficient data models? We are looking for a Databricks Data Engineer to join our team and play a key role in managing and transforming data in Azure cloud environments.

In this role, you will work with Azure Data Factory (ADF), Databricks, Python, and SQL to develop robust data ingestion and transformation workflows. You'll also be responsible for integrating SAP IS-Auto data, optimizing performance, and ensuring data quality & governance.

If you have strong experience in big data processing, distributed computing (Spark), and data modeling, we'd love to hear from you

Key Responsibilities:

  1. Develop & Optimize ETL Pipelines:
    • Build robust and scalable data pipelines using ADF, Databricks, and Python for data ingestion, transformation, and loading.
  2. Data Modeling & Systematic Layer Modeling:
    • Design logical, physical, and systematic data models for structured and unstructured data.
  3. Integrate SAP IS-Auto:
    • Extract, transform, and load data from SAP IS-Auto into Azure-based data platforms.
  4. Database Management:
    • Develop and optimize SQL queries, stored procedures, and indexing strategies to enhance performance.
  5. Big Data Processing:
    • Work with Azure Databricks for distributed computing, Spark for large-scale processing, and Delta Lake for optimized storage.
  6. Data Quality & Governance:
    • Implement data validation, lineage tracking, and security measures for high-quality, compliant data.
  7. Collaboration:
    • Work closely with business analysts, data scientists, and DevOps teams to ensure data availability and usability.
  8. Testing and Debugging:
    • Write unit tests and perform debugging to ensure the implementation is robust and error-free.
    • Conduct performance optimization and security audits.

Required Skills and Qualifications:

  1. Azure Cloud Expertise:
    • Strong experience in Azure Data Factory (ADF), Databricks, and Azure Synapse.
  2. Programming:
    • Proficiency in Python for data processing, automation, and scripting.
  3. SQL & Database Skills:
    • Advanced knowledge of SQL, T-SQL, or PL/SQL for data manipulation.
  4. SAP IS-Auto Data Handling:
    • Experience integrating SAP IS-Auto as a data source into data pipelines.
  5. Data Modeling:
    • Hands-on experience in dimensional modeling, systematic layer modeling, and entity-relationship modeling.
  6. Big Data Frameworks:
    • Strong understanding of Apache Spark, Delta Lake, and distributed computing.
  7. Performance Optimization:
    • Expertise in query optimization, indexing, and performance tuning.
  8. Data Governance & Security:
    • Knowledge of RBAC, encryption, and data privacy standards.

Preferred Qualifications:

  1. Experience with CI/CD for data pipelines using Azure DevOps.
  2. Knowledge of Kafka/Event Hub for real-time data processing.
  3. Experience with Power BI/Tableau for data visualization (not mandatory but a plus).
#J-18808-Ljbffr

  • Business Intelligence Engineers- Offshore(Business Req)

    4 weeks ago

    Hyderabad City Taluka, Pakistan techcarrot Full time

    Job Title: Business IntelligenceEngineerJob Summary:We are seeking a highly skilled andexperienced Business Intelligence (BI) Engineer to design, develop, andmaintain cutting-edge BI solutions that empower data-driven decision-makingacross the organization. This role is ideal for individuals with a strongtechnical background in data analytics, visualization,...

  • Data Engineering Leader

    2 weeks ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Job SummaryThe role of Data Engineering Leader involves overseeing the design, development, and implementation of data engineering solutions to support business objectives. This includes ensuring efficient data processing, quality, and compliance with standards.

  • Senior Data Engineer

    6 days ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Unlock the Power of Big DataWe are seeking a skilled Data Engineer to join our team. As a key member, you will be responsible for designing and developing our data platform to efficiently address various data needs across the business.The ideal candidate will have expertise in building and managing large volume data processing (both streaming and batch)...

  • Data Engineering Lead

    1 week ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Data Engineering LeadWe are seeking a Data Engineering Lead to join our team. As a key member of our data engineering group, you will be responsible for leading the design, development, and implementation of large-scale data processing systems.Key Responsibilities:Lead a team of data engineers to design, develop, and implement large-scale data processing...

  • Data Engineering Professional

    2 weeks ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Job Description:We are seeking a highly skilled Data Engineering Professional to design, develop, and maintain large-scale data systems. This individual will collaborate with cross-functional teams to create efficient data pipelines, ensuring seamless integration with various tools and technologies.Main Responsibilities:Develop and implement robust data...

  • Data Engineering Specialist

    6 days ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Data Engineering Role OverviewIn this critical role, you will design and develop the data platform to efficiently and cost-effectively address various data needs across the business. This involves building software across our entire cutting-edge data platform, including event-driven and batch data processing, storage, and serving through various data...

  • Big Data Engineer

    2 weeks ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Data Engineering OpportunitiesJoin us as a Data Engineer to create scalable data solutions that drive business value.You will work with distributed data platforms, cloud infrastructure, and modern data engineering tools to enable efficient data processing, storage, and analytics.Your ResponsibilitiesDevelop and maintain data pipelines and workflows on...

  • Data Engineering Specialist

    2 weeks ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    Job OverviewThe role of a Data Engineer is crucial in turning complex data sources into business insights. As part of our team, you will be responsible for assessing new data sources and implementing them into our Azure Data platform.This position requires a strong analytical mindset, advanced knowledge of relational and non-relational databases, and...

  • Data Engineer Leader

    1 week ago

    Hyderabad City Taluka, Pakistan beBee Careers Full time

    About the RoleWe are seeking a seasoned Principal Data Engineer to join our team and drive the development of complex data management systems. As a key member of our Engineering Team, you will be responsible for designing, building, and maintaining large-scale data infrastructure that meets the needs of our business.ResponsibilitiesLead the development of...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad City Taluka, Pakistan FANATICS INC Full time

    Job Overview:We are looking for a Senior Data Engineer with a deep understanding of Apache Spark (Scala & PySpark), Kafka Streams (Java), AWS services, Snowflake, Apache Iceberg, Tableau, and Data Lake architectures. As a senior member of our team, you will be responsible for leading the design, implementation, and optimization of large-scale data systems,...