Senior Databricks Administrator

7 days ago

Hyderābād, Sindh, Pakistan Sonatype Full time 1,200,000 - 3,600,000 per year
Sonatype is the software supply chain security company. We provide the world's best end-to-end software supply chain security solution, combining the only proactive protection against malicious open source, the only enterprise grade SBOM management and the leading open source dependency management platform. This empowers enterprises to create and maintain secure, quality, and innovative software at scale.
As founders of Nexus Repository and stewards of Maven Central, the world's largest repository of Java open-source software, we are software pioneers and our open source expertise is unmatched. We empower innovation with an unparalleled commitment to build faster, safer software and harness AI and data intelligence to mitigate risk, maximize efficiencies, and drive powerful software development.
More than 2,000 organizations, including 70% of the Fortune 100 and 15 million software developers, rely on Sonatype to optimize their software supply chains.
Role Summary
  • The Databricks Administrator will be responsible for the overall health, security, and performance of the Databricks platform. This includes managing user access, implementing and enforcing data governance policies, optimizing cluster resources, and ensuring data sensitivity policies are effectively applied across the data lakehouse. The administrator will also be crucial in identifying, reporting, and resolving discrepancies within the platform's operation and configuration.
Key Responsibilities
  • User Provisioning and Management:
  • Onboard and offboard users, groups, and service principals within Databricks, including integration with identity providers (IdPs) like Azure Active Directory or Okta via SCIM.
  • Manage user roles and entitlements at both the account and workspace levels (Account Admins, Workspace Admins, Metastore Admins, etc.).
  • Implement and maintain role-based access control (RBAC) and attribute-based access control (ABAC) to ensure appropriate data and resource access.
  • Data Lake Governance (Unity Catalog focus):
  • Configure and manage Unity Catalog metastores, catalogs, schemas, and tables.
  • Define and enforce data access policies (e.g., table-level, column-level, row-level security) using Unity Catalog.
  • Manage data lineage and auditing capabilities to track data flow and usage.
  • Collaborate with data owners and stakeholders to define data quality standards and ensure data integrity.Implement data retention and lifecycle management policies.
  • Aligning Data Sensitivity Policy to Enforceable Data Governance:
  • Translate organizational data classification and sensitivity policies into technical controls within Databricks.
  • Utilize features like data masking and encryption to protect sensitive information.
  • Ensure compliance with regulatory requirements (e.g., GDPR, HIPAA, CCPA) by implementing appropriate security measures.
  • Conduct regular security audits and vulnerability assessments.
  • Managing Cluster and Budget Policies:
  • Define and implement compute policies to control cluster creation, configuration, and resource usage, ensuring cost optimization.
  • Monitor and manage serverless budget policies to attribute usage to specific teams or projects.
  • Optimize cluster configurations for performance and cost-effectiveness, leveraging features like auto-scaling and auto-termination.
  • Manage cluster pools to reduce startup times and improve resource allocation.
  • Reporting and Addressing Discrepancies:
  • Monitor Databricks platform health, performance, and resource utilization.
  • Identify and troubleshoot issues related to user access, data availability, cluster performance, and policy violations.
  • Generate reports on platform usage, costs, security incidents, and compliance.
  • Investigate and resolve discrepancies in data, reports, or system behavior in collaboration with data engineers, data scientists, and other teams.
  • Develop and maintain comprehensive documentation of configurations, procedures, and best practices.
  • Collaboration and Support:
  • Provide technical support and guidance to Databricks users, data engineers, and data scientists.
  • Collaborate with cloud infrastructure teams (AWS, Azure, GCP) to manage underlying cloud resources.
  • Stay up-to-date with the latest Databricks features, best practices, and industry trends.
Technical Skills:
  • Databricks Platform Expertise:
  • Deep understanding of Databricks architecture, workspaces, and key components (Unity Catalog, Delta Lake, Spark, SQL Analytics).Proficiency in Databricks administration console and APIs.
  • Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT) for orchestration and pipeline management.
  • Cloud Platform Knowledge: 
  • Strong experience with AWS and its relevant services.
  • Data Governance & Security:
  • Solid understanding of data governance principles, data classification, and data lifecycle management.
  • Experience implementing security controls, access policies (RBAC), and encryption.
  • Familiarity with compliance standards (GDPR, HIPAA, CCPA) and auditing practices.
  • Programming & Scripting:
  • Proficiency in SQL for data querying and access control.
  • Deep expertise in Terraform is essential, extending beyond basic knowledge to managing complex, multi-project infrastructure. This includes hands-on experience with custom Terraform modules crucial for Data Mesh orchestration.
  • Scripting skills (e.g., Python, Terraform) for automation and administrative tasks.
  • Familiarity with Spark and PySpark concepts for troubleshooting and optimization.
  • Identity and Access Management (IAM): 
  • Experience with enterprise identity providers (e.g., Azure AD, Okta, Active Directory) and SCIM provisioning.
  • Networking Concepts: 
  • Understanding of network security, VPNs, VPCS, private links, VPC peering, and connectivity within cloud environments.
  • Monitoring & Logging Tools: 
  • Experience with monitoring tools (e.g., Datadog, Observe, cloud-native monitoring) for platform health and performance.
Soft Skills
  • Problem-Solving and Troubleshooting: Ability to diagnose and resolve complex technical issues efficiently.
  • Communication: Excellent verbal and written communication skills to interact with technical and non-technical stakeholders.
  • Attention to Detail: Meticulous in configuring policies, managing access, and ensuring data integrity.
  • Proactive and Self-Driven: Ability to anticipate issues, recommend solutions, and continuously improve the platform.
  • Collaboration: Work effectively with cross-functional teams (data engineers, data scientists, security teams).
  • Analytical Thinking: Ability to analyze data and system logs to identify trends and discrepancies.
At Sonatype, we value diversity and inclusivity. We offer perks such as parental leave, diversity and inclusion working groups, and flexible working practices to allow our employees to show up as their whole selves. We are an equal-opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. If you have a disability or special need that requires accommodation, please do not hesitate to let us know.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.

  • Databricks with Python

    5 days ago

    Hyderābād, Sindh, Pakistan Tata Consultancy Services (TCS) Full time 60,000 - 120,000 per year

    Role & responsibilitiesMust HaveExtensive expertise in designing and implementing data load processes using Azure Data Factory, Azure Databricks, Delta Lake, Azure Delta Lake Storage and Python/PySparkProficient with Databricks & PythonSenior developers with Full Database/ Datawarehouse/ DataMart development capabilitiesSenior SME with SQL Server...

  • Senior Project Manager

    7 days ago

    Hyderābād, Sindh, Pakistan Prolifics Full time $600,000 - $800,000 per year

    OverviewAt Prolifics, we are currently implementing multiple solutions and we are looking to hire talented Senior Project Manager for our development centre in India. This position would be based out of Hyderabad and is a permanent position.If you are looking for a high growth company with rock-solid stability, if you thrive in the energetic atmosphere of...

  • Senior Data Engineer

    5 days ago

    Hyderābād, Sindh, Pakistan Appen Full time 1,200,000 - 2,400,000 per year

    About Appen Appen is a leader in AI enablement for critical tasks such as model improvement, supervision, and evaluation. To do this we leverage our global crowd of over one million skilled contractors, speaking over 180 languages and dialects, representing 130 countries. In addition, we utilize the industry's most advanced AI-assisted data annotation...

  • Senior Data Engineer

    5 days ago

    Hyderābād, Sindh, Pakistan Egen Solutions Inc Full time

    Job Overview:We are seeking a highly skilled Senior Data Engineer with deep expertise in Microsoft Azure data ecosystem to design, develop, and maintain scalable data pipelines and architectures. The ideal candidate will play a key role in building robust data solutions that support advanced analytics, BI, and AI workloads across the organization. This role...

  • Senior Data Engineer

    5 days ago

    Hyderābād, Sindh, Pakistan Egen Full time 1,500,000 - 3,000,000 per year

    Job Overview: We are seeking a highly skilled Senior Data Engineer with deep expertise in Microsoft Azure data ecosystem to design, develop, and maintain scalable data pipelines and architectures. The ideal candidate will play a key role in building robust data solutions that support advanced analytics, BI, and AI workloads across the organization. This...

  • Senior Consultant

    5 days ago

    Hyderābād, Sindh, Pakistan TTEC Digital Full time 900,000 - 1,200,000 per year

    • 4-6 years in analytics domain developing & deploying ML/AI solutions • 1+ years working on NLP projects – text mining, having used LLMs, worked on Hugging Face/Transformer models; worked on publicly available LLM APIs • Proficient level of understanding in machine learning and deep learning methods • Excellent programming skills in Python,...

  • Senior Executive

    7 days ago

    Hyderābād, Sindh, Pakistan Bhavna Corp. Full time 900,000 - 1,200,000 per year

    Company DescriptionBhavna Corp. is driven by a passion to deliver the most innovative and cost-effective solutions to its clients worldwide. We provide a combination of engineering consultancy, offshore engineering services, and turnkey solutions. Bhavna Corp. has a strong track record of reducing costs, improving quality, and accelerating time to market. We...

  • Senior Data Engineer

    5 days ago

    Hyderābād, Sindh, Pakistan Egen Full time 900,000 - 1,200,000 per year

    Job Overview:   We are looking for a skilled and motivated Senior Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team. The ideal candidate will be responsible for designing, developing, and maintaining robust and scalable ETL (Extract, Transform, Load) data pipelines. The role...

  • Senior Data Scientist

    7 days ago

    Hyderābād, Sindh, Pakistan Sonatype Full time 120,000 - 240,000 per year

    Sonatype is the software supply chain security company. We provide the world's best end-to-end software supply chain security solution, combining the only proactive protection against malicious open source, the only enterprise grade SBOM management and the leading open source dependency management platform. This empowers enterprises to create and maintain...

  • Senior Recruitment Specialist

    7 days ago

    Hyderābād, Sindh, Pakistan Oxiliry Full time 450,000 - 600,000 per year

    Company DescriptionOxiliry connects talented individuals with opportunities by providing global businesses with reliable staffing solutions, ethical HR practices, and employee-focused strategies. Committed to fostering workplaces where people can thrive, Oxiliry works to link talent with purpose and support teams with integrity. With operations spanning...