Job Description

Role Descriptions:

• Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
• Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala
• Implement and support Data Lake / Lakehouse architectures
• Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.
• Expertise in handling Data migration using API’s.
• Apply data quality checks and support data governance basics (metadata, lineage, access controls)
• Optimize Spark jobs for performance, reliability, and cost
• Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers

Required Skills & Experience:

• 8+ years of experience in data engineering in azure using Azure Databricks.
• Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
• Strong Data Engineering experience in Big Data (Cloudera)
• Hands-on expertise in SQL, Python and PySpark or Scala
• Proven experience with batch & real-time ingestion
• Experience with Kafka and big data processing

Added good to have experience with:

• Informatica PowerCenter
• Informatica IDMC or equivalent cloud data integration tools
• SQL Server, Oracle, Azure SQL
• Cosmos DB
• Unix/Linux
• Maestro (or similar enterprise schedulers)
• Data tools such as Precisely or equivalent

Added Advantage

• Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)
• Experience with Azure DevOps (AzDO) and GitHub-based CI/CD
• Practical usage of Copilot for data engineering and development productivity
• Exposure to Generative AI / GenAI use cases in data platforms and analytics

Job Tags

Similar Jobs