Review and translate requirements to end-to-end technical design with a goal of building data products.

Develop pipelines in Databricks using PySpark and DBT to onboard various sources.

Working knowledge in Kafka to stream the data into the Databricks platform.

Identify common approaches and best practices to data integration, such as metadata-driven ingestion, data validations, and complex data type handling.

Drive standards for reusability while developing pipelines.

Create or update documentation associated with key decisions on requirements, design, and development.

Liaise with architects, data owners, and users for reviews on a day-to-day basis to ensure the quality of deliverables is high.

Skill Proficiency Requirements:
• 8+ years of overall IT experience with 6 years as a Data Engineer
• Experience in providing data engineering solutions
• Working knowledge of the following technology toolsets:
• Cloud (Azure/AWS)
• Databricks, Data Mesh, Data as a Product
• Python
• Data streaming (Kafka, EventHub, Kinesis, SNS)
• Cloud-based API management – good to have
• Immuta – good to have
• Multi-cloud and on-prem application integration – good to have