Lead Data Engineer - IM

**Job Description** Lead Data Engineer - IM Must Have Technical/Functional Skills Data Bricks , EBT ,Airflow , Asset Management exp Roles & Responsibilities Lead Data Engineer – Information Management (IM) **Experience** 8–15 Years \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ Role Overview We are looking for a highly skilled Lead Data Engineer with strong hands-on experience in Databricks, dbt, and Python, and a proven track record of leading and executing data platform migration and modernization programs. The ideal candidate will have experience migrating legacy data warehouses, ETL platforms, and cloud/on-prem data ecosystems to a modern Databricks Lakehouse architecture using dbt-based transformation frameworks. This is a hands-on leadership role requiring deep technical expertise, solution design capabilities, and the ability to mentor engineering teams while driving enterprise-scale data modernization initiatives. Key Responsibilities Data Engineering & Development - Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python. - Develop and optimize ELT/ETL processes for large-scale data ingestion and transformation. - Implement robust data quality, monitoring, and reconciliation frameworks. - Build reusable data engineering components and frameworks. Data Platform Migration & Modernization - Lead migration of legacy data platforms, data warehouses, and ETL ecosystems to Databricks Lakehouse. - Transform existing ETL workloads to modern ELT patterns using dbt. - Analyze source environments and define migration approaches, roadmap, and execution strategy. - Drive code conversion, performance optimization, and workload modernization efforts. Databricks Engineering **Develop Solutions Leveraging** - Databricks Lakehouse Platform - Delta Lake - Unity Catalog - Databricks Workflows - Structured Streaming - Medallion Architecture (Bronze, Silver, Gold) Optimize Spark jobs and Databricks workloads for performance and cost efficiency. dbt Development - Build and maintain dbt models, macros, tests, and documentation. - Implement incremental processing and reusable transformation frameworks. - Establish data lineage, testing, and CI/CD best practices. - Work closely with business and analytics teams to build trusted datasets. Leadership & Collaboration - Lead a team of data engineers and developers. - Perform code reviews and enforce engineering standards. - Collaborate with arch itects, product owners, business analysts, and stakeholders. - Mentor junior engineers and drive adoption of best practices. Salary Range- $110,000-$140,000 a year