Design and implement scalable ETL frameworks to ingest, transform, and extract data from various source systems into Hive, PostgreSQL, and Redshift, generating business-driven extracts. Develop automated data ingestion pipelines for processing different file types (TXT, CSV) from AWS S3, dynamically loading them into Hive, PostgreSQL, Redshift, and Databricks based on project requirements. Lead database migration efforts, migrating SQL databases to AWS Redshift, leveraging Apache Spark for data transformation and AWS Athena for efficient querying of S3-stored data. Implement Type-1 and Type-2 Slowly Changing Dimensions (SCD) in ETL pipelines, ensuring historical data accuracy and compliance by handling overwrites (Type-1) and preserving versioned records (Type-2). Design and develop Spark-based ETL pipelines to migrate data from AWS Redshift to Databricks, applying complex transformations using PySpark/Scala and optimizing job performance. Work with AWS S3, Python, Scala, Spark, Spark SQL, AWS Athena, PostgreSQL, AWS Redshift, Databricks, Stonebranch, Snowflake, Eclipse, VS Code, Jira, Jenkins, Teradata, DB2, and GIT. Worksite: Relocation for short/long term projects to client sites at varying unanticipated locations throughout the US required. Master's degree plus 2 years of experience required.
