Analyze, design, develop, and optimize enterprise data systems and scalable data pipelines using Python and PySpark to support batch and streaming data workloads; gather user business requirements and translate them into technical specifications, system workflows, and data processing solutions; develop and maintain object-oriented Python code and manage software dependencies to ensure reliability and maintainability of data engineering applications; build and support CI/CD pipelines, automated testing, version control, and deployment processes to maintain data pipeline integrity; perform unit and integration testing to ensure accuracy, stability, and performance of data processing solutions; create technical documentation, including process workflows, data flow diagrams, and system architecture artifacts; monitor and troubleshoot data pipeline performance and operational issues; participate in project planning, code reviews, and implementation meetings. May need to work in other locations as required.
