Job Description:
Senior Data Engineer positions offered by StockX LLC (Detroit, MI). Design and build mission critical data pipelines with a highly scalable distributed architecture - including data ingestion (streaming, events and batch), data integration, data curation. Help continually improve ongoing reporting and analysis processes, simplifying self-service support for business stakeholders. Build and support reusable framework to ingest, integration and provision data. Automation of end to end data pipeline with metadata, data quality checks and audit. Build and support a big data platform on the cloud. Define and implement automation of jobs and testing. Optimize the data pipeline to support ML workloads and use cases. Support mission critical applications and near real time data needs from the data platform. Capture and publish metadata and new data to subscribed users. Work collaboratively with business analysts, product managers, data scientists as well as business partners and actively participate in design thinking session. Participate in design and code reviews. Motivate, coach, and serve as a role model and mentor for other development team associates/members that leverage the platform. May work remotely from anywhere in the U.S.
Minimum Requirements:
Must have at least a Bachelor’s degree, or foreign equivalent, in computer science or related technical field, and at least 5 years of experience in the job offered or at least 5 years of experience in data warehouse or data lake technical architecture. Stated experience must include the following: 3 years of experience using programming languages (Python / Scala / Java / C#) to build data pipelines; 3 years of Big Data and Big Data tools in one or more of the following: Batch Processing (e.g. Hadoop distributions, Spark), Real time processing (e.g. Kafka, Flink/Spark Streaming); 2 years of experience with AWS or engineering in other cloud environments; Database architecture/schema design, batch processing and workflow tools, such as AirFlow, NiFi; and Technical leadership and mentoring of other engineers with respect to best practices on data engineering. If interested, please send resume to: applications@stockx.com.
