Intro:
Facebook's mission is to give people the power to build community and bring the world closer together. Through our family of apps and services, we're building a different kind of company that connects billions of people around the world, gives them ways to share what matters most to them, and helps bring people closer together. Whether we're creating new products or helping a small business expand its reach, people at Facebook are builders at heart. Our global teams are constantly iterating, solving problems, and working together to empower people around the world to build community and connect in meaningful ways. Together, we can help people build stronger communities - we're just getting started.
Summary:
Facebook is seeking an Engineer to join our Release to Production (RTP) team. Our servers and data centers are the foundation upon which our rapidly scaling infrastructure operates efficiently to deliver our innovative services. The RTP team is responsible for the Hardware Lifecycle of all Facebook servers including pre-production hands-on system and hardware debugging and stress testing, enabling production-ready system monitoring, automated provisioning and automated remediation of issues.RTP Engineers work closely with hardware designers, system manufacturers, component vendors, capacity engineering, production engineering, Facebook services, and data center operations teams to test systems before release to our production data centers, and to track the health and lifecycle of servers in production.
Required Skills:
Interface with external vendors and internal hardware, mechanical, power, thermal, manufacturing and software engineers to understand system architecture to develop and execute the test suites for various architectures
Proactively create experiments and tooling to detect and diagnose hardware/firmware/software health issues
Develop test framework for large-scale test automation inside fleet during product development and after mass production
Implement remediations across software and hardware stack according to plan, while keeping a thorough procedural record and data log
Develop and publish updates on resolutions and communicate findings internally
Troubleshoot, diagnose and root cause of system failures and isolate the components/failure scenarios while working with internal & external stakeholders
Develop visibility through data visualization and implement systemic solutions to hardware health issues
Drive necessary discussion with external and internal teams on test specification and methodologies to improve test quality continuously
Minimum Qualifications:
Bachelor's degree in Engineering or Computer Science
Minimum of 5 years of work experience in one or more of domains such as: compute (x86), storage (SAS), device management (OOBM), AI-ML hardware (GPUs & TPUs), data center network hardware (switching, l2, l3, fabric), interconnect technologies (e.g. - Optics, DAC), power (PSU, battery backup units, AC/DC conversion)
Experience with Linux and scripting
Experience in changing system configurations and measuring change impact
Knowledge of server architecture and components
Troubleshooting and analytical experience
Experience working in a matrix organization
Experience working through full life cycle for different server system/data center products
Experience in post-production hyperscale post-production environments, solutions engineering for different server system/data center products
Preferred Qualifications:
Familiar with lab equipment, protocol analyzers, oscilloscopes, power meters, air flow chambers
4+ years experience in 24x7 Production support at scale (e.g. - 10K storage servers and over 100K HDD)
4+ years experience scripting automation in Python or PHP or Perl
4+ years experience in full system technologies (including PCIe)
Industry: Internet
Equal Opportunity: Facebook is proud to be an Equal Opportunity and Affirmative Action employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics. We also consider qualified applicants with criminal histories, consistent with applicable federal, state and local law. Facebook is committed to providing reasonable accommodations for candidates with disabilities in our recruiting process. If you need any assistance or accommodations due to a disability, please let us know at accommodations-ext@fb.com.