Research Scientist, Generalist Embodied Agent Research

About NVIDIA's Generalist Embodied Agent Research (GEAR) Group

NVIDIA is seeking an exceptional research scientist to contribute to building humanoid robot foundation models and systems within the Generalist Embodied Agent Research (GEAR) group. Our core belief is that all moving entities will eventually achieve autonomy. Our mission drives us to create general-purpose embodied agents capable of exploring and mastering intricate skills across both virtual and physical environments.

You will join a highly collaborative and renowned research team known for its impactful contributions in multimodal foundation models, large-scale robot learning, game AI, and physical simulation. Notable past projects include Eureka, VIMA, Voyager, MineDojo, MimicPlay, and Prismer. The team recently achieved a significant milestone with Project GR00T, a pioneering foundation model designed for humanoid robots. Your work will directly influence our ambitious research initiatives and future product roadmaps.

What You'll Be Doing as a Research Scientist, Generalist Embodied Agent Research

• Design and implement novel AI algorithms and models specifically for general-purpose humanoid robots and embodied agents.
• Develop large-scale AI training and inference methods crucial for foundation models.
• Optimize and deploy advanced AI models within physical simulations and directly onto robot hardware.
• Engage in close collaboration with research and engineering teams throughout NVIDIA to seamlessly transition groundbreaking research into tangible products and services.

What We Need To See

• A Ph.D. in Computer Science/Engineering, Electrical Engineering, or equivalent advanced research experience.
• Exceptional engineering proficiency in rapid prototyping and experience with leading model training frameworks such as PyTorch, Jax, or TensorFlow.
• Required programming skills in Python or C++; CUDA or ROS proficiencies are a significant advantage.
• Extensive knowledge and practical experience in training LLMs, multimodal foundation models, or large generative models.
• Proven experience in foundation and diffusion models, reinforcement learning, agent learning, and applied robotics.
• Demonstrated expertise in one or both of the following specialized fields:

Multimodal Foundation Models

• Hands-on training experience and relevant publications in at least one of these areas: Large Language Models (LLMs); large vision-language models; video generative models and diffusion algorithms; or action-based transformers.
• Excellent capabilities in managing large-scale machine learning/AI systems and high-performance computing infrastructure.

Robotics

• Practical training experience and publications in robot learning, encompassing reinforcement learning, imitation learning, and classical control methods.
• A deep understanding of robot kinematics, dynamics, and sensor technologies.
• The ability to safely operate robot hardware, lab equipment, and specialized tools.
• Knowledge of various control methods, including PID, model predictive control, and whole-body control.
• Familiarity with leading physics simulation frameworks such as MuJoCo and Isaac Sim.
• Experience in robot hardware design and hands-on building.

NVIDIA is celebrated as one of the technology industry's most sought-after employers, home to some of the world's most visionary and productive individuals. Join us at the forefront of developing general-purpose robots and embodied agents!

Key skills/competency

• AI algorithms
• Humanoid robotics
• Foundation models
• Large-scale machine learning
• PyTorch framework
• Robot learning
• Multimodal models
• Physics simulation
• C++ programming
• Reinforcement learning