Published in IEEE Int. Conf. Systems, Man, and Cybernetics (SMC 2024), Sarawak, Malaysia, 6-10 Oct., 2024, 2024
This research introduces the Bi-VLA (Vision-Language-Action) model, a novel system designed for bimanual robotic dexterous manipulation that seamlessly integrates vision for scene understanding, language comprehension for translating human instructions into executable code, and physical action generation. We evaluated the system’s functionality through a series of household tasks, including the preparation of a desired salad upon human request. Bi-VLA demonstrates the ability to interpret complex human instructions, perceive and understand the visual context of ingredients, and execute precise bimanual actions to prepare the requested salad. We assessed the system’s performance in terms of accuracy, efficiency, and adaptability to different salad recipes and human preferences through a series of experiments. Our results show a 100% success rate in generating the correct executable code by the Language Module, a 96.06% success rate in detecting specific ingredients by the Vision Module, and an overall success rate of 83.4% in correctly executing user-requested tasks.
Recommended citation: @misc{gbagbe2024bivlavisionlanguageactionmodelbasedbimanual, title={Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulations}, author={Koffivi Fidèle Gbagbe and Miguel Altamirano Cabrera and Ali Alabbas and Oussama Alyunes and Artem Lykov and Dzmitry Tsetserukou}, year={2024}, eprint={2405.06039}, archivePrefix={arXiv}, primaryClass={cs.RO}, url={https://arxiv.org/abs/2405.06039}, }
Download Paper | Download Slides
