# An open call to test OpenVLA **URL:** \ **Category:** Open Source AI\ **Tags:** process\ **Created:** [June 23, 2024, 9:34pm UTC](https://discuss.opensource.org/t/an-open-call-to-test-openvla/424 "2024-06-23T21:34:50Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![quaid](https://yyz1.discourse-cdn.com/flex011/user_avatar/discuss.opensource.org/quaid/32/277_2.png) [@quaid](https://discuss.opensource.org/u/quaid)\ **Post date:** [June 23, 2024, 9:34pm UTC](https://discuss.opensource.org/t/an-open-call-to-test-openvla/424/1 "2024-06-23T21:34:50Z")
Now that I’m a bit caught-up on many of these discussions — I’ve been sitting on the sidelines looking for signal amidst the unexpected levels of distress displayed — I saw at least one good idea get walked by in another thread, one I believe might help inform the discussion. I also felt the soothing calm of the Debian experience and wisdom that @hartmans ladeled onto these troubled waters. What if we bring to bear the idea that @jasonbrooks raised, which is conducting a controlled experiment on the question, “Do you need a full training dataset or is Data information enough to recreate a model that can be tested for fidelity to the original?” Yes, it is just one experiment and one or maybe several data points. At the least, it may indicate if further experimentation is called for. It may also bring into light what @hartmans has raised, which is the difficulty of deciding _for someone else_ what is the preferred form of modification? So this week I was doing my journalistic due diligence understanding the new OpenVLA project: > **[OpenVLA: An Open-Source Vision-Language-Action Model](https://openvla.github.io/)** > > OpenVLA: An Open-Source Vision-Language-Action Model > **[GitHub - openvla/openvla: OpenVLA: An open-source vision-language-action...](https://github.com/openvla/openvla)** > > OpenVLA: An open-source vision-language-action model for robotic manipulation. - openvla/openvla The due diligence is for a newsletter and the research effort is heavily informed by my work with the OSI on the OSAID and in reading these discussions. For each claim of “open” for an AI project or system I’ve been diving as deep as I can to answer the question, “Is it Open Source AI or not?” In good news for Jason’s experiment idea, I discovered what appears to be an Open Source AI project that has an OSI-licensed publicly available dataset. To double-check, I created a single AI systems review spreadsheet of the kind used by the OSI workgroup recently: > **[quaid AI System Review for OpenVLA](https://docs.google.com/spreadsheets/d/1ouMGc33F20xp57P-3h3jSd011ocmJUiQIsf_XI9XYpU/edit)** > > Definition Validation -- OpenVLA > > Component,Legal Framework,Legal Document,OpenVLA Document Analysis,Freedoms defined: OSAID v. 0.0.8,Notes > Component definitions: Model Openness Framework It is not complete, mainly I wanted to do the first sheet review and see if it appeared to hit all the points. I may have made some mistakes in my understanding of the criteria and my observation, but I reckon the correct answer is out there. These two projects (the model and the dataset) seem to have put a tremendous effort into being and confirming Openness. I spot-checked a half-dozen data sources in the Open X-Embodiment Dataset: > **[Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://robotics-transformer-x.github.io/)** > > Project page for Open X-Embodiment: Robotic Learning Datasets and RT-X Models. > **[Open X-Embodiment Dataset Overview](https://docs.google.com/spreadsheets/d/1rPBD77tk60AEIGZrGSODwyyzs5FgCU9Uz3h-3_t2A9g/edit)** > > Dataset Overview > > For all data release details, models, Colabs etc, please check out our Github Repo > To filter the datasets based on attributes of your choice: > (1) select the columns you want to filter by (select row 15 and below)... So far all I’m seeing are CC-BY licensing, which matches the claims in the project README file. > **[GitHub - google-deepmind/open\_x\_embodiment](https://github.com/google-deepmind/open_x_embodiment)** > > Contribute to google-deepmind/open\_x\_embodiment development by creating an account on GitHub. My conclusion is OpenVLA is a project that matches all the OSAID requirements and optionals, including the under-debate requirements-or-not on having an OSI-licensed/OSD-compliant dataset. Please double-check my conclusions. Presuming for now my conclusion is accurate, how about some folks volunteer to run parallel experiments like this straw-draft: 1. Create a test suite against the existing openvla-7b model, verify the test suite reliably proves that a model is the same as openvla-7b. 2. Set about recreating openvla-7b in two ways using identical methods, except one uses the original dataset and the other only the Data information, along with all over available materials. 3. Run the test suite on all resulting pairs of experimental test models for fidelity to openvla7b. By straw-draft I mean, I am not claiming this draft stands as-is for being of a high quality of scientific or machine learning methods. As a straw-draft, is it good enough for now? Waddya think, Jason, are we in the ballpark? ---
**Author:** ![shujisado](https://yyz1.discourse-cdn.com/flex011/user_avatar/discuss.opensource.org/shujisado/32/230_2.png) [@shujisado](https://discuss.opensource.org/u/shujisado)\ **Post date:** [June 24, 2024, 12:04am UTC](https://discuss.opensource.org/t/an-open-call-to-test-openvla/424/2 "2024-06-24T00:04:10Z")
Such an experiment is intriguing. It would be extremely beneficial for us if someone were to conduct this experiment. The question of whether there is complete reproducibility or to what extent reproducibility exists has always been a challenge that has troubled me. ---
**Author:** ![stefano](https://yyz1.discourse-cdn.com/flex011/user_avatar/discuss.opensource.org/stefano/32/4_2.png) [@stefano](https://discuss.opensource.org/u/stefano)\ **Post date:** [June 24, 2024, 9:48am UTC](https://discuss.opensource.org/t/an-open-call-to-test-openvla/424/3 "2024-06-24T09:48:40Z")
> [@quaid](#): > > What if we bring to bear the idea that @jasonbrooks raised, which is conducting a controlled experiment on the question, “Do you need a full training dataset or is Data information enough to recreate a model that can be tested for fidelity to the original?” This is indeed a good question. We’ve started talking to a group of CMU students who will try to run this sort of experiment. More tests like this are welcome. Would it make sense to split this to a separate thread? ---
**Author:** ![quaid](https://yyz1.discourse-cdn.com/flex011/user_avatar/discuss.opensource.org/quaid/32/277_2.png) [@quaid](https://discuss.opensource.org/u/quaid)\ **Post date:** [June 25, 2024, 8:16pm UTC](https://discuss.opensource.org/t/an-open-call-to-test-openvla/424/4 "2024-06-25T20:16:22Z")
I’m cool with that, I didn’t choose ‘sameness’ as an idea per se, and we clearly need a more precise criteria (even if it’s from an imprecise measurement.) How about ‘‘functionally similar’’? Or ‘‘practically similar’’? Would it be useful to innumerate some of the value being sought with an Open Data dataset into the proof? Is this measurement inclusive of the ‘‘model vs model’’ matrix values, looking to keep QA benchmark values within a certain value of the original? What else matters in a recreated/retrained-from-source model?