Scope of this exercise is to find the answers to the questions below, keeping in mind a simplified OECD’s definition: “An AI system is a system that given an input produces an output. With this in mind, think of what is the preferred form to make modifications to it.”
- What do you need to give an input and get an output from LLaMA2? (Use)
- What do you need to understand how LLaMA2 was built, how can it be fine-tuned, what biases, get a sense of why it gives an output to an input … ? (Study)
- How do you understand how it was built? Explain its performance, its biases? How do you spot if there are ethical considerations? Can you understand its base architecture? Can you interpret its output?
- What do you need to give an input and get a different output from LLaMA2? (Modify)
- Techniques to adapt/modify the model for use including fine-tune and optimize for usage.
- Tools that will help to have a fine-tuned model using user’s data. There are some public fine tuning models in HuggingFace, when fine tuning just some part of the model is updated, so the model is being modified.
- Optimization for usage? This part can refer to techniques to make the model suitable to be executed in light environments. In this case the model is modified because the weights are changed on the way they are represented. There is a challenge to get the same results with a “compressed” model.
- RAG or future techniques that will use the base model but the context data will be sent on the fly.
- What do you need to let others give an input and get an output from LLaMA2? (Share)
- This part should refer to how the model is shared, as received or after it was fine-tuned or modified in any way.
Unresolved questions
- If we consider the freedom to share as a special case for the freedom to let others use the requirements of some components may change. For example, what happens if the AI system is accessed (used) through a hosted API? We may need to clarify the use cases in order to give a more precise answer to what’s expected by sharing and also using a system.
- How deep does the freedom to STUDY go? Open science is one standard, but there are others. Is Open (AI) Science out of scope? Where do we set the minimum requirement to study an Open Source AI? Shall we say that the main purpose to study is to gain enough knowledge of the system to be able to modify it? Would it be more helpful to set the bar of study to being able to debug the system (why is the AI system classifying a train as a cat?)
- The list of components is a good starting point but needs to be reviewed. We’re waiting for the AI&Data Foundation GenAICommons to publish their paper so we can simply align to their list and definitions.
Participants to the WG
(In their personal capacity, not representing the views of the companies they work for)
- Stefano Maffulli – Open Source Initiative (convener)
- Mer Joyce – Do Big Good (facilitator)
- Bastien Guerry – DINUM, French public administration
- Ezequiel Lanza – Intel
- Roman Shaposhnik – Apache Software Foundation
- Davide Testuggine – Meta
- Jonathan Torres – Meta
- Stefano Zacchiroli – Polytechnic Institute of Paris
Results of the analysis
The numbers in the table below represent the total of people who believe that a component is required to exercise one of the freedoms. For example, 6 people said that Inference code is required to use LLama2.
| Code All code used to parse and process data, including: | Required to Use? | Required to Study? | Required to Modify? | Required to Share? |
|---|---|---|---|---|
| Data preprocessing code | 2 | 3 | ||
| Training code | 3 | 3 | ||
| Test code | ||||
| Code used to perform inference for benchmark tests | 1 | |||
| Validation code | 1 | |||
| Inference code | 6 | 1 | 1 | 2 |
| Evaluation code | ||||
| Other libraries or code artifacts that are part of the system, such as tokenizers and hyperparameter search code, if used. | 6 | 5 | 5 | 5 |
| Data All data sets, including: | ||||
| Training data sets | 2 | 1 | ||
| Testing data sets | 2 | 1 | ||
| Validation data sets | 1 | |||
| Benchmarking data sets | 1 | |||
| Data card | 2 | |||
| Evaluation data | ||||
| Evaluation metrics and results | 2 | |||
| All other data documentation | 1 | 1 | ||
| Model All model elements, including: | ||||
| Model architecture | 3 | 3 | ||
| Model parameters | 4 | 2 | 2 | 1 |
| Model card | 1 | 1 | 2 | 1 |
| Sample model outputs | ||||
| Other Any other documentation or tools produced or used, including: | ||||
| Research paper | 2 | 1 | ||
| Usage documentation | 1 | 1 | 1 | |
| Technical report | ||||
| Supporting tools |