Scope of this exercise is to find the answers to the questions below, keeping in mind a simplified OECD’s definition: “An AI system is a system that given an input produces an output. With this in mind, think of what is the preferred form to make modifications to it.”
- Use: What do you need to give an input and get an output from BLOOM?
- Study: What do you need to understand how BLOOM was built, how can it be fine-tuned, what biases, get a sense of why it gives an output to an input … ?
- Understand how it was built, its biases, limitations, potential pitfalls, etc.
- Modify: What do you need to give an input and get a different output from BLOOM?
- Techniques to adapt/modify the model for use including fine-tune and optimize for usage.
- Share: What do you need to let others give an input and get an output from BLOOM?
- This part should refer to how the model is shared, as received or after it was fine-tuned or modified in any way.
Unresolved questions
These issues were raised but deserve to be discussed more widely
- It was suggested that we disaggregate the category “other libraries or code artifacts that are part of the system, such as tokenizers and hyperparameter search code, if used,” which is currently serving as a catch-all.
- It was suggested that we add a number of floating point operations and hardware and software environment configurations required to conduct hyperparameter selection, pre-training, validation, and fine-tuning.
Participants to the WG
In their personal capacity, not representing the views of the companies they work for:
- George C. G. Barbosa (Fundação Oswaldo Cruz)
- Daniel Brumund (GIZ FAIR Forward - Artificial Intelligence for All)
- Danish Contractor (BLOOM Model Governance Working Group)
- Abdoulaye Diack (Google)
- Deshni Govender – (GIZ FAIR Forward - Artificial Intelligence for All)
- Jaan Li – (University of Tartu, Phare Health)
- Jean-Pierre Lorre – (LINAGORA OpenLLM-France)
- Ofentse Phuti – (Women in Machine Learning and Data Science, Gaborone)
- Caleb Fianku Quao – (Kwame Nkrumah University of Science and Technology, Kumasi)
Results of the analysis
| Code All code used to parse and process data, including: | Required to Use? | Required to Study? | Required to Modify? | Required to Share? |
|---|---|---|---|---|
| Data preprocessing code | 1 | 5 | 4 | 1 |
| Training, validation and testing code | 1 | 5 | 5 | 1 |
| Code used to perform inference for benchmark tests | 1 | 3 | 1 | 1 |
| Inference code | 5 | 5 | 4 | 4 |
| Evaluation code | 1 | 3 | 1 | 1 |
| Other libraries or code artifacts that are part of the system, such as tokenizers and hyperparameter search code, if used. | 5 | 5 | 5 | 5 |
| Data All data sets, including: | ||||
| Training data sets | 2 | 5 | 3 | 1 |
| Testing data sets | 2 | 5 | 2 | 1 |
| Validation data sets | 1 | 3 | 2 | 1 |
| Benchmarking data sets | 1 | 5 | 2 | 1 |
| Data card | 3 | |||
| Evaluation data | 2 | 1 | ||
| Evaluation results | 4 | 1 | ||
| All other data documentation | 1 | 3 | 2 | 1 |
| Model All model elements, including: | ||||
| Model architecture | 1 | 5 | 5 | 1 |
| Model parameters | 1 | 5 | 5 | 1 |
| Model metadata | 1 | |||
| Model card | 1 | 2 | 1 | |
| Sample model outputs | ||||
| Other Any other documentation or tools produced or used, including: | ||||
| Research paper | 3 | 5 | 1 | 1 |
| Usage documentation | 2 | 3 | 3 | 2 |
| Technical report | 2 | 1 | ||
| Supporting tools | 5 | 5 | 5 | 5 |
All the systems we analyzed are self-described as open, which includes BLOOM. In answer to your second question, we did initially intend to review Mistral, though it’s not part of the initial cohort. One of the decisions we need to make now is whether analyzing additional systems, such as Mistral, is required to enrich the working group recommendations based on that initial cohort, which I posted on Friday: