Scope of this exercise is to find the answers to the questions below, keeping in mind a simplified OECD’s definition: “An AI system is a system that given an input produces an output. With this in mind, think of what is the preferred form to make modifications to it.”
Use: What do you need to give an input and get an output from OpenCV?
Study: What do you need to understand how OpenCV was built, how can it be fine-tuned, what biases, get a sense of why it gives an output to an input … ?
Understand how it was built, its biases, limitations, potential pitfalls, etc.
Modify: What do you need to give an input and get a different output from OpenCV?
Techniques to adapt/modify the model for use including fine-tune and optimize for usage.
Share: What do you need to let others give an input and get an output from OpenCV?
This part should refer to how the model is shared, as received or after it was fine-tuned or modified in any way.
Unresolved questions
These issues were raised and deserve to be discussed more widely.
There was a discussion of whether it is feasible to simply make all components open. The conclusion in the meeting was that each member should vote according to their own openness preferences and beliefs.
There was also an interesting discussion about how to make data open that contains no private or sensitive information, but is being monetized and is inaccessible to researchers due to its high cost. In the meeting, a “data equity fund” was proposed to partially ameliorate this problem.
Participants to the WG
In their personal capacity, not representing the views of the companies or organizations they work for:
Rahmat Akintola (Cubeseed Africa)
Ignatius Ezeani (Lancaster University)
Kevin Harerimana (Carnegie Mellon University Africa)
Satya Mallick (OpenCV)
David Manset (International Telecommunication Union)
Phil Nelson (OpenCV)
Tlamelo Makati (WiMLDS Gaborone, Technological University Dublin)
Minyechil Alehegn Tefera (Mizan Tepi University)
Akosua Twumasi (Ghana Health Service)
Results of the analysis
Code All code used to parse and process data, including:
Required to Use?
Required to Study?
Required to Modify?
Required to Share?
Data preprocessing code
3
5
3
1
Training, validation and testing code
5
5
2
1
Code used to perform inference for benchmark tests
Inference code
2
4
2
2
Evaluation code
2
4
2
1
Other libraries or code artifacts that are part of the system, such as tokenizers and hyperparameter search code, if used.
Data All data sets, including:
Training data sets
4
4
2
Testing data sets
4
5
1
Validation data sets
3
3
2
Benchmarking data sets
2
3
2
2
Data card
1
Evaluation data
1
Evaluation results
1
All other data documentation
1
3
1
Model All model elements, including:
Model architecture
1
4
1
1
Model parameters
1
3
1
1
Model metadata
1
3
1
Model card
1
3
1
1
Sample model outputs
1
3
1
2
Other Any other documentation or tools produced or used, including:
I should point out that ResNet is trained on ImageNet.
I have no objection to what you said, but under your framework (training data being required) ResNet is nowhere close to being open source and it never will be because ImageNet is proprietary and can only be accessed and used under certain conditions. You need to accept proprietary non-commercial “terms of access” to even download the dataset.
Yes. That’s why I propose the dataset being “open access”, although the phrasing is not quite accurate to cover the ImageNet case.
On other other side, IMHO it does not worth it to allow open source AI to hide their original training dataset just because we want to reshape the OSAID to fit into current practice. OSAID aims to provide a standard that is constructive enough for the future development of the ecosystem, instead of providing a compromised standard that is permissive enough to the issues in the current practice.
“Open access” does no good for open source because open source is about software freedom and is incompatible with any and all kinds of usage restrictions.
Ideally, I wanted to suggest using “open source” requirement for datasets (such as CC-BY-SA). But a wide range of existing models cannot satisfy this at all. That could also potentially irritate or drive industry players away from the OSAID, which is surely bad for the ecosystem’s future.
So my motivation of “open access” is to provide a middle place, to make OSAID less strict to adopt, while not fully loosing the ground of “open source”. Optimistically this might encourage more future works to “open source” their datasets.
Well, I also realize during the discussions that the middle ground “open access” is much more complicated to define and characterize. Pessimistically this might become a direct loophole to reduce people’s motivation to “open source” their datasets. I don’t know how things will develop.
If OSI does not fear irritating industry players who need the “open source” tag, I fully agree to ramp up the standards towards “open source dataset”, which is much easier to handle.
The middle place seems to have mos of the disadvantages of both and miss both of the advantages of either.
It’s still a requirement to high to meet for many existing models under open licenses and it doesn’t provide actual freedom in any sense which is in any way traditional in the scope of FLOSS.
Exactly. The contradictory points is whether OSAID is oriented to guide the future ecosystem development, or providing a taxonomy for the currently undefined messy practice.
If we provide suggestion for the future, “open source” is the way to go. If we provide taxonomy for current practice, “open access” is less likely to exclude all players.
I can only answer with another question: What would you expect to find that is so radically different from what we already learned from looking at OpenCV? Why would we want to add more working groups, what would you expect to find out that is novel and worth continuing the exercise?