111 create yaml template for problem submission

[kvdblom]

Draft yaml template for review

[Dvermetten]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

[CIGbalance]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

[CIGbalance]

I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?

[kvdblom]

Yes, it would be nice to have problems separately, I have also prototyped how it might work to have both the suite and component problems, see here:

OPL/problems.yaml

Line 1107 in 84b34b6

problems:

I am not sure about separate templates for suites or problems, but we can think about this and discuss what makes sense. I would imagine, e.g., the BBOB sphere still has a bunch of "varies", because it is scalable in the number of variables for example. For a suite, I would want an exhaustive list (e.g., noise: yes, no / optional, or something like that) of the options, rather than a vague "varies". I will describe this in a comment in the template.

[kvdblom]

TODO: create a new issue for this.

[bn12]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

• name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  • name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `

[kvdblom]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

Yes we should do that.

[kvdblom]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

We should split between

• A template
• An example

[Dvermetten]

To check what fields from the other info (based on the form) should be added to the template (taken from a different problem):
other info:
name: null
partial evaluations: Not Present
full name: Electric Motor Design Optimization
constraint properties: Hard Constraints, Soft Constraints, Box Constraints
number of constraints: '12'
type of dynamicism: ''
form of noise model: ''
type of noise space: ''
other noise properties: ''
description of multimodality: Constraints are multimodal
key challenges / characteristics: Time-consuming solution evaluation, highly-constrained
problem
scientific motivation: Challenging to find good solutions in a limited time
limitations: 'Unavailability, even if available, it wouldn''t be helpful to use
for benchmarking due taking a long time to evaluate a single solution '
implementation languages: Python
links to implementations: Implementation not freely available
approximate evaluation time: 8 minutes
links to usage examples: ''
general: This is not an available problem, but could be interesting to show to
researchers which difficulties appear in real-world problems

[kvdblom]

To check what fields from the other info (based on the form) should be added to the template (taken from a different problem): other info: name: null partial evaluations: Not Present full name: Electric Motor Design Optimization constraint properties: Hard Constraints, Soft Constraints, Box Constraints number of constraints: '12' type of dynamicism: '' form of noise model: '' type of noise space: '' other noise properties: '' description of multimodality: Constraints are multimodal key challenges / characteristics: Time-consuming solution evaluation, highly-constrained problem scientific motivation: Challenging to find good solutions in a limited time limitations: 'Unavailability, even if available, it wouldn''t be helpful to use for benchmarking due taking a long time to evaluate a single solution ' implementation languages: Python links to implementations: Implementation not freely available approximate evaluation time: 8 minutes links to usage examples: '' general: This is not an available problem, but could be interesting to show to researchers which difficulties appear in real-world problems

I wonder if we should separate this into numbers for soft and hard constraints:

• number of constraints: '12'

These are currently not yet included

• form of noise model: ''
• type of noise space: ''
• description of multimodality: Constraints are multimodal

[kvdblom]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

* name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  
  * name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `

This should be included now

[kvdblom]

TODO:

• Definitely still needs a consistency check
• When we are happy (to avoid double work), split into two files: (a) template.yaml and (b) example.yaml