Going further with iota2 launching parameters
When iota2 is installed, it is possible to display all launch options with the following command
Iota2.py -h
the next appears
usage: Iota2.py [-h] -config CONFIG_PATH [-starting_step START]
[-ending_step END] [-config_ressources CONFIG_RESSOURCES]
[-execution_graph_files GRAPH_FIGURES [GRAPH_FIGURES ...]]
[-restart_from RESTART_FROM] [-restart] [-only_summary]
[-nb_parallel_tasks NB_PARALLEL_TASKS]
[-scheduler_type [{debug,localCluster,cluster,PBS}]]
This function allow you to launch iota2 processing chain
optional arguments:
-h, --help show this help message and exit
-config CONFIG_PATH path to the configurationfile which rule le run
-starting_step START start chain from 'starting_step'
-ending_step END run chain until 'ending_step'-1 mean 'to the end'
-config_ressources CONFIG_RESSOURCES
path to IOTA2 ressources configuration file
-execution_graph_files GRAPH_FIGURES [GRAPH_FIGURES ...]
output figure for execution graphs
-restart_from RESTART_FROM
file containing task's states generated by iota2
-restart flag to restart iota2 from a previous run
-only_summary if set, only the summary will be printed. The chain
will not be launched
-nb_parallel_tasks NB_PARALLEL_TASKS
define the number of worker launching tasks. One
worker consumes one thread
-scheduler_type [{debug,cluster,PBS}]
list of available scheduler in iota2
This reminder may help users to set up the iota2 launch. This page also offers to explain each of these parameters
launching parameters
config
The configuration file that defines what iota2 will do
starting_step / ending_step
iota2 is made up of steps, this parameter indicates what will be the index (an integer) of the first / last step to launch. They overwrite the configuration file values.
configuration_ressources
It is also possible to limit the resources consumed for each steps task, this parameter will not be taken into account if the scheduler_type is ‘debug’. In the ‘debug’ case, nothing limits the ram consumed and only the environment variable ‘ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS’ allows you to limit the number of threads used per task (be careful according to nb_parallel_tasks, several tasks can run at the same time).
How to fill this configuration file ? Once activated, iota2 will for each step show the resources assigned to each step in the following format :
Iota2.py -config i2.cfg -config_ressources i2_resources.cfg -only_summary
Group init:
[x] Step 1: check inputs
resources block name : check_inputs -> MISSING
cpu : 1
ram : 5gb
walltime : 00:10:00
log identifier : CheckInputsClassifWorkflow
[x] Step 2: Sensors pre-processing
resources block name : preprocess_data
cpu : 2
ram : 10gb
walltime : 60:00:00
log identifier : sensorsPreprocess
In this case, it is specified that step 1 “check inputs” will be launched with the default resources because the resources in the i2_resources.cfg file were missing. Each step has an attribute allowing it to find in the resource file the resources assigned to it. This attribute is named: “resources block name”. In the example presented, it is indicated that the “resource block name” “check_inputs” is not found.
We can then add the “check_inputs” block to the resource file in the following way :
check_inputs:{
nb_cpu : 2
ram : "10gb"
walltime : "00:10:00"
}
execution_graph_files
Iota2 launches a sequence of steps, itself made up of tasks. This parameter allows you to save the dependencies between tasks in a figure in the form of a graphic.
for instance, a 4 tiles launched with the folowing command
Iota2.py -config my_config.cfg -execution_graph_files graph_of_tasks.png
can produce a task graph as
graph_of_tasks.png
Note
Some graphics can only be produced in the course of an iota2 execution. This is why the execution_graph_file parameter accepts a list of files. However, if the list contains only one item and several graphs are produced, the graphs will be produced in files named by the user with the suffix “_N.png” where N is an integer ranging from 0 to the number of graphs to be produced.
restart_from
Iota2 is able to restart from a given state. iota2 saves its state of progress in the file named ‘IOTA2_tasks_status.txt’ at the root of the output file. This parameter, if used, must be a path to such a file previously created by iota2.
restart
Allows you to restart iota2 from its previous state. Tasks already done will not be rerun.
only_summary
At the launch of iota2, a series of information is shown, then the chain starts. If this parameter is enabled, the launch information will be shown but the processes will not be started.
nb_parallel_tasks
This parameter allows you to set the number of tasks in parallel. Attention this parameter does not allow to define the number of threads used by a task. To set the number of threads per task, use the environment variable “ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS”.
scheduler_type
Please use the default value (localCluster) if you are running iota2 on shared memory architecture (no cluster or cloud computing). Else, you can choose between cluster and PBS where tasks are respectively send to the scheduler thanks to dask or by iota2. In iota2, calculations can be ‘distributed’. This means that several tasks can be computed at the same time on different machines or on the same machine (scheduler_type localCluster, cluster and PBS). In a debugging situation, it is sometimes easier if all tasks are not run in parallel and all logs are displayed directly in the terminal: this is the debug value for the parameter scheduler_type.
Warning
We found that some inter-process communication issues may occur when using ‘cluster’. Use ‘PBS’ instead if you experience communication timeouts.
dashboard
At launch, iota2 informs users of their ability to monitor the progress of the processing chain via a dashboad like the following
dashboard available at : http://some_ip_address:some_port/status
dashboard