Going further with iota2 launching parameters ############################################# When iota2 is installed, it is possible to display all launch options with the following command .. code-block:: python Iota2.py -h the next appears .. code-block:: console usage: Iota2.py [-h] -config CONFIG_PATH [-starting_step START] [-ending_step END] [-config_ressources CONFIG_RESSOURCES] [-execution_graph_files GRAPH_FIGURES [GRAPH_FIGURES ...]] [-restart_from RESTART_FROM] [-restart] [-only_summary] [-nb_parallel_tasks NB_PARALLEL_TASKS] [-scheduler_type [{debug,localCluster,cluster,PBS}]] This function allow you to launch iota2 processing chain optional arguments: -h, --help show this help message and exit -config CONFIG_PATH path to the configurationfile which rule le run -starting_step START start chain from 'starting_step' -ending_step END run chain until 'ending_step'-1 mean 'to the end' -config_ressources CONFIG_RESSOURCES path to IOTA2 ressources configuration file -execution_graph_files GRAPH_FIGURES [GRAPH_FIGURES ...] output figure for execution graphs -restart_from RESTART_FROM file containing task's states generated by iota2 -restart flag to restart iota2 from a previous run -only_summary if set, only the summary will be printed. The chain will not be launched -nb_parallel_tasks NB_PARALLEL_TASKS define the number of worker launching tasks. One worker consumes one thread -scheduler_type [{debug,cluster,PBS, Slurm}] list of available scheduler in iota2 This reminder may help users to set up the iota2 launch. This page also offers to explain each of these parameters launching parameters ******************** config ------ The configuration file that defines what iota2 will do starting_step / ending_step --------------------------- iota2 is made up of steps, this parameter indicates what will be the index (an integer) of the first / last step to launch. They overwrite the configuration file values. configuration_ressources ------------------------ It is also possible to limit the resources consumed for each steps task, this parameter will not be taken into account if the scheduler_type is 'debug'. In the 'debug' case, nothing limits the ram consumed and only the environment variable 'ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS' allows you to limit the number of threads used per task (be careful according to nb_parallel_tasks, several tasks can run at the same time). How to fill this configuration file ? Once activated, iota2 will for each step show the resources assigned to each step in the following format : .. code-block:: bash Iota2.py -config i2.cfg -config_ressources i2_resources.cfg -only_summary Group init: [x] Step 1: check inputs resources block name : check_inputs -> MISSING cpu : 1 ram : 5gb walltime : 00:10:00 log identifier : CheckInputsClassifWorkflow [x] Step 2: Sensors pre-processing resources block name : preprocess_data cpu : 2 ram : 10gb walltime : 60:00:00 log identifier : sensorsPreprocess In this case, it is specified that step 1 "check inputs" will be launched with the default resources because the resources in the i2_resources.cfg file were missing. Each step has an attribute allowing it to find in the resource file the resources assigned to it. This attribute is named: "resources block name". In the example presented, it is indicated that the "resource block name" "check_inputs" is not found. We can then add the "check_inputs" block to the resource file in the following way : .. code-block:: bash check_inputs:{ nb_cpu : 2 ram : "10gb" walltime : "00:10:00" } .. Note:: some other entry are available, **nb_gpu**, which allow user to ask some gpus. Using a PBS scheduler the key **queue** will spawn jobs one the dedicated queue. Considering a Slurm scheduler, **partition** key will request the targeting partition and the key **qos** the quality of service. execution_graph_files --------------------- Iota2 launches a sequence of steps, itself made up of tasks. This parameter allows you to save the dependencies between tasks in a figure in the form of a graphic. for instance, a 4 tiles launched with the folowing command .. code-block:: bash Iota2.py -config my_config.cfg -execution_graph_files graph_of_tasks.png can produce a task graph as .. figure:: ./Images/graph_example.png :scale: 30 % :align: center :alt: graph_of_tasks.png graph_of_tasks.png .. Note:: Some graphics can only be produced in the course of an iota2 execution. This is why the `execution_graph_file` parameter accepts a list of files. However, if the list contains only one item and several graphs are produced, the graphs will be produced in files named by the user with the suffix "_N.png" where N is an integer ranging from 0 to the number of graphs to be produced. restart_from ------------ Iota2 is able to restart from a given state. iota2 saves its state of progress in the file named 'IOTA2_tasks_status.txt' at the root of the output file. This parameter, if used, must be a path to such a file previously created by iota2. restart ------- Allows you to restart iota2 from its previous state. Tasks already done will not be rerun. only_summary ------------ At the launch of iota2, a series of information is shown, then the chain starts. If this parameter is enabled, the launch information will be shown but the processes will not be started. nb_parallel_tasks ----------------- This parameter allows you to set the number of tasks in parallel. Attention this parameter does not allow to define the number of threads used by a task. To set the number of threads per task, use the environment variable "ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS". scheduler_type -------------- Please use the default value (localCluster) if you are running iota2 on shared memory architecture (no cluster or cloud computing). Otherwise, you can choose between `cluster`, `PBS` and `Slurm` where tasks are respectively sent to the scheduler thanks to dask or by iota2. In iota2, computations can be 'distributed'. This means that several tasks can be computed at the same time on different machines or on the same machine (scheduler_type `localCluster`, `cluster`, `PBS`, `Slurm`). In a debugging situation, it is sometimes easier if tasks are not run in parallel and if logs are displayed directly in the terminal: this is the `debug` value for the parameter `scheduler_type`. .. WARNING:: We found that some inter-process communication issues may occur when using 'cluster'. Use 'PBS' instead if you experience communication timeouts. .. note:: It is possible to limit OTB memory usage in `debug` mode by setting the environment variable: OTB_MAX_RAM_HINT dashboard ********* At launch, iota2 informs users of their ability to monitor the progress of the processing chain via a dashboad like the following .. code-block:: bash dashboard available at : http://some_ip_address:some_port/status .. figure:: ./Images/dask_dashboard.png :align: center :alt: iota2 dashboard dashboard