iota2 feature maps

Assuming iota2 is fully operational, this chapter presents an usage of iota2: the production of features maps using satelitte images time series.

Introduction to data

iota2 handles several sensors images :

Landsat 5 and 8 (old and new THEIA format)
Sentinel 1, Sentinel 2 L2A(THEIA and Sen2cor), Sentinel L3A (Theia Format)
Various other images already processed, with the userFeat sensor

In this chapter, only the use of Sentinel 2 L2A will be illustrated. To use other sensor, it is necessary to adapt the inputs parameters.

To produce spectral indices, an user can use all spectral indices already computed by iota “link to spectral features in iota2 (NDVI…)” ad provide python code to produce custom features.

Get the data-set

Two data-set are available, containing minimal data required to run Iota2 builders:

A entire Sentinel 2 tile, with two dates (8.8 Go)
An extraction of Sentinel 2 data, with three dates over different eco-climatic region (Soon)

The archive contains:

/XXXX/IOTA2_TEST_S2
archive content
content of the tutorial archive after content extraction
- ! external_code
  
  python code folder
  user custom python code used for external features / feature maps
  
  external_code.py
  
  python code file
  contains user python code used to produce the spectral indices. Rules for creating user code are explained in external features page.
- ! IOTA2_Outputs
  
  output folder
  folder used for iota2 output folders
  
  (empty)
- sensor_data
  
  input raster data
  the directory which contains Sentinel-2 data. These data must be stored by tiles as in the archive.
  
  T31TCJ
  
  ! SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7
  
  MASKS
  
  SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_*.tif
  
  SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_FRE_B*.tif
  
  SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_FRE_STACK.tif
  
  ! SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7
  
  MASKS
  
  SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_*.tif
  
  SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_FRE_B*.tif
  
  SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_FRE_STACK.tif
  
  T31TDJ
  
  ! SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7
  
  MASKS
  
  SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_*.tif
  
  SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_ATB_R1.tif
  
  SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_FRE_B*.tif
  
  SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_FRE_STACK.tif
  
  SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_MTD_ALL.xml
  
  SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_QKL_ALL.jpg
  
  ! SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7
  
  MASKS
  
  SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_*.tif
  
  SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_ATB_R1.tif
  
  SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_FRE_B*.tif
  
  SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_FRE_STACK.tif
  
  SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_MTD_ALL.xml
  
  SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_QKL_ALL.jpg
- vector_data
  
  input vector data
  directory containing input vector data
  
  reference_data.cpg
  
  reference_data.dbf
  
  reference_data.prj
  
  reference_data.shp
  
  shapefile
  the shapeFile containing geo-referenced and labelled polygons (no multi-polygons, no overlapping) used to train a classifier.
  
  reference_data.shx
  
  EcoRegion.dbf
  
  EcoRegion.prj
  
  EcoRegion.qpj
  
  EcoRegion.shp
  
  shapefile
  shapeFile containing two geo-referenced polygons representing a spatial stratification (eco-climatic areas, for instance).
  
  EcoRegion.shx
- colorFile.txt
  
  color table
  
  colors used in classification map
  
  $ cat colorFile.txt ... 211 255 85 0 ...
  
  Here the class 211 has the RGB color 255 85 0.
- IOTA2_Example.cfg
  
  example config file
  the file used to set iota2’s parameters such as inputs/outputs paths, classifier parameters etc.
- i2_tutorial_classification.cfg
  
  sample config file
  sample config file for classification builder
- i2_tutorial_features_map.cfg
  
  sample config file
  sample config file for feature map builder
- i2_tutorial_obia.cfg
  
  sample config file
  sample config file for object base image analysis
- nomenclature23.txt
  
  nomenclature file
  
  label’s name. The purpose of the file is to get a pretty results report at the end of the chain by relabeling integers labels by a more verbose type.
  
  & cat nomenclature.txt ... prairie:211 ...
  
  Here the class 211 corresponds to the class prairie
- vecteur_23.qml

Understand the configuration file

iota2 exploits hundreds of parameters, native or coming from other libraries such as scikit-learn or OTB.

These combinations of parameters will make it possible to choose the treatments to be carried out, the choice of the algorithms used or which output to write. A documentation of all these parameters is available parameters descriptions.

To simplify the use, iota2 read a configuration file which is a simple text file containing sections and fields.

The minimal configuration file contains all required fields to produce a land cover map.

chain :
{
  output_path : '/XXXX/output_path_features'
  remove_output_path : True
  list_tile : 'T31TCJ'
  s2_path : '/XXXX/IOTA2_TEST_S2/sensor_data'
  first_step : 'init'
  last_step : 'mosaic'
  spatial_resolution : 10
  proj : 'EPSG:2154'
}

sensors_data_interpolation :
{
  auto_date : False
}
Sentinel_2 :
{
  start_date : '20180101'
  end_date : '20181231'
}
external_features :
{
  functions : 'get_cumulative_productivity get_minimum_productivity get_seasonal_variation'
  number_of_chunks : 25
  concat_mode : False
  no_data_value : -10000
}
builders :
{
  builders_class_name : ['I2FeaturesMap']
}
task_retry_limits :
{
  allowed_retry : 0
  maximum_ram : 180
  maximum_cpu : 40
}

For an end user, launching iota2 requires to fill correctly the configuration file.

In the above example, replace the XXXX by the path where the archive has been extracted.

Running the chain

iota2 launch

To launch the chain, simply use the command line

Iota2.py -config /XXXX/IOTA2_TESTS_DATA/i2_tutorial_features_map.cfg

First, the chain displays the list of all steps activated by the configuration file

Group init:
         [x] Step 1: Sensors pre-processing
         [x] Step 2: Generate a common masks for each sensors
Group writing:
         [x] Step 3: Write features by chunk
         [x] Step 4: Merge features chunks per tile
Group mosaic:
         [x] Step 5: Merge features tiles into a mosaic

Once the processing start, a large amount of information will be printed, most of them concerning the dask-scheduler.

Did it all go well?

iota2 displays a lot of information during execution. It becomes difficult to identify if everything went well or if an error occurred.

To this end, in the output_path a log directory is available : output_path/logs.

The logs directory is ordered as:

├── CommonMasks
│   └── common_mask_T31TCJ.out
├── html
│   └── *
├── merge_feat_maps_by_tiles.out
├── merge_feat_maps.out
├── run_informations.txt
├── sensorsPreprocess
│   └── preprocessing_T31TCJ.out
├── tasks_status_i2_features_map_1.svg
└── write_features_map
├── write_features_T31TCJ_chunk_0.out
├── write_features_T31TCJ_chunk_10.out
├── write_features_T31TCJ_chunk_11.out
├── write_features_T31TCJ_chunk_12.out
├── write_features_T31TCJ_chunk_13.out
├── write_features_T31TCJ_chunk_14.out
├── write_features_T31TCJ_chunk_15.out
├── write_features_T31TCJ_chunk_16.out
├── write_features_T31TCJ_chunk_17.out
├── write_features_T31TCJ_chunk_18.out
├── write_features_T31TCJ_chunk_19.out
├── write_features_T31TCJ_chunk_1.out
├── write_features_T31TCJ_chunk_2.out
├── write_features_T31TCJ_chunk_3.out
├── write_features_T31TCJ_chunk_4.out
├── write_features_T31TCJ_chunk_5.out
├── write_features_T31TCJ_chunk_6.out
├── write_features_T31TCJ_chunk_7.out
├── write_features_T31TCJ_chunk_8.out
└── write_features_T31TCJ_chunk_9.out

Each step has its own log directory. In these directories two kind of log can be found *_out.log and *_err.log. The error are compiled in “err” file and the standard output in “out” file. With the scheduler dask, iota2 go as far as possible while the data required for the next steps is available. To simplify the error identification, an interactive graph is produced in a html page. Nodes in graph can have three colors (red: error, blue: done, white: not yielded). By clicking on graph node, the corresponding log file is openned.

If despite all this information, the errors can not be identifiyed or solved, the iota2 can help all users. The simplest way to ask help is to create an issue on framagit by adding the archive available in log directory.

Output tree structure

In this section, all output folders are described.

├── by_tiles
│   └── T31TCJ.tif
├── customF
│   ├── T31TCJ_chunk_0.tif
│   ├── T31TCJ_chunk_10.tif
│   ├── T31TCJ_chunk_11.tif
│   ├── T31TCJ_chunk_12.tif
│   ├── T31TCJ_chunk_13.tif
│   ├── T31TCJ_chunk_14.tif
│   ├── T31TCJ_chunk_15.tif
│   ├── T31TCJ_chunk_16.tif
│   ├── T31TCJ_chunk_17.tif
│   ├── T31TCJ_chunk_18.tif
│   ├── T31TCJ_chunk_19.tif
│   ├── T31TCJ_chunk_1.tif
│   ├── T31TCJ_chunk_2.tif
│   ├── T31TCJ_chunk_3.tif
│   ├── T31TCJ_chunk_4.tif
│   ├── T31TCJ_chunk_5.tif
│   ├── T31TCJ_chunk_6.tif
│   ├── T31TCJ_chunk_7.tif
│   ├── T31TCJ_chunk_8.tif
│   └── T31TCJ_chunk_9.tif
├── features
│   └── T31TCJ
│       └── tmp
│           ├── MaskCommunSL.dbf
│           ├── MaskCommunSL.prj
│           ├── MaskCommunSL.shp
│           ├── MaskCommunSL.shx
│           ├── MaskCommunSL.tif
│           ├── Sentinel2_T31TCJ_input_dates.txt
│           ├── Sentinel2_T31TCJ_interpolation_dates.txt
│           └── Sentinel2_T31TCJ_reference.tif
├── final
│   └── features_map.tif
├── IOTA2_tasks_status.txt
├── logs
│   ├── *
├── logs.zip
└── T31TCJ

customF

Temporary data used for reassemble maps for each tiles. Can be removed by hand by user at the end of the processing.

by_tiles

The tile chunks are merged to produce a map for each tile

final

This folder contains the final products of features map builder.

features

For each tiles, contains useful information:

tmp/MaskCommunSL : the mask of all sensors for this tile
tmp/Sentinel2_T31TCJ_reference.tif : the image used for reprojecting data
tmp/Sentinel2_T31TCJ_input_dates.txt : the list of date detected in s2_path
tmp/Sentinel2_T31TCJ_interpolation_dates.txt : the list of output interpolation dates

Final products

Features map

To go further

External features