iota2 feature maps

Assuming iota2 is fully operational, this chapter presents an usage of iota2: the production of features maps using satelitte images time series.

Introduction to data

iota2 handles several sensors images :

  • Landsat 5 and 8 (old and new THEIA format)

  • Sentinel 1, Sentinel 2 L2A(THEIA and Sen2cor), Sentinel L3A (Theia Format)

  • Various other images already processed, with the userFeat sensor

In this chapter, only the use of Sentinel 2 L2A will be illustrated. To use other sensor, it is necessary to adapt the inputs parameters.

To produce spectral indices, an user can use all spectral indices already computed by iota “link to spectral features in iota2 (NDVI…)” ad provide python code to produce custom features.

Get the data-set

Two data-set are available, containing minimal data required to run Iota2 builders:

  • A entire Sentinel 2 tile, with two dates (8.8 Go)

  • An extraction of Sentinel 2 data, with three dates over different eco-climatic region (Soon)

The archive contains:

  • /XXXX/IOTA2_TEST_S2
    archive content

    content of the tutorial archive after content extraction

    • ! external_code
      python code folder

      user custom python code used for external features / feature maps

      • external_code.py
        python code file

        contains user python code used to produce the spectral indices. Rules for creating user code are explained in external features page.

    • ! IOTA2_Outputs
      output folder

      folder used for iota2 output folders

      • (empty)

    • sensor_data
      input raster data

      the directory which contains Sentinel-2 data. These data must be stored by tiles as in the archive.

      • T31TCJ
        • ! SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TCJ_D_V1-7_FRE_STACK.tif

        • ! SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TCJ_D_V1-7_FRE_STACK.tif

      • T31TDJ
        • ! SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_ATB_R1.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_FRE_STACK.tif

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_MTD_ALL.xml

          • SENTINEL2A_20180511-105804-037_L2A_T31TDJ_D_V1-7_QKL_ALL.jpg

        • ! SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7
          • MASKS

            • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_ATB_R1.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_FRE_B*.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_FRE_STACK.tif

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_MTD_ALL.xml

          • SENTINEL2A_20180521-105702-711_L2A_T31TDJ_D_V1-7_QKL_ALL.jpg

    • vector_data
      input vector data

      directory containing input vector data

      • reference_data.cpg

      • reference_data.dbf

      • reference_data.prj

      • reference_data.shp
        shapefile

        the shapeFile containing geo-referenced and labelled polygons (no multi-polygons, no overlapping) used to train a classifier.

      • reference_data.shx

      • EcoRegion.dbf

      • EcoRegion.prj

      • EcoRegion.qpj

      • EcoRegion.shp
        shapefile

        shapeFile containing two geo-referenced polygons representing a spatial stratification (eco-climatic areas, for instance).

      • EcoRegion.shx

    • colorFile.txt
      color table

      colors used in classification map

      $ cat colorFile.txt
...
211 255 85 0
...

      Here the class 211 has the RGB color 255 85 0.

    • IOTA2_Example.cfg
      example config file

      the file used to set iota2’s parameters such as inputs/outputs paths, classifier parameters etc.

    • i2_tutorial_classification.cfg
      sample config file

      sample config file for classification builder

    • i2_tutorial_features_map.cfg
      sample config file

      sample config file for feature map builder

    • i2_tutorial_obia.cfg
      sample config file

      sample config file for object base image analysis

    • nomenclature23.txt
      nomenclature file

      label’s name. The purpose of the file is to get a pretty results report at the end of the chain by relabeling integers labels by a more verbose type.

      & cat nomenclature.txt
...
prairie:211
...

      Here the class 211 corresponds to the class prairie

    • vecteur_23.qml

Understand the configuration file

iota2 exploits hundreds of parameters, native or coming from other libraries such as scikit-learn or OTB.

These combinations of parameters will make it possible to choose the treatments to be carried out, the choice of the algorithms used or which output to write. A documentation of all these parameters is available parameters descriptions.

To simplify the use, iota2 read a configuration file which is a simple text file containing sections and fields.

The minimal configuration file contains all required fields to produce a land cover map.

chain :
{
  output_path : '/XXXX/output_path_features'
  remove_output_path : True
  list_tile : 'T31TCJ'
  s2_path : '/XXXX/IOTA2_TEST_S2/sensor_data'
  first_step : 'init'
  last_step : 'mosaic'
  spatial_resolution : 10
  proj : 'EPSG:2154'
}

sensors_data_interpolation :
{
  auto_date : False
}
Sentinel_2 :
{
  start_date : '20180101'
  end_date : '20181231'
}
external_features :
{
  functions : 'get_cumulative_productivity get_minimum_productivity get_seasonal_variation'
  number_of_chunks : 25
  concat_mode : False
  no_data_value : -10000
}
builders :
{
  builders_class_name : ['i2_features_map']
}
task_retry_limits :
{
  allowed_retry : 0
  maximum_ram : 180
  maximum_cpu : 40
}

For an end user, launching iota2 requires to fill correctly the configuration file.

In the above example, replace the XXXX by the path where the archive has been extracted.

Running the chain

iota2 launch

To launch the chain, simply use the command line

Iota2.py -config /XXXX/IOTA2_TESTS_DATA/i2_tutorial_features_map.cfg

First, the chain displays the list of all steps activated by the configuration file

Group init:
         [x] Step 1: Sensors pre-processing
         [x] Step 2: Generate a common masks for each sensors
Group writing:
         [x] Step 3: Write features by chunk
         [x] Step 4: Merge features chunks per tile
Group mosaic:
         [x] Step 5: Merge features tiles into a mosaic

Once the processing start, a large amount of information will be printed, most of them concerning the dask-scheduler.

Did it all go well?

iota2 displays a lot of information during execution. It becomes difficult to identify if everything went well or if an error occurred.

To this end, in the output_path a log directory is available : output_path/logs.

The logs directory is ordered as:

├── CommonMasks
│   └── common_mask_T31TCJ.out
├── html
│   └── *
├── merge_feat_maps_by_tiles.out
├── merge_feat_maps.out
├── run_informations.txt
├── sensorsPreprocess
│   └── preprocessing_T31TCJ.out
├── tasks_status_i2_features_map_1.svg
└── write_features_map
├── write_features_T31TCJ_chunk_0.out
├── write_features_T31TCJ_chunk_10.out
├── write_features_T31TCJ_chunk_11.out
├── write_features_T31TCJ_chunk_12.out
├── write_features_T31TCJ_chunk_13.out
├── write_features_T31TCJ_chunk_14.out
├── write_features_T31TCJ_chunk_15.out
├── write_features_T31TCJ_chunk_16.out
├── write_features_T31TCJ_chunk_17.out
├── write_features_T31TCJ_chunk_18.out
├── write_features_T31TCJ_chunk_19.out
├── write_features_T31TCJ_chunk_1.out
├── write_features_T31TCJ_chunk_2.out
├── write_features_T31TCJ_chunk_3.out
├── write_features_T31TCJ_chunk_4.out
├── write_features_T31TCJ_chunk_5.out
├── write_features_T31TCJ_chunk_6.out
├── write_features_T31TCJ_chunk_7.out
├── write_features_T31TCJ_chunk_8.out
└── write_features_T31TCJ_chunk_9.out

Each step has its own log directory. In these directories two kind of log can be found *_out.log and *_err.log. The error are compiled in “err” file and the standard output in “out” file. With the scheduler dask, iota2 go as far as possible while the data required for the next steps is available. To simplify the error identification, an interactive graph is produced in a html page. Nodes in graph can have three colors (red: error, blue: done, white: not yielded). By clicking on graph node, the corresponding log file is openned.

If despite all this information, the errors can not be identifiyed or solved, the iota2 can help all users. The simplest way to ask help is to create an issue on framagit by adding the archive available in log directory.

Output tree structure

In this section, all output folders are described.

├── by_tiles
│   └── T31TCJ.tif
├── customF
│   ├── T31TCJ_chunk_0.tif
│   ├── T31TCJ_chunk_10.tif
│   ├── T31TCJ_chunk_11.tif
│   ├── T31TCJ_chunk_12.tif
│   ├── T31TCJ_chunk_13.tif
│   ├── T31TCJ_chunk_14.tif
│   ├── T31TCJ_chunk_15.tif
│   ├── T31TCJ_chunk_16.tif
│   ├── T31TCJ_chunk_17.tif
│   ├── T31TCJ_chunk_18.tif
│   ├── T31TCJ_chunk_19.tif
│   ├── T31TCJ_chunk_1.tif
│   ├── T31TCJ_chunk_2.tif
│   ├── T31TCJ_chunk_3.tif
│   ├── T31TCJ_chunk_4.tif
│   ├── T31TCJ_chunk_5.tif
│   ├── T31TCJ_chunk_6.tif
│   ├── T31TCJ_chunk_7.tif
│   ├── T31TCJ_chunk_8.tif
│   └── T31TCJ_chunk_9.tif
├── features
│   └── T31TCJ
│       └── tmp
│           ├── MaskCommunSL.dbf
│           ├── MaskCommunSL.prj
│           ├── MaskCommunSL.shp
│           ├── MaskCommunSL.shx
│           ├── MaskCommunSL.tif
│           ├── Sentinel2_T31TCJ_input_dates.txt
│           ├── Sentinel2_T31TCJ_interpolation_dates.txt
│           └── Sentinel2_T31TCJ_reference.tif
├── final
│   └── features_map.tif
├── IOTA2_tasks_status.txt
├── logs
│   ├── *
├── logs.zip
└── T31TCJ

customF

Temporary data used for reassemble maps for each tiles. Can be removed by hand by user at the end of the processing.

by_tiles

The tile chunks are merged to produce a map for each tile

final

This folder contains the final products of features map builder.

features

For each tiles, contains useful information:

  • tmp/MaskCommunSL : the mask of all sensors for this tile

  • tmp/Sentinel2_T31TCJ_reference.tif : the image used for reprojecting data

  • tmp/Sentinel2_T31TCJ_input_dates.txt : the list of date detected in s2_path

  • tmp/Sentinel2_T31TCJ_interpolation_dates.txt : the list of output interpolation dates

Final products

Features map

features map

Features_map.tif Example

To go further