nnUNet Experiment in NVFlare#

This tutorial will guide you through the process of running a nnUNet experiment in a Federated Learning context, by using the NVIDIA Flare API.

In this tutorial, we will use the nnUNet framework to train a model on a dataset that is distributed across multiple sites. The training will be done in a secure and privacy-preserving manner, by using the Federated Learning capabilities of the NVIDIA Flare API.

Model Training#

In detail, the following steps will be performed: 1. Dataset Preparation: The dataset in the different sites will be prepared for training, harmonizing the data and creating the necessary files for training according to the nnUNet framework. 2. nnUNet Experiment Planning and Preprocessing: One of the sites will be selected as the main site, where the nnUNet experiment will be planned and the data will be preprocessed. This step has to be done only on one site, as the nnUNet plans will be shared with the other sites. 3. nnUNet Preprocessing: The data will be preprocessed according to the nnUNet plan in all the other sites. 4. nnUNet Training: The model will be trained on the data of all the sites, using the nnUNet framework and aggregating the local gradients from the different sites.

Offline Training#

Offline training is also made available for the clients, to test the training process and evaluate the model at each individual site, establishing a baseline before moving on to the Federated Learning phase. In this fase, Data Preparation, Experiment Planning and Preprocessing, Training, Model Conversion, Validation and Packaging will be done on each site separately.

Model Deployment#

In addition, some specific jobs will be executed to: - Convert the model from the MONAI Bundle standard to the nnUNet format. - Validate the model using the nnUNet framework. - Package the model into a MONAI Bundle, ready for deployment. - Evaluate the model across different sites.

PREREQUISITES#

This tutorial assumes that you have already installed the NVIDIA Flare API and have access to a Federated Learning cluster, with Lead role. If you haven’t done so, please refer to the Getting Started section of the Tutorial section.

Additionally, all the sites should have the necessary data for training ready in a known location ( not necessarily the same location across all sites).

Configure the Federation in POC Mode#

Before starting the tutorial, we need to configure the federation in NVFlare POC (Proof of Concept) mode. This mode allows the federation to be configured within a single site, which is useful for testing and development purposes.

[ ]:
%%bash

export NVFLARE_POC_WORKSPACE="/home/maia-user/Tutorials/NVFlare_POC"

/opt/conda/envs/MAIA/bin/nvflare poc prepare
[ ]:
%%bash

export NVFLARE_POC_WORKSPACE="/home/maia-user/Tutorials/NVFlare_POC"

/opt/conda/envs/MAIA/bin/nvflare poc start

or

[ ]:
from nvflare.tool.poc.poc_commands import _prepare_poc, _start_poc, _stop_poc, _clean_poc

_prepare_poc(
    ["site-1","site-2"],
    2,
    "/home/maia-user/Tutorials/NVFlare_POC",
)
_start_poc("/home/maia-user/Tutorials/NVFlare_POC",[0])

Split the Data Across Sites#

After starting the federation in POC mode, we need to split the data across the two different sites. In this tutorial, we will randomly split the Decathlon Spleen dataset into two parts, one for each site.

[ ]:
from PyMAIA.utils.file_utils import subfiles
from random import shuffle
import shutil
from pathlib import Path
image_dir = "/home/maia-user/Tutorials/MAIA/Task09_Spleen/imagesTr"
label_dir = "/home/maia-user/Tutorials/MAIA/Task09_Spleen/labelsTr"

# Site 1 will have the "Decathlon" dataset format
Path("/home/maia-user/Tutorials/NVFlare_POC/data/site-1/imagesTr").mkdir(parents=True, exist_ok=True)
Path("/home/maia-user/Tutorials/NVFlare_POC/data/site-1/labelsTr").mkdir(parents=True, exist_ok=True)

# Site 2 will have the "Subfolders" dataset format
Path("/home/maia-user/Tutorials/NVFlare_POC/data/site-2").mkdir(parents=True, exist_ok=True)

images = subfiles(image_dir, suffix=".nii.gz")
labels = subfiles(label_dir, suffix=".nii.gz")

print(f"Images: {len(images)}")
print(f"Labels: {len(labels)}")

shuffle(images)

for image in images[:len(images)//2]:
    shutil.copy(image, "/home/maia-user/Tutorials/NVFlare_POC/data/site-1/imagesTr")
    shutil.copy(image.replace("imagesTr", "labelsTr"), "/home/maia-user/Tutorials/NVFlare_POC/data/site-1/labelsTr")

for image in images[len(images)//2:]:
    id = image.split("/")[-1].split(".")[0]
    Path(f"/home/maia-user/Tutorials/NVFlare_POC/data/site-2/{id}").mkdir(parents=True, exist_ok=True)
    shutil.copy(image, f"/home/maia-user/Tutorials/NVFlare_POC/data/site-2/{id}/{id}_CT.nii.gz")
    shutil.copy(image.replace("imagesTr", "labelsTr"), f"/home/maia-user/Tutorials/NVFlare_POC/data/site-2/{id}/{id}_label.nii.gz")

FL Cluster Authentication#

[3]:
from nvflare.fuel.flare_api.flare_api import new_secure_session
sess = new_secure_session(
    "admin@nvidia.com",
    "NVFlare_POC/example_project/prod_00/admin@nvidia.com"
)
[4]:
print(sess.get_system_info())
SystemInfo
server_info:
status: stopped, start_time: Tue Dec 17 07:16:29 2024
client_info:
site-2(last_connect_time: Tue Dec 17 07:16:36 2024)
site-1(last_connect_time: Tue Dec 17 07:16:37 2024)
job_info:

List Jobs#

[5]:
jobs = sess.list_jobs()
[6]:
jobs[-1]
[6]:
{'job_id': '6b26b950-4f40-40e0-81ae-27430d5f24a7',
 'job_name': 'fl_bundle',
 'status': 'FINISHED:ABANDONED',
 'submit_time': '2024-12-16T16:36:25.865534+00:00',
 'duration': 'N/A'}
[8]:
job_id = jobs[-1]['job_id']

Terminate Jobs#

[10]:
sess.abort_job(job_id)

TODO#

pymaia_nvflare from official PyMAIA patient_id_in_file_identifier: True #Add Docs

Job Preparation#

The first step in the Federated Learning process is to prepare the job configuration files. The job configuration files contain the necessary information to run the job on the Federated Learning cluster, such as the job type, the resources required, and the parameters for the job execution.

The job configurations files are automatically generated by the script generate_job_configs, which is installed together with this package. The script takes as input client-specific configuration, together with the experiment-specific configuration, and generates the job configuration files for each client.

Client Configuration#

The client-specific configuration file should be in the following format:

data_dir: "<DATASET_FOLDER>"
modality_dict:
  ct: "<CT_SUFFIX>"
  label: "<SEG_MASK_SUFFIX>"
dataset_format: "<DATASET_FORMAT>"
patient_id_in_file_identifier: True
nnunet_root_folder: "<NNUNET_ROOT_FOLDER>"
Client_Name: "<CLIENT_NAME>"
pymaia_config_file: "<CONFIG_FILE_PATH>"
nnunet_model_folder: "<NNUNET_MODEL_FOLDER>"
bundle_root: "<BUNDLE_ROOT>"

dataset_format should refer to one of these three different formats, according to the data_dir structure: 1. subfolders: The dataset is organized in subfolders, where each subfolder corresponds to a subject and contains the images and labels for that subject.

[Dataset_folder]
      [Subject_0]
          - Subject_0_image0.nii.gz    # Subject_0 modality 0
          - Subject_0_image1.nii.gz    # Subject_0 modality 1
          - Subject_0_mask.nii.gz      # Subject_0 semantic segmentation mask
      [Subject_1]
          - Subject_1_image0.nii.gz    # Subject_1 modality 0
          - Subject_1_image1.nii.gz    # Subject_1 modality 1
          - Subject_1_mask.nii.gz      # Subject_1 semantic segmentation mask
      ...
  1. decathlon: The dataset is organized in the format of the Medical Decathlon challenge, where the images and labels are stored in separate folders.

[Dataset_folder]
      [imagesTr]
          - Subject_0_image0.nii.gz    # Subject_0 modality 0
          - Subject_0_image1.nii.gz    # Subject_0 modality 1
          - Subject_1_image0.nii.gz    # Subject_1 modality 0
          - Subject_1_image1.nii.gz    # Subject_1 modality 1
      [labelsTr]
          - Subject_1_mask.nii.gz      # Subject_1 semantic segmentation mask
          - Subject_0_mask.nii.gz      # Subject_0 semantic segmentation mask
      ...
  1. nnunet: The dataset has been already prepared according to the nnUNet framework, with the images and labels stored in separate folders.

[nnUNet_raw]
    [DatasetXYZ_TaskName]  # THIS IS THE DATASET FOLDER
        dataset.json
        [imagesTr]
            - Subject_0_image0.nii.gz    # Subject_0 modality 0
            - Subject_0_image1.nii.gz    # Subject_0 modality 1
            - Subject_1_image0.nii.gz    # Subject_1 modality 0
            - Subject_1_image1.nii.gz    # Subject_1 modality 1
        [labelsTr]
            - Subject_1_mask.nii.gz      # Subject_1 semantic segmentation mask
            - Subject_0_mask.nii.gz      # Subject_0 semantic segmentation mask
      ...

nnunet_root_folder should refer to the root folder used by the nnUnet framework, where the nnUNet experiments are stored. For the subfolders and decathlon dataset formats, this folder is created during the dataset preparation step. For the nnunet dataset format, this folder should contain the nnUNet experiments, with the following structure:

[nnunet_root_folder]
    [nnUNet_raw]
        [DatasetXYZ_TaskName]
            dataset.json
            [imagesTr]
                - Subject_0_image0.nii.gz    # Subject_0 modality 0
                - Subject_0_image1.nii.gz    # Subject_0 modality 1
                - Subject_1_image0.nii.gz    # Subject_1 modality 0
                - Subject_1_image1.nii.gz    # Subject_1 modality 1
            [labelsTr]
                - Subject_1_mask.nii.gz      # Subject_1 semantic segmentation mask
                - Subject_0_mask.nii.gz      # Subject_0 semantic segmentation mask
          ...

modality_dict is a dictionary that maps the modality names to the file suffixes. The suffixes are used to identify the files that correspond to the different modalities in the dataset. For example, if the CT images have the suffix _CT.nii.gz, the entry in the modality_dict should be ct: "_CT.nii.gz".

pymaia_config_file is the path to the configuration file generated by PyMAIA. The file can be found in the results_folder ( nnunet_root_folder/TaskName/TaskName_results) of the PyMAIA experiment, with the name DatasetXYZ_TaskName.json.

nnunet_model_folder is the folder where the nnUNet folds and models are stored. This folder is created during the nnUNet experiment run: results_folder/DatasetXYZ_TaskName/TrainerClass__PlansName__3d_fullres

bundle_root is the root folder where the MONAI Bundle will be stored in the client. It can be retrieved at the end of the prepare_bundle job.

Experiment Configuration#

The experiment-specific configuration file should be in the following format:

task_ID: <NNUNET_TASK_ID>
Experiment Name: <NNUNET_TASK_NAME>
label_dict:
  background: 0
  Lesion: 1
MLFlow_Token: <MLFLOW_TOKEN>
Tracking_URI: <MLFLOW_URL>
Bundle_File: <BUNDLE_FILE.zip>
num_rounds: 100
start_round: 0
local_epochs: 10
server_bundle_root: "<SERVER_BUNDLE_ROOT>"
Minio_Endpoint: "MINIO_URL"
Minio_Access_Key: "ACCESS_KEY"
Minio_Secret_Key: "SECRET_KEY"
Minio_Bucket: "BUCKET_NAME"

task_ID and Experiment Name are used as a reference to the nnUNet task ID and name, respectively. These values are used to identify the nnUNet experiment in the nnUNet framework. Experiment Name is also used to identify the experiment in the MLFlow server, and to generate the zipped nnUNet model file (as <Experiment Name>.zip).

label_dict is a dictionary that maps the label names to the label values. The label values are used to identify the different classes in the semantic segmentation masks. For example, if the background class is labeled as 0 and the lesion class is labeled as 1, the entry in the label_dict should be background: 0, Lesion: 1.

MLFlow_Token and Tracking_URI are used to connect to the MLFlow server, where the experiments are logged, and the trained models are uploaded.

Bundle_File is the name of the zipped MONAI Bundle file that contains the python code and the necessary configuration files to run the model training. This file will be uploaded on each site during the training phase, during the execution of the training job.

server_bundle_root is the root folder where the MONAI Bundle will be stored on the server.

The MinIO parameters are needed to connect to the MinIO server and upload the trained FL models. You can leave these parameters empty if you are not using MinIO.

Prepare the Job Configuration Files#

To prepare the job configuration files, run the following command:

generate_job_configs -c <CLIENT_CONFIG_FILE_1> <CLIENT_CONFIG_FILE_2> ... -e <EXPERIMENT_CONFIG_FILE> -sd <SCRIPT_DIR> -j <JOB_DIR>

where: - <CLIENT_CONFIG_FILE_1>, <CLIENT_CONFIG_FILE_2>, … are the client-specific configuration files for each client. - <EXPERIMENT_CONFIG_FILE> is the experiment-specific configuration file. - <SCRIPT_DIR> is the directory where the job scripts and python files are stored nnUnet_NVFlare. - <JOB_DIR> is the directory where the job configuration files will be stored.

[1]:
import sys
from pathlib import Path

ROOT_FOLDER = "/home/maia-user/shared"
sys.path.append(ROOT_FOLDER)
[16]:
%%writefile /home/maia-user/shared/Experiments/Spleen.yaml

task_ID: "109"
Experiment Name: Task09_Spleen
label_dict:
  background: 0
  Spleen: 1
MLFlow_Token: ""
Tracking_URI: "https://monai-demo.maia.cloud.cbh.kth.se/mlflow"
Bundle_File: "Spleen_Bundle.zip"
num_rounds: 10
start_round: 0
local_epochs: 10
server_bundle_root: "/home/maia-user/shared/src/Spleen_Bundle"
Minio_Endpoint: ""
Minio_Access_Key: ""
Minio_Secret_Key: ""
Minio_Bucket: ""
Overwriting /home/maia-user/shared/Experiments/Spleen.yaml
[65]:
%%writefile /home/maia-user/shared/Clients/site-1.yaml

data_dir: "/home/maia-user/Tutorials/NVFlare_POC/data/site-1"
modality_dict:
  ct: ".nii.gz"
  label: ".nii.gz"
dataset_format: "decathlon"
patient_id_in_file_identifier: True
nnunet_root_folder: "/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-1"
Client_Name: "site-1"
pymaia_config_file: "/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-1/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen.json"
nnunet_model_folder: "/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-1/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen/nnUNetTrainer__nnUNetPlans_3d_fullres"
bundle_root: "/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-1/Spleen_Bundle"
Overwriting /home/maia-user/shared/Clients/site-1.yaml
[66]:
%%writefile /home/maia-user/shared/Clients/site-2.yaml

data_dir: "/home/maia-user/Tutorials/NVFlare_POC/data/site-2"
modality_dict:
  ct: "_CT.nii.gz"
  label: "_label.nii.gz"
dataset_format: "subfolders"
patient_id_in_file_identifier: True
nnunet_root_folder: "/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-2"
Client_Name: "site-2"
pymaia_config_file: "/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-2/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen.json"
nnunet_model_folder: "/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-2/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen/nnUNetTrainer__nnUNetPlans_3d_fullres"
bundle_root: "/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-2/Spleen_Bundle"
Overwriting /home/maia-user/shared/Clients/site-2.yaml
[7]:
%%writefile /home/maia-user/shared/src/Spleen/requirements.txt
pymaia-learn==1.2.1
nnunetv2==2.5.1
fire
pytorch-ignite==0.4.11
monai[nibabel, skimage, scipy, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, pandas, mlflow, matplotlib, pydicom]
pydicom==2.4.4
minio
numpy==1.24.0
pandas
mlflow
monai-nvflare
Overwriting /home/maia-user/shared/src/Spleen/requirements.txt
[67]:
experiment = "Spleen"
clients = [
    "site-1",
    "site-2"
]
[68]:
%%bash

rm -r /home/maia-user/shared/Jobs/Spleen/
[69]:
from generate_jobs_config import generate_config

generate_config(
    [str(Path(ROOT_FOLDER).joinpath("Clients",client+".yaml")) for client in clients],
    str(Path(ROOT_FOLDER).joinpath("Experiments",experiment+".yaml")),
    str(Path(ROOT_FOLDER).joinpath("src",experiment)),
    str(Path(ROOT_FOLDER).joinpath("Jobs",experiment)),
    nvflare_exec="/home/maia-user/.conda/envs/NVFlare/bin/nvflare",
    pymaia_nvflare_exec="pymaia_nvflare"

)

The following are the variables you can change in the template

---------------------------------------------------------------------------------------------------------------------------------------

  job folder: /home/maia-user/shared/Jobs/Spleen/jobs/prepare

---------------------------------------------------------------------------------------------------------------------------------------
  file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
  meta.conf                      mandatory_clients              ['site-1', 'site-2']
  meta.conf                      min_clients                    1
  meta.conf                      prepare-client-site-1          ['site-1']
  meta.conf                      prepare-client-site-2          ['site-2']
  meta.conf                      prepare-server                 ['server']

  app: prepare-client-site-1

  config_fed_client.conf         Client_Name                    site-1
  config_fed_client.conf         Experiment Name                Task09_Spleen
  config_fed_client.conf         FileExtension                  .nii.gz
  config_fed_client.conf         MLFlow_Token
  config_fed_client.conf         Seed                           12345
  config_fed_client.conf         Spleen                         1
  config_fed_client.conf         Tracking_URI                   https://monai-demo.maia.cloud.cbh.k
  config_fed_client.conf         background                     0
  config_fed_client.conf         components                     []
  config_fed_client.conf         ct                             .nii.gz
  config_fed_client.conf         data_dir                       /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         dataset_format                 decathlon
  config_fed_client.conf         label                          .nii.gz
  config_fed_client.conf         n_folds                        5
  config_fed_client.conf         nnunet_root_folder             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         patient_id_in_file_identifier  True
  config_fed_client.conf         task_ID                        109

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: prepare-client-site-2

  config_fed_client.conf         Client_Name                    site-2
  config_fed_client.conf         Experiment Name                Task09_Spleen
  config_fed_client.conf         FileExtension                  .nii.gz
  config_fed_client.conf         MLFlow_Token
  config_fed_client.conf         Seed                           12345
  config_fed_client.conf         Spleen                         1
  config_fed_client.conf         Tracking_URI                   https://monai-demo.maia.cloud.cbh.k
  config_fed_client.conf         background                     0
  config_fed_client.conf         components                     []
  config_fed_client.conf         ct                             _CT.nii.gz
  config_fed_client.conf         data_dir                       /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         dataset_format                 subfolders
  config_fed_client.conf         label                          _label.nii.gz
  config_fed_client.conf         n_folds                        5
  config_fed_client.conf         nnunet_root_folder             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         patient_id_in_file_identifier  True
  config_fed_client.conf         task_ID                        109

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: prepare-server

  config_fed_server.conf         heart_beat_timeout             600
  config_fed_server.conf         min_responses_required         0
  config_fed_server.conf         timeout                        6000
  config_fed_server.conf         wait_time_after_min_received   10

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

---------------------------------------------------------------------------------------------------------------------------------------

The following are the variables you can change in the template

---------------------------------------------------------------------------------------------------------------------------------------

  job folder: /home/maia-user/shared/Jobs/Spleen/jobs/plan_and_preprocess

---------------------------------------------------------------------------------------------------------------------------------------
  file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
  meta.conf                      mandatory_clients              ['site-1', 'site-2']
  meta.conf                      min_clients                    1
  meta.conf                      plan_and_preprocess-client-sit ['site-1']
  meta.conf                      plan_and_preprocess-client-sit ['site-2']
  meta.conf                      plan_and_preprocess-server     ['server']

  app: plan_and_preprocess-client-site-2

  config_fed_client.conf         Client_Name                    site-2
  config_fed_client.conf         Experiment Name                Task09_Spleen
  config_fed_client.conf         MLFlow_Token
  config_fed_client.conf         Tracking_URI                   https://monai-demo.maia.cloud.cbh.k
  config_fed_client.conf         components                     []
  config_fed_client.conf         nnunet_root_folder             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         task_ID                        109

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: plan_and_preprocess-client-site-1

  config_fed_client.conf         Client_Name                    site-1
  config_fed_client.conf         Experiment Name                Task09_Spleen
  config_fed_client.conf         MLFlow_Token
  config_fed_client.conf         Tracking_URI                   https://monai-demo.maia.cloud.cbh.k
  config_fed_client.conf         components                     []
  config_fed_client.conf         nnunet_root_folder             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         task_ID                        109

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: plan_and_preprocess-server

  config_fed_server.conf         heart_beat_timeout             600
  config_fed_server.conf         min_responses_required         0
  config_fed_server.conf         timeout                        6000
  config_fed_server.conf         wait_time_after_min_received   10

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

---------------------------------------------------------------------------------------------------------------------------------------

The following are the variables you can change in the template

---------------------------------------------------------------------------------------------------------------------------------------

  job folder: /home/maia-user/shared/Jobs/Spleen/jobs/preprocess

---------------------------------------------------------------------------------------------------------------------------------------
  file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
  meta.conf                      mandatory_clients              ['site-1', 'site-2']
  meta.conf                      min_clients                    1
  meta.conf                      preprocess-client-site-1       ['site-1']
  meta.conf                      preprocess-client-site-2       ['site-2']
  meta.conf                      preprocess-server              ['server']

  app: preprocess-client-site-2

  config_fed_client.conf         Client_Name                    site-2
  config_fed_client.conf         Experiment Name                Task09_Spleen
  config_fed_client.conf         MLFlow_Token
  config_fed_client.conf         Tracking_URI                   https://monai-demo.maia.cloud.cbh.k
  config_fed_client.conf         components                     []
  config_fed_client.conf         nnunet_root_folder             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         task_ID                        109

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: preprocess-client-site-1

  config_fed_client.conf         Client_Name                    site-1
  config_fed_client.conf         Experiment Name                Task09_Spleen
  config_fed_client.conf         MLFlow_Token
  config_fed_client.conf         Tracking_URI                   https://monai-demo.maia.cloud.cbh.k
  config_fed_client.conf         components                     []
  config_fed_client.conf         nnunet_root_folder             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         task_ID                        109

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: preprocess-server

  config_fed_server.conf         heart_beat_timeout             600
  config_fed_server.conf         min_responses_required         0
  config_fed_server.conf         timeout                        6000
  config_fed_server.conf         wait_time_after_min_received   10

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

---------------------------------------------------------------------------------------------------------------------------------------

The following are the variables you can change in the template

---------------------------------------------------------------------------------------------------------------------------------------

  job folder: /home/maia-user/shared/Jobs/Spleen/jobs/check_client_packages

---------------------------------------------------------------------------------------------------------------------------------------
  file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
  meta.conf                      check_client_packages-client-s ['site-1']
  meta.conf                      check_client_packages-client-s ['site-2']
  meta.conf                      check_client_packages-server   ['server']
  meta.conf                      mandatory_clients              ['site-1', 'site-2']
  meta.conf                      min_clients                    1

  app: check_client_packages-client-site-1

  config_fed_client.conf         components                     []

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: check_client_packages-server

  config_fed_server.conf         heart_beat_timeout             600
  config_fed_server.conf         min_responses_required         0
  config_fed_server.conf         timeout                        6000
  config_fed_server.conf         wait_time_after_min_received   10

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: check_client_packages-client-site-2

  config_fed_client.conf         components                     []

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

---------------------------------------------------------------------------------------------------------------------------------------

The following are the variables you can change in the template

---------------------------------------------------------------------------------------------------------------------------------------

  job folder: /home/maia-user/shared/Jobs/Spleen/jobs/prepare_bundle

---------------------------------------------------------------------------------------------------------------------------------------
  file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
  meta.conf                      mandatory_clients              ['site-1', 'site-2']
  meta.conf                      min_clients                    1
  meta.conf                      prepare_bundle-client-site-1   ['site-1']
  meta.conf                      prepare_bundle-client-site-2   ['site-2']
  meta.conf                      prepare_bundle-server          ['server']

  app: prepare_bundle-server

  config_fed_server.conf         heart_beat_timeout             600
  config_fed_server.conf         min_responses_required         0
  config_fed_server.conf         timeout                        600000
  config_fed_server.conf         wait_time_after_min_received   10

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: prepare_bundle-client-site-2

  config_fed_client.conf         bundle_file                    Spleen_Bundle.zip
  config_fed_client.conf         bundle_root                    /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         components                     []
  config_fed_client.conf         mlflow_experiment_name         Task09_Spleen-FL
  config_fed_client.conf         mlflow_run_name                site-2
  config_fed_client.conf         nnunet_model_folder            /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         pymaia_config_file             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         task_id                        109
  config_fed_client.conf         tracking_uri                   https://monai-demo.maia.cloud.cbh.k

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: prepare_bundle-client-site-1

  config_fed_client.conf         bundle_file                    Spleen_Bundle.zip
  config_fed_client.conf         bundle_root                    /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         components                     []
  config_fed_client.conf         mlflow_experiment_name         Task09_Spleen-FL
  config_fed_client.conf         mlflow_run_name                site-1
  config_fed_client.conf         nnunet_model_folder            /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         pymaia_config_file             /home/maia-user/Tutorials/NVFlare_P
  config_fed_client.conf         task_id                        109
  config_fed_client.conf         tracking_uri                   https://monai-demo.maia.cloud.cbh.k

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

---------------------------------------------------------------------------------------------------------------------------------------

The following are the variables you can change in the template

---------------------------------------------------------------------------------------------------------------------------------------

  job folder: /home/maia-user/shared/Jobs/Spleen/jobs/fl_bundle

---------------------------------------------------------------------------------------------------------------------------------------
  file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
  meta.conf                      fl_bundle-server               ['server']
  meta.conf                      fl_bundle-site-1               ['site-1']
  meta.conf                      fl_bundle-site-2               ['site-2']
  meta.conf                      mandatory_clients              ['site-1', 'site-2']
  meta.conf                      min_clients                    2

  app: fl_bundle-site-1

  config_fed_client.conf         best_model_filepath            models/model.pt                     MonaiAlgo
  config_fed_client.conf         bundle_root                    /home/maia-user/Tutorials/NVFlare_P MonaiAlgo
  config_fed_client.conf         config_evaluate_filename       default                             MonaiAlgo
  config_fed_client.conf         config_train_filename          configs/train.yaml                  MonaiAlgo
  config_fed_client.conf         disable_ckpt_loading           True                                MonaiAlgo
  config_fed_client.conf         eval_workflow_name             train                               MonaiAlgo
  config_fed_client.conf         final_model_filepath           models/model_final.pt               MonaiAlgo
  config_fed_client.conf         key_metric                     Val_Dice
  config_fed_client.conf         local_epochs                   10                                  MonaiAlgo
  config_fed_client.conf         save_dict_key                  network_weights                     MonaiAlgo
  config_fed_client.conf         send_weight_diff               True                                MonaiAlgo
  config_fed_client.conf         submit_model_task              submit_model
  config_fed_client.conf         train_task                     train
  config_fed_client.conf         validate_task                  validate

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: fl_bundle-site-2

  config_fed_client.conf         best_model_filepath            models/model.pt                     MonaiAlgo
  config_fed_client.conf         bundle_root                    /home/maia-user/Tutorials/NVFlare_P MonaiAlgo
  config_fed_client.conf         config_evaluate_filename       default                             MonaiAlgo
  config_fed_client.conf         config_train_filename          configs/train.yaml                  MonaiAlgo
  config_fed_client.conf         disable_ckpt_loading           True                                MonaiAlgo
  config_fed_client.conf         eval_workflow_name             train                               MonaiAlgo
  config_fed_client.conf         final_model_filepath           models/model_final.pt               MonaiAlgo
  config_fed_client.conf         key_metric                     Val_Dice
  config_fed_client.conf         local_epochs                   10                                  MonaiAlgo
  config_fed_client.conf         save_dict_key                  network_weights                     MonaiAlgo
  config_fed_client.conf         send_weight_diff               True                                MonaiAlgo
  config_fed_client.conf         submit_model_task              submit_model
  config_fed_client.conf         train_task                     train
  config_fed_client.conf         validate_task                  validate

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

  app: fl_bundle-server

  config_fed_server.conf         best_global_model_file_name    best_FL_global_model.pt             MonaiBundlePersistor
  config_fed_server.conf         bundle_root                    /home/maia-user/shared/src/Spleen_B MonaiBundlePersistor
  config_fed_server.conf         cleanup_models                 True                                CrossSiteModelEval
  config_fed_server.conf         config_train_filename          configs/train.yaml                  MonaiBundlePersistor
  config_fed_server.conf         expected_data_kind             WEIGHT_DIFF                         InTimeAccumulateWeightedAggregator
  config_fed_server.conf         global_model_file_name         FL_global_model.pt                  MonaiBundlePersistor
  config_fed_server.conf         min_clients                    {min_clients}
  config_fed_server.conf         network_def_key                network_def                         MonaiBundlePersistor
  config_fed_server.conf         num_rounds                     {num_rounds}
  config_fed_server.conf         start_round                    0
  config_fed_server.conf         submit_model_timeout           600                                 CrossSiteModelEval
  config_fed_server.conf         train_timeout                  0
  config_fed_server.conf         validation_timeout             6000                                CrossSiteModelEval
  config_fed_server.conf         wait_time_after_min_received   10

  nnUNetPlans-checkpoint.json    _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans-checkpoint.json    affine                         True
  nnUNetPlans-checkpoint.json    batch_dice                     True
  nnUNetPlans-checkpoint.json    batch_size                     2
  nnUNetPlans-checkpoint.json    conv_bias                      True
  nnUNetPlans-checkpoint.json    conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans-checkpoint.json    data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans-checkpoint.json    dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans-checkpoint.json    dropout_op                     None
  nnUNetPlans-checkpoint.json    dropout_op_kwargs              None
  nnUNetPlans-checkpoint.json    eps                            1e-05
  nnUNetPlans-checkpoint.json    experiment_planner_used        ExperimentPlanner
  nnUNetPlans-checkpoint.json    features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans-checkpoint.json    force_separate_z               None
  nnUNetPlans-checkpoint.json    image_reader_writer            SimpleITKIO
  nnUNetPlans-checkpoint.json    inherits_from                  3d_fullres
  nnUNetPlans-checkpoint.json    inplace                        True
  nnUNetPlans-checkpoint.json    is_seg                         False
  nnUNetPlans-checkpoint.json    kernel_sizes[0][0]             1
  nnUNetPlans-checkpoint.json    kernel_sizes[0][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[0][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[1][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[2][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[3][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[4][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[5][2]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[6][1]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][0]             3
  nnUNetPlans-checkpoint.json    kernel_sizes[7][1]             3
  nnUNetPlans-checkpoint.json    label_manager                  LabelManager
  nnUNetPlans-checkpoint.json    max                            1038.0
  nnUNetPlans-checkpoint.json    mean                           94.82794189453125
  nnUNetPlans-checkpoint.json    median                         99.0
  nnUNetPlans-checkpoint.json    median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans-checkpoint.json    min                            -620.0
  nnUNetPlans-checkpoint.json    n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans-checkpoint.json    n_stages                       6
  nnUNetPlans-checkpoint.json    network_class_name             dynamic_network_architectures.archi
  nnUNetPlans-checkpoint.json    next_stage                     3d_cascade_fullres
  nnUNetPlans-checkpoint.json    nonlin                         torch.nn.LeakyReLU
  nnUNetPlans-checkpoint.json    norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans-checkpoint.json    normalization_schemes          ['CTNormalization']
  nnUNetPlans-checkpoint.json    order                          1
  nnUNetPlans-checkpoint.json    order_z                        0
  nnUNetPlans-checkpoint.json    original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans-checkpoint.json    original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    patch_size                     [48, 224, 192]
  nnUNetPlans-checkpoint.json    percentile_00_5                -40.0
  nnUNetPlans-checkpoint.json    percentile_99_5                176.0
  nnUNetPlans-checkpoint.json    plans_name                     nnUNetPlans
  nnUNetPlans-checkpoint.json    preprocessor_name              DefaultPreprocessor
  nnUNetPlans-checkpoint.json    previous_stage                 3d_lowres
  nnUNetPlans-checkpoint.json    resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans-checkpoint.json    spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans-checkpoint.json    std                            40.52839660644531
  nnUNetPlans-checkpoint.json    strides[0][0]                  1
  nnUNetPlans-checkpoint.json    strides[0][1]                  1
  nnUNetPlans-checkpoint.json    strides[0][2]                  1
  nnUNetPlans-checkpoint.json    strides[1][0]                  1
  nnUNetPlans-checkpoint.json    strides[1][1]                  2
  nnUNetPlans-checkpoint.json    strides[1][2]                  2
  nnUNetPlans-checkpoint.json    strides[2][0]                  2
  nnUNetPlans-checkpoint.json    strides[2][1]                  2
  nnUNetPlans-checkpoint.json    strides[2][2]                  2
  nnUNetPlans-checkpoint.json    strides[3][0]                  2
  nnUNetPlans-checkpoint.json    strides[3][1]                  2
  nnUNetPlans-checkpoint.json    strides[3][2]                  2
  nnUNetPlans-checkpoint.json    strides[4][0]                  2
  nnUNetPlans-checkpoint.json    strides[4][1]                  2
  nnUNetPlans-checkpoint.json    strides[4][2]                  2
  nnUNetPlans-checkpoint.json    strides[5][0]                  1
  nnUNetPlans-checkpoint.json    strides[5][1]                  2
  nnUNetPlans-checkpoint.json    strides[5][2]                  2
  nnUNetPlans-checkpoint.json    strides[6][0]                  2
  nnUNetPlans-checkpoint.json    strides[6][1]                  2
  nnUNetPlans-checkpoint.json    strides[7][0]                  2
  nnUNetPlans-checkpoint.json    strides[7][1]                  2
  nnUNetPlans-checkpoint.json    transpose_backward             [0, 1, 2]
  nnUNetPlans-checkpoint.json    transpose_forward              [0, 1, 2]
  nnUNetPlans-checkpoint.json    use_mask_for_norm              [False]

  nnUNetPlans.json               _kw_requires_import            ['conv_op', 'norm_op', 'dropout_op'
  nnUNetPlans.json               affine                         True
  nnUNetPlans.json               batch_dice                     True
  nnUNetPlans.json               batch_size                     2
  nnUNetPlans.json               conv_bias                      True
  nnUNetPlans.json               conv_op                        torch.nn.modules.conv.Conv3d
  nnUNetPlans.json               data_identifier                nnUNetPlans_3d_fullres
  nnUNetPlans.json               dataset_name                   Dataset109_Task09_Spleen
  nnUNetPlans.json               dropout_op                     None
  nnUNetPlans.json               dropout_op_kwargs              None
  nnUNetPlans.json               eps                            1e-05
  nnUNetPlans.json               experiment_planner_used        ExperimentPlanner
  nnUNetPlans.json               features_per_stage             [32, 64, 128, 256, 320, 320]
  nnUNetPlans.json               force_separate_z               None
  nnUNetPlans.json               image_reader_writer            SimpleITKIO
  nnUNetPlans.json               inherits_from                  3d_fullres
  nnUNetPlans.json               inplace                        True
  nnUNetPlans.json               is_seg                         False
  nnUNetPlans.json               kernel_sizes[0][0]             1
  nnUNetPlans.json               kernel_sizes[0][1]             3
  nnUNetPlans.json               kernel_sizes[0][2]             3
  nnUNetPlans.json               kernel_sizes[1][0]             3
  nnUNetPlans.json               kernel_sizes[1][1]             3
  nnUNetPlans.json               kernel_sizes[1][2]             3
  nnUNetPlans.json               kernel_sizes[2][0]             3
  nnUNetPlans.json               kernel_sizes[2][1]             3
  nnUNetPlans.json               kernel_sizes[2][2]             3
  nnUNetPlans.json               kernel_sizes[3][0]             3
  nnUNetPlans.json               kernel_sizes[3][1]             3
  nnUNetPlans.json               kernel_sizes[3][2]             3
  nnUNetPlans.json               kernel_sizes[4][0]             3
  nnUNetPlans.json               kernel_sizes[4][1]             3
  nnUNetPlans.json               kernel_sizes[4][2]             3
  nnUNetPlans.json               kernel_sizes[5][0]             3
  nnUNetPlans.json               kernel_sizes[5][1]             3
  nnUNetPlans.json               kernel_sizes[5][2]             3
  nnUNetPlans.json               kernel_sizes[6][0]             3
  nnUNetPlans.json               kernel_sizes[6][1]             3
  nnUNetPlans.json               kernel_sizes[7][0]             3
  nnUNetPlans.json               kernel_sizes[7][1]             3
  nnUNetPlans.json               label_manager                  LabelManager
  nnUNetPlans.json               max                            1038.0
  nnUNetPlans.json               mean                           94.82794189453125
  nnUNetPlans.json               median                         99.0
  nnUNetPlans.json               median_image_size_in_voxels    [112.0, 512.0, 512.0]
  nnUNetPlans.json               min                            -620.0
  nnUNetPlans.json               n_conv_per_stage               [2, 2, 2, 2, 2, 2]
  nnUNetPlans.json               n_conv_per_stage_decoder       [2, 2, 2, 2, 2]
  nnUNetPlans.json               n_stages                       6
  nnUNetPlans.json               network_class_name             dynamic_network_architectures.archi
  nnUNetPlans.json               next_stage                     3d_cascade_fullres
  nnUNetPlans.json               nonlin                         torch.nn.LeakyReLU
  nnUNetPlans.json               norm_op                        torch.nn.modules.instancenorm.Insta
  nnUNetPlans.json               normalization_schemes          ['CTNormalization']
  nnUNetPlans.json               order                          1
  nnUNetPlans.json               order_z                        0
  nnUNetPlans.json               original_median_shape_after_tr [83, 512, 512]
  nnUNetPlans.json               original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               patch_size                     [48, 224, 192]
  nnUNetPlans.json               percentile_00_5                -40.0
  nnUNetPlans.json               percentile_99_5                176.0
  nnUNetPlans.json               plans_name                     nnUNetPlans
  nnUNetPlans.json               preprocessor_name              DefaultPreprocessor
  nnUNetPlans.json               previous_stage                 3d_lowres
  nnUNetPlans.json               resampling_fn_data             resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_probabilities    resample_data_or_seg_to_shape
  nnUNetPlans.json               resampling_fn_seg              resample_data_or_seg_to_shape
  nnUNetPlans.json               spacing                        [2.5, 0.7929689884185791, 0.7929689
  nnUNetPlans.json               std                            40.52839660644531
  nnUNetPlans.json               strides[0][0]                  1
  nnUNetPlans.json               strides[0][1]                  1
  nnUNetPlans.json               strides[0][2]                  1
  nnUNetPlans.json               strides[1][0]                  1
  nnUNetPlans.json               strides[1][1]                  2
  nnUNetPlans.json               strides[1][2]                  2
  nnUNetPlans.json               strides[2][0]                  2
  nnUNetPlans.json               strides[2][1]                  2
  nnUNetPlans.json               strides[2][2]                  2
  nnUNetPlans.json               strides[3][0]                  2
  nnUNetPlans.json               strides[3][1]                  2
  nnUNetPlans.json               strides[3][2]                  2
  nnUNetPlans.json               strides[4][0]                  2
  nnUNetPlans.json               strides[4][1]                  2
  nnUNetPlans.json               strides[4][2]                  2
  nnUNetPlans.json               strides[5][0]                  1
  nnUNetPlans.json               strides[5][1]                  2
  nnUNetPlans.json               strides[5][2]                  2
  nnUNetPlans.json               strides[6][0]                  2
  nnUNetPlans.json               strides[6][1]                  2
  nnUNetPlans.json               strides[7][0]                  2
  nnUNetPlans.json               strides[7][1]                  2
  nnUNetPlans.json               transpose_backward             [0, 1, 2]
  nnUNetPlans.json               transpose_forward              [0, 1, 2]
  nnUNetPlans.json               use_mask_for_norm              [False]

---------------------------------------------------------------------------------------------------------------------------------------
[70]:
JOB_DIR=str(Path(ROOT_FOLDER).joinpath("Jobs",experiment))
[60]:
client_id = "site-2"

Create and Modify Job Configuration#

Jobs configuration files are automatically created, but can be recreated or modified using the following NVFlare command:

nvflare job create -j <CUSTOM_JOB_DIR>/<TASK_NAME> -w <JOB_DIR>/<TASK_NAME  -sd nnUNet_NVFlare/ --force
nvflare job show_variables -j <CUSTOM_JOB_DIR>/<TASK_NAME>
nvflare job create -j <CUSTOM_JOB_DIR>/<TASK_NAME> -w <JOB_DIR>/<TASK_NAME  -sd nnUNet_NVFlare/ -f <app>/<file_name> var_name=new_value

0. Check Python Packages#

[25]:
from pathlib import Path

job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","check_client_packages")))
[26]:
sess.monitor_job(job_id)
[26]:
<MonitorReturnCode.JOB_FINISHED: 0>
[46]:
client_id = "site-2"
[31]:
#print(sess.api.do_command(f"cat server log.txt")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} nvidia-smi.log")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} init_logfile_out.log")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} init_logfile_err.log")['data'][0]['data'])
Mon Dec 16 13:23:30 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        Off | 00000000:01:00.0 Off |                  N/A |
| 30%   35C    P8              26W / 350W |    261MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[32]:
job_dir = sess.download_job_result(job_id)
[33]:
import json
from pathlib import Path

with open(Path(job_dir).joinpath("workspace","package_report","package_report.json"),"r") as f:
    package_report = json.load(f)
[34]:
package_report
[34]:
{'site-2': {'nvflare': 'nvflare 2.5.2 is installed.',
  'pymaia-learn': 'pymaia-learn 1.2.1 is installed.',
  'torch': 'torch 2.5.1 is installed.',
  'monai': 'monai 1.4.0 is installed.',
  'numpy': 'numpy 1.24.0 is installed.',
  'nnunetv2': 'nnunetv2 2.5.1 is installed.',
  'MONAI': '1.4.0',
  'Numpy': '1.24.0',
  'Pytorch': '',
  'Pytorch Ignite': '0.4.11',
  'ITK': '5.4.0',
  'Nibabel': '5.3.2',
  'scikit-image': '0.25.0',
  'scipy': '1.14.1',
  'Pillow': '11.0.0',
  'Tensorboard': '2.18.0',
  'gdown': '5.2.0',
  'TorchVision': '0.20.1',
  'tqdm': '4.67.1',
  'lmdb': 'NOT INSTALLED or UNKNOWN VERSION.',
  'psutil': '6.1.0',
  'pandas': '2.2.3',
  'einops': '0.8.0',
  'transformers': 'NOT INSTALLED or UNKNOWN VERSION.',
  'mlflow': '2.19.0',
  'pynrrd': 'NOT INSTALLED or UNKNOWN VERSION.',
  'clearml': 'NOT INSTALLED or UNKNOWN VERSION.',
  'Num GPUs': '1',
  'cuDNN enabled': 'True',
  'GPU 0 Name': 'NVIDIA GeForce RTX 3090',
  'GPU 0 Total memory  GB ': '23.7',
  'CPU_Cores': 20,
  'Total Memory': 62.38420104980469,
  'Memory Used': 24.1},
 'site-1': {'nvflare': 'nvflare 2.5.2 is installed.',
  'pymaia-learn': 'pymaia-learn 1.2.1 is installed.',
  'torch': 'torch 2.5.1 is installed.',
  'monai': 'monai 1.4.0 is installed.',
  'numpy': 'numpy 1.24.0 is installed.',
  'nnunetv2': 'nnunetv2 2.5.1 is installed.',
  'MONAI': '1.4.0',
  'Numpy': '1.24.0',
  'Pytorch': '',
  'Pytorch Ignite': '0.4.11',
  'ITK': '5.4.0',
  'Nibabel': '5.3.2',
  'scikit-image': '0.25.0',
  'scipy': '1.14.1',
  'Pillow': '11.0.0',
  'Tensorboard': '2.18.0',
  'gdown': '5.2.0',
  'TorchVision': '0.20.1',
  'tqdm': '4.67.1',
  'lmdb': 'NOT INSTALLED or UNKNOWN VERSION.',
  'psutil': '6.1.0',
  'pandas': '2.2.3',
  'einops': '0.8.0',
  'transformers': 'NOT INSTALLED or UNKNOWN VERSION.',
  'mlflow': '2.19.0',
  'pynrrd': 'NOT INSTALLED or UNKNOWN VERSION.',
  'clearml': 'NOT INSTALLED or UNKNOWN VERSION.',
  'Num GPUs': '1',
  'cuDNN enabled': 'True',
  'GPU 0 Name': 'NVIDIA GeForce RTX 3090',
  'GPU 0 Total memory  GB ': '23.7',
  'CPU_Cores': 20,
  'Total Memory': 62.38420104980469,
  'Memory Used': 24.1}}
[ ]:

1. Prepare Dataset#

In this step, the dataset in the different sites will be prepared for training, harmonizing the data and creating the necessary files for training according to the nnUNet framework. The following python packages will be installed during the execution of this step:

pymaia-learn==1.1
nnunetv2==2.5.1
fire
pytorch-ignite==0.4.11
monai[nibabel, skimage, scipy, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, pandas, mlflow, matplotlib, pydicom]

Submit the Job#

[35]:
from pathlib import Path

job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","prepare")))
[36]:
sess.monitor_job(job_id)
[36]:
<MonitorReturnCode.JOB_FINISHED: 0>

To monitor the job, or print the logs from either the server or the client side, you can use the following commands:

[ ]:
#print(sess.api.do_command(f"cat server log.txt")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} init_logfile_out.log")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} init_logfile_err.log")['data'][0]['data'])

#print(sess.api.do_command(f"cat {client_id} logfile_datadir.log")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} logfile_storage.log")['data'][0]['data'])

#print(sess.api.do_command(f"cat {client_id} logfile_prepare_out.log")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} logfile_prepare_err.log")['data'][0]['data'])

Download the Job Results#

When the job is completed, you can download the results, which, for the Prepare Dataset job, will contain the dataset.json file, containing the information about the dataset. In detail, for each client, the dataset.json file will list the dataset files and wheter all the files are valid or not.

[37]:
job_dir = sess.download_job_result(job_id)
[38]:
import json
from pathlib import Path

with open(Path(job_dir).joinpath("workspace","prepare_dataset","data_prepare.json"),"r") as f:
    dataset_dict = json.load(f)

To display the PyMAIA Experiment Configuration, run the following command:

[42]:
dataset_dict[client_id]['nnUNet_Dataset']['imagesTr']#['pymaia_config']
#dataset_dict[client_id]['pymaia_config']
[42]:
['spleen_49_0000.nii.gz',
 'spleen_8_0000.nii.gz',
 'spleen_16_0000.nii.gz',
 'spleen_31_0000.nii.gz',
 'spleen_52_0000.nii.gz',
 'spleen_56_0000.nii.gz',
 'spleen_40_0000.nii.gz',
 'spleen_61_0000.nii.gz',
 'spleen_62_0000.nii.gz',
 'spleen_22_0000.nii.gz',
 'spleen_9_0000.nii.gz',
 'spleen_45_0000.nii.gz',
 'spleen_59_0000.nii.gz',
 'spleen_53_0000.nii.gz',
 'spleen_17_0000.nii.gz',
 'spleen_41_0000.nii.gz',
 'spleen_13_0000.nii.gz',
 'spleen_60_0000.nii.gz',
 'spleen_3_0000.nii.gz',
 'spleen_46_0000.nii.gz',
 'spleen_28_0000.nii.gz',
 'spleen_21_0000.nii.gz',
 'spleen_14_0000.nii.gz',
 'spleen_10_0000.nii.gz',
 'spleen_47_0000.nii.gz',
 'spleen_26_0000.nii.gz',
 'spleen_19_0000.nii.gz',
 'spleen_32_0000.nii.gz',
 'spleen_33_0000.nii.gz',
 'spleen_29_0000.nii.gz',
 'spleen_6_0000.nii.gz']
[43]:
dataset_dict[client_id]['nnUNet_Dataset']
[43]:
{'imagesTr': ['spleen_49_0000.nii.gz',
  'spleen_8_0000.nii.gz',
  'spleen_16_0000.nii.gz',
  'spleen_31_0000.nii.gz',
  'spleen_52_0000.nii.gz',
  'spleen_56_0000.nii.gz',
  'spleen_40_0000.nii.gz',
  'spleen_61_0000.nii.gz',
  'spleen_62_0000.nii.gz',
  'spleen_22_0000.nii.gz',
  'spleen_9_0000.nii.gz',
  'spleen_45_0000.nii.gz',
  'spleen_59_0000.nii.gz',
  'spleen_53_0000.nii.gz',
  'spleen_17_0000.nii.gz',
  'spleen_41_0000.nii.gz',
  'spleen_13_0000.nii.gz',
  'spleen_60_0000.nii.gz',
  'spleen_3_0000.nii.gz',
  'spleen_46_0000.nii.gz',
  'spleen_28_0000.nii.gz',
  'spleen_21_0000.nii.gz',
  'spleen_14_0000.nii.gz',
  'spleen_10_0000.nii.gz',
  'spleen_47_0000.nii.gz',
  'spleen_26_0000.nii.gz',
  'spleen_19_0000.nii.gz',
  'spleen_32_0000.nii.gz',
  'spleen_33_0000.nii.gz',
  'spleen_29_0000.nii.gz',
  'spleen_6_0000.nii.gz'],
 'labelsTr': ['spleen_9.nii.gz',
  'spleen_31.nii.gz',
  'spleen_28.nii.gz',
  'spleen_59.nii.gz',
  'spleen_10.nii.gz',
  'spleen_47.nii.gz',
  'spleen_41.nii.gz',
  'spleen_45.nii.gz',
  'spleen_21.nii.gz',
  'spleen_26.nii.gz',
  'spleen_13.nii.gz',
  'spleen_33.nii.gz',
  'spleen_6.nii.gz',
  'spleen_8.nii.gz',
  'spleen_53.nii.gz',
  'spleen_52.nii.gz',
  'spleen_22.nii.gz',
  'spleen_16.nii.gz',
  'spleen_49.nii.gz',
  'spleen_17.nii.gz',
  'spleen_29.nii.gz',
  'spleen_56.nii.gz',
  'spleen_62.nii.gz',
  'spleen_14.nii.gz',
  'spleen_46.nii.gz',
  'spleen_19.nii.gz',
  'spleen_3.nii.gz',
  'spleen_61.nii.gz',
  'spleen_40.nii.gz',
  'spleen_32.nii.gz',
  'spleen_60.nii.gz']}

And to inspect the dataset, run the following command:

[47]:
verified = True

for case in dataset_dict[client_id]["train"]:
    for key in case:
        if key.endswith("_is_file") and not case[key]:
            file = case[key[:-len("_is_file")]]
            print(f"Error: {file} is not a valid file!")
            verified = False
if verified:
    print(f"Dataset succesfully verified for client {client_id}")
Error: /home/maia-user/Tutorials/NVFlare_POC/data/site-2/.ipynb_checkpoints/.ipynb_checkpoints_CT.nii.gz is not a valid file!
Error: /home/maia-user/Tutorials/NVFlare_POC/data/site-2/.ipynb_checkpoints/.ipynb_checkpoints_label.nii.gz is not a valid file!
[48]:
print(len(dataset_dict[client_id]["train"]))
25

2. Plan and Preprocess#

After the dataset has been prepared, the nnUNet experiment has to be planned and the data preprocessed. This step has to be done only on one site, as the nnUNet plans will be shared with the other sites.

The steps to plan and preprocess the nnUNet experiment are the following: 1. Run the plan_and_preprocess job on the chosen site. 2. Extract the nnUNet plans from the job results. 3. Share the nnUNet plans with the other sites. 4. Run the preprocess job on the other sites.

Submit the Job#

[55]:
from pathlib import Path

job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","plan_and_preprocess")))
[56]:
sess.monitor_job(job_id)
[56]:
<MonitorReturnCode.JOB_FINISHED: 0>
[ ]:
#print(sess.api.do_command(f"tail server log.txt")['data'][0]['data'])
#print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} logfile_plan_and_preprocess_out.log")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} logfile_plan_and_preprocess_err.log")['data'][0]['data'])
[57]:
job_dir = sess.download_job_result(job_id)

Inspect nnUNetPlans.json#

The nnUNet plans are stored in the nnUNetPlans.json file, which contains the configuration for the nnUNet experiment. The file can be found in the workspace/nnUNet_preprocessing folder of the job results.

[58]:
import json

with open(Path(job_dir).joinpath("workspace","nnUNet_preprocessing","nnUNetPlans.json"),"r") as f:
    nnunet_plans = json.load(f)
[59]:
print(json.dumps(nnunet_plans,indent=4))
{
    "site-1": {
        "dataset_name": "Dataset109_Task09_Spleen",
        "plans_name": "nnUNetPlans",
        "original_median_spacing_after_transp": [
            5.0,
            0.7929689884185791,
            0.7929689884185791
        ],
        "original_median_shape_after_transp": [
            83,
            512,
            512
        ],
        "image_reader_writer": "SimpleITKIO",
        "transpose_forward": [
            0,
            1,
            2
        ],
        "transpose_backward": [
            0,
            1,
            2
        ],
        "configurations": {
            "2d": {
                "data_identifier": "nnUNetPlans_2d",
                "preprocessor_name": "DefaultPreprocessor",
                "batch_size": 12,
                "patch_size": [
                    512,
                    512
                ],
                "median_image_size_in_voxels": [
                    512.0,
                    512.0
                ],
                "spacing": [
                    0.7929689884185791,
                    0.7929689884185791
                ],
                "normalization_schemes": [
                    "CTNormalization"
                ],
                "use_mask_for_norm": [
                    false
                ],
                "resampling_fn_data": "resample_data_or_seg_to_shape",
                "resampling_fn_seg": "resample_data_or_seg_to_shape",
                "resampling_fn_data_kwargs": {
                    "is_seg": false,
                    "order": 3,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "resampling_fn_seg_kwargs": {
                    "is_seg": true,
                    "order": 1,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "resampling_fn_probabilities": "resample_data_or_seg_to_shape",
                "resampling_fn_probabilities_kwargs": {
                    "is_seg": false,
                    "order": 1,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "architecture": {
                    "network_class_name": "dynamic_network_architectures.architectures.unet.PlainConvUNet",
                    "arch_kwargs": {
                        "n_stages": 8,
                        "features_per_stage": [
                            32,
                            64,
                            128,
                            256,
                            512,
                            512,
                            512,
                            512
                        ],
                        "conv_op": "torch.nn.modules.conv.Conv2d",
                        "kernel_sizes": [
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ],
                            [
                                3,
                                3
                            ]
                        ],
                        "strides": [
                            [
                                1,
                                1
                            ],
                            [
                                2,
                                2
                            ],
                            [
                                2,
                                2
                            ],
                            [
                                2,
                                2
                            ],
                            [
                                2,
                                2
                            ],
                            [
                                2,
                                2
                            ],
                            [
                                2,
                                2
                            ],
                            [
                                2,
                                2
                            ]
                        ],
                        "n_conv_per_stage": [
                            2,
                            2,
                            2,
                            2,
                            2,
                            2,
                            2,
                            2
                        ],
                        "n_conv_per_stage_decoder": [
                            2,
                            2,
                            2,
                            2,
                            2,
                            2,
                            2
                        ],
                        "conv_bias": true,
                        "norm_op": "torch.nn.modules.instancenorm.InstanceNorm2d",
                        "norm_op_kwargs": {
                            "eps": 1e-05,
                            "affine": true
                        },
                        "dropout_op": null,
                        "dropout_op_kwargs": null,
                        "nonlin": "torch.nn.LeakyReLU",
                        "nonlin_kwargs": {
                            "inplace": true
                        }
                    },
                    "_kw_requires_import": [
                        "conv_op",
                        "norm_op",
                        "dropout_op",
                        "nonlin"
                    ]
                },
                "batch_dice": true
            },
            "3d_lowres": {
                "data_identifier": "nnUNetPlans_3d_lowres",
                "preprocessor_name": "DefaultPreprocessor",
                "batch_size": 2,
                "patch_size": [
                    64,
                    192,
                    192
                ],
                "median_image_size_in_voxels": [
                    102,
                    292,
                    292
                ],
                "spacing": [
                    2.7318175000000005,
                    1.390475921094405,
                    1.390475921094405
                ],
                "normalization_schemes": [
                    "CTNormalization"
                ],
                "use_mask_for_norm": [
                    false
                ],
                "resampling_fn_data": "resample_data_or_seg_to_shape",
                "resampling_fn_seg": "resample_data_or_seg_to_shape",
                "resampling_fn_data_kwargs": {
                    "is_seg": false,
                    "order": 3,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "resampling_fn_seg_kwargs": {
                    "is_seg": true,
                    "order": 1,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "resampling_fn_probabilities": "resample_data_or_seg_to_shape",
                "resampling_fn_probabilities_kwargs": {
                    "is_seg": false,
                    "order": 1,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "architecture": {
                    "network_class_name": "dynamic_network_architectures.architectures.unet.PlainConvUNet",
                    "arch_kwargs": {
                        "n_stages": 6,
                        "features_per_stage": [
                            32,
                            64,
                            128,
                            256,
                            320,
                            320
                        ],
                        "conv_op": "torch.nn.modules.conv.Conv3d",
                        "kernel_sizes": [
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ]
                        ],
                        "strides": [
                            [
                                1,
                                1,
                                1
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                1,
                                2,
                                2
                            ]
                        ],
                        "n_conv_per_stage": [
                            2,
                            2,
                            2,
                            2,
                            2,
                            2
                        ],
                        "n_conv_per_stage_decoder": [
                            2,
                            2,
                            2,
                            2,
                            2
                        ],
                        "conv_bias": true,
                        "norm_op": "torch.nn.modules.instancenorm.InstanceNorm3d",
                        "norm_op_kwargs": {
                            "eps": 1e-05,
                            "affine": true
                        },
                        "dropout_op": null,
                        "dropout_op_kwargs": null,
                        "nonlin": "torch.nn.LeakyReLU",
                        "nonlin_kwargs": {
                            "inplace": true
                        }
                    },
                    "_kw_requires_import": [
                        "conv_op",
                        "norm_op",
                        "dropout_op",
                        "nonlin"
                    ]
                },
                "batch_dice": false,
                "next_stage": "3d_cascade_fullres"
            },
            "3d_fullres": {
                "data_identifier": "nnUNetPlans_3d_fullres",
                "preprocessor_name": "DefaultPreprocessor",
                "batch_size": 2,
                "patch_size": [
                    48,
                    224,
                    192
                ],
                "median_image_size_in_voxels": [
                    112.0,
                    512.0,
                    512.0
                ],
                "spacing": [
                    2.5,
                    0.7929689884185791,
                    0.7929689884185791
                ],
                "normalization_schemes": [
                    "CTNormalization"
                ],
                "use_mask_for_norm": [
                    false
                ],
                "resampling_fn_data": "resample_data_or_seg_to_shape",
                "resampling_fn_seg": "resample_data_or_seg_to_shape",
                "resampling_fn_data_kwargs": {
                    "is_seg": false,
                    "order": 3,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "resampling_fn_seg_kwargs": {
                    "is_seg": true,
                    "order": 1,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "resampling_fn_probabilities": "resample_data_or_seg_to_shape",
                "resampling_fn_probabilities_kwargs": {
                    "is_seg": false,
                    "order": 1,
                    "order_z": 0,
                    "force_separate_z": null
                },
                "architecture": {
                    "network_class_name": "dynamic_network_architectures.architectures.unet.PlainConvUNet",
                    "arch_kwargs": {
                        "n_stages": 6,
                        "features_per_stage": [
                            32,
                            64,
                            128,
                            256,
                            320,
                            320
                        ],
                        "conv_op": "torch.nn.modules.conv.Conv3d",
                        "kernel_sizes": [
                            [
                                1,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ],
                            [
                                3,
                                3,
                                3
                            ]
                        ],
                        "strides": [
                            [
                                1,
                                1,
                                1
                            ],
                            [
                                1,
                                2,
                                2
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                2,
                                2,
                                2
                            ],
                            [
                                1,
                                2,
                                2
                            ]
                        ],
                        "n_conv_per_stage": [
                            2,
                            2,
                            2,
                            2,
                            2,
                            2
                        ],
                        "n_conv_per_stage_decoder": [
                            2,
                            2,
                            2,
                            2,
                            2
                        ],
                        "conv_bias": true,
                        "norm_op": "torch.nn.modules.instancenorm.InstanceNorm3d",
                        "norm_op_kwargs": {
                            "eps": 1e-05,
                            "affine": true
                        },
                        "dropout_op": null,
                        "dropout_op_kwargs": null,
                        "nonlin": "torch.nn.LeakyReLU",
                        "nonlin_kwargs": {
                            "inplace": true
                        }
                    },
                    "_kw_requires_import": [
                        "conv_op",
                        "norm_op",
                        "dropout_op",
                        "nonlin"
                    ]
                },
                "batch_dice": true
            },
            "3d_cascade_fullres": {
                "inherits_from": "3d_fullres",
                "previous_stage": "3d_lowres"
            }
        },
        "experiment_planner_used": "ExperimentPlanner",
        "label_manager": "LabelManager",
        "foreground_intensity_properties_per_channel": {
            "0": {
                "max": 1038.0,
                "mean": 94.82794189453125,
                "median": 99.0,
                "min": -620.0,
                "percentile_00_5": -40.0,
                "percentile_99_5": 176.0,
                "std": 40.52839660644531
            }
        }
    }
}

Copy nnUNetPlans into Transfer Folder#

To share the nnUNet plans with the other sites, copy the nnUNetPlans.json file into the src/Spleen/pymaia_nvflare Folder.

[61]:
with open(Path("/home/maia-user/shared/src/Spleen/pymaia_nvflare/nnUNetPlans.json"),"w") as f:
    json.dump(nnunet_plans,f)

3. Preprocess#

After the nnUNet plans have been shared with the other sites, the data has to be preprocessed according to the nnUNet plans. This step has to be done on all the sites, except the one where the nnUNet experiment has been planned.

Submit Job#

[34]:
job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","preprocess")))
[35]:
sess.monitor_job(job_id)
[35]:
<MonitorReturnCode.JOB_FINISHED: 0>
[98]:
#print(sess.api.do_command(f"cat server log.txt")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} logfile_preprocess_out.log")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} logfile_preprocess_err.log")['data'][0]['data'])
Dataset109_Task09_Spleen
Preprocessing dataset Dataset109_Task09_Spleen
Configuration: 3d_fullres...
2024-12-16 14:13:18,552 - worker_process - INFO - Worker_process started.
2024-12-16 14:13:18,584 - AuxRunner - INFO - registered aux handler for topic __end_run__
2024-12-16 14:13:18,584 - AuxRunner - INFO - registered aux handler for topic __do_task__
2024-12-16 14:13:18,652 - CoreCell - INFO - site-2.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce: created backbone internal connector to tcp://localhost:7367 on parent
2024-12-16 14:13:18,653 - CoreCell - INFO - site-2.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce: created backbone external connector to grpc://localhost:8002
2024-12-16 14:13:18,653 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE tcp://localhost:7367] is starting
2024-12-16 14:13:18,653 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 ACTIVE grpc://localhost:8002] is starting
2024-12-16 14:13:18,654 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 127.0.0.1:40002 => 127.0.0.1:7367] is created: PID: 10699
2024-12-16 14:13:18,654 - FederatedClient - INFO - Wait for client_runner to be created.
2024-12-16 14:13:18,655 - FederatedClient - INFO - Got client_runner after 0.0007114410400390625 seconds
2024-12-16 14:13:18,656 - FederatedClient - INFO - Got the new primary SP: grpc://localhost:8002
2024-12-16 14:13:18,656 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2024-12-16 14:13:18,657 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-12-16 14:13:18,657 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => localhost:8002] is created: PID: 10699
2024-12-16 14:13:19,162 - Cell - INFO - broadcast: channel='aux_communication', topic='__sync_runner__', targets=['server.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce'], timeout=2.0
2024-12-16 14:13:19,183 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: synced to Server Runner in 0.5214066505432129 seconds
2024-12-16 14:13:19,184 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REQUEST
2024-12-16 14:13:19,184 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REPLY
2024-12-16 14:13:19,185 - ReliableMessage - INFO - enabled reliable message: max_request_workers=20 query_interval=2.0
2024-12-16 14:13:20,389 - AuxRunner - INFO - registered aux handler for topic fed.event
2024-12-16 14:13:20,389 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: client runner started
2024-12-16 14:13:20,445 - Communicator - INFO - Received from example_project server. getTask: preprocess size: 688B (688 Bytes) time: 0.054908 seconds
2024-12-16 14:13:20,445 - FederatedClient - INFO - pull_task completed. Task name:preprocess Status:True
2024-12-16 14:13:20,445 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: got task assignment: name=preprocess, id=aa5312dd-d56b-4077-8606-a51f1919b20e
2024-12-16 14:13:20,445 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: invoking task executor nnUNetExecutor
2024-12-16 14:13:21,330 - PyMAIA - INFO - PyMAIA GPU Available: True
2024-12-16 14:13:21,835 - numexpr.utils - INFO - Note: NumExpr detected 20 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2024-12-16 14:13:21,835 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.
2024-12-16 14:17:33,179 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: finished processing task
2024-12-16 14:17:33,179 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: try #1: sending task result to server
2024-12-16 14:17:33,179 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: checking task ...
2024-12-16 14:17:33,179 - Cell - INFO - broadcast: channel='aux_communication', topic='__task_check__', targets=['server.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce'], timeout=5.0
2024-12-16 14:17:33,183 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: start to send task result to server
2024-12-16 14:17:33,183 - FederatedClient - INFO - Starting to push execute result.
2024-12-16 14:17:33,187 - Communicator - INFO -  SubmitUpdate size: 4.9KB (4932 Bytes). time: 0.003701 seconds
2024-12-16 14:17:33,187 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: task result sent to server
2024-12-16 14:17:33,406 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: received request from Server to end current RUN
2024-12-16 14:17:35,189 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: started end-run events sequence
2024-12-16 14:17:35,189 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: ABOUT_TO_END_RUN fired
2024-12-16 14:17:35,189 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: Firing CHECK_END_RUN_READINESS ...
2024-12-16 14:17:35,190 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: END_RUN fired
2024-12-16 14:17:35,190 - ReliableMessage - INFO - ReliableMessage is shutdown
2024-12-16 14:17:35,192 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 10699
2024-12-16 14:17:35,193 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 10699
2024-12-16 14:17:35,193 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-12-16 14:17:35,193 - FederatedClient - INFO - Shutting down client run: site-2
2024-12-16 14:17:35,422 - ReliableMessage - INFO - shutdown reliable message monitor
2024-12-16 14:17:36,016 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: Client is stopping ...
2024-12-16 14:17:37,524 - MPM - INFO - MPM: Good Bye!
2024-12-16 14:13:25 - PyMAIA - INFO - PyMAIA GPU Available: True

  0%|                                                                                                                                                                                  | 0/24 [00:00<?, ?it/s]
  4%|███████                                                                                                                                                                   | 1/24 [00:12<04:53, 12.76s/it]
  8%|██████████████▏                                                                                                                                                           | 2/24 [00:20<03:35,  9.79s/it]
 12%|█████████████████████▎                                                                                                                                                    | 3/24 [00:30<03:23,  9.71s/it]
 17%|████████████████████████████▎                                                                                                                                             | 4/24 [00:33<02:25,  7.26s/it]
 21%|███████████████████████████████████▍                                                                                                                                      | 5/24 [00:47<03:00,  9.48s/it]
 25%|██████████████████████████████████████████▌                                                                                                                               | 6/24 [00:53<02:33,  8.54s/it]
 29%|█████████████████████████████████████████████████▌                                                                                                                        | 7/24 [01:05<02:42,  9.58s/it]
 33%|████████████████████████████████████████████████████████▋                                                                                                                 | 8/24 [01:18<02:49, 10.58s/it]
 38%|███████████████████████████████████████████████████████████████▊                                                                                                          | 9/24 [01:28<02:38, 10.59s/it]
 42%|██████████████████████████████████████████████████████████████████████▍                                                                                                  | 10/24 [01:32<02:00,  8.59s/it]
 46%|█████████████████████████████████████████████████████████████████████████████▍                                                                                           | 11/24 [01:44<02:02,  9.42s/it]
 50%|████████████████████████████████████████████████████████████████████████████████████▌                                                                                    | 12/24 [01:48<01:33,  7.81s/it]
 54%|███████████████████████████████████████████████████████████████████████████████████████████▌                                                                             | 13/24 [02:01<01:42,  9.36s/it]
 58%|██████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                      | 14/24 [02:15<01:48, 10.89s/it]
 62%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                               | 15/24 [02:29<01:47, 11.92s/it]
 67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                        | 16/24 [02:41<01:35, 11.92s/it]
 71%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋                                                 | 17/24 [02:56<01:29, 12.85s/it]
 75%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                          | 18/24 [03:07<01:12, 12.06s/it]
 79%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                   | 19/24 [03:22<01:04, 12.95s/it]
 83%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                            | 20/24 [03:30<00:46, 11.65s/it]
 88%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉                     | 21/24 [03:41<00:34, 11.46s/it]
 92%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉              | 22/24 [03:49<00:20, 10.18s/it]
 96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉       | 23/24 [03:56<00:09,  9.23s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [04:00<00:00,  7.84s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [04:00<00:00, 10.03s/it]

4. MONAI Bundle nnUNet Training#

Once the dataset has been prepared and preprocessed following the nnUNet plans, the nnUNet training is ready to begin. In this phase, we will train an nnUNet model using a MONAI Bundle. It’s important to note that the training at this stage will be conducted separately at each site, with the model saved in the MONAI Bundle format. There will be no aggregation of local gradients and no Federated Learning applied in this step. This approach is intended to test the training process and evaluate the model at each individual site, establishing a baseline before moving on to the Federated Learning phase.

To resume the training process from an existing checkpoint:

nvflare job create -j <JOB_DIR>/jobs/bundle_training -w <JOB_DIR>/bundle_training  -sd nnUNet_NVFlare/ -f bundle_training-client-<CLIENT_NAME>/config_fed_client.conf resume_training=epoch_number
[ ]:
from pathlib import Path

job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","bundle_training")))
[ ]:
job_id = "e5072797-f93b-4d43-9bbd-6d947f894856"

sess.monitor_job(job_id)
[ ]:
job_dir = sess.download_job_result(job_id)
[ ]:
from pathlib import Path


with open(Path(job_dir).joinpath("workspace","run_bundle_train","cmd.sh"),"r") as f:
    print(f.read())

To check the training logs, run the following command:

[ ]:

#bundle_name = "<BUNDLE_NAME>" #AutoPETLymphoma_nnUNet print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data']) print(sess.api.do_command(f"cat {client_id} logfile.log")['data'][0]['data']) print(sess.api.do_command(f"cat {client_id} logfile_train_out.log")['data'][0]['data']) print(sess.api.do_command(f"cat {client_id} logfile_train_err.log")['data'][0]['data']) #print(sess.api.do_command(f"ls {client_id} {bundle_name}/models")['data'][0]['data'])

4.1 Federated Learning Training#

After the individual training at each site, the Federated Learning training can be started. The Federated Learning training will aggregate the local gradients from the different sites and update the global model. The Federated Learning training will be conducted in rounds, with each round consisting of a number of local epochs. The Federated Learning training will be conducted in a secure and privacy-preserving manner, with the data remaining on the client side and only the local gradients being shared with the server.

Prior to starting the FL training, follow the steps below to correctly configure the federation:

  1. Copy the <BUNDLE_ROOT>/src folder to the src folder of the NVFlare jobs.

  2. Upload requirements.txt and the MONAI Bundle to the server.

  3. Upload the plans.json and dataset.json files to the server (in <BUNDLE_ROOT>/models). dataset.json can be in the form: json         {         "task": "Dataset109_Task09_Spleen",         "dim": 3,         "test_labels": true,         "tensorImageSize": "4D",         "channel_names": {"0": "ct"},         "labels": {"background": 0, "Spleen": 1},         "numTraining": 0,         "numTest": 0,         "training": [],         "test": [],         "file_ending": ".nii.gz"     }

  4. Specify BUNDLE_ROOT in the server bundle configuration file (<BUNDLE_ROOT>/configs/train.yaml).

  5. Install the required packages on the server: bash     pip install -r requirements.txt

  6. Reconfigure the clients accordingly, adding: yaml     pymaia_config_file: "<RESULTS_FOLDER>/Dataset<TASK_ID>_<TASK_NAME>.json"     nnunet_model_folder: "<RESULTS_FOLDER>/Dataset<TASK_ID>_<TASK_NAME>/<TRAINER_CLASS>__<PLANS_IDENTIFIER>_3d_fullres"     bundle_root: "<BUNDLE_ROOT>"

  7. Specify the following parameters in the Experiment configuration: yaml     num_rounds: 100     server_bundle_root: "/workspace/Spleen_Bundle"     start_round: 0     local_epochs: 10     Bundle_File: "<BUNDLE_FILE>"

  8. Copy the nnUNet Bundle into the src/<Project> folder.

  9. Re-execute the generate_job_configs script to update the job configurations.

Prepare Bundle for FL Training#

The MONAI Bundle has to be prepared for the Federated Learning training. In this step, the MONAI Bundle will be extracted on client, and the train and evaluation bundle parameters will be overwritten to match the Federated Learning training configuration.

To prepare the MONAI Bundle for the Federated Learning training, run the following command:

[22]:
from pathlib import Path

job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","prepare_bundle")))
[23]:
sess.monitor_job(job_id)
[23]:
<MonitorReturnCode.JOB_FINISHED: 0>
[24]:
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
2024-12-16 15:01:02,444 - worker_process - INFO - Worker_process started.
2024-12-16 15:01:02,476 - AuxRunner - INFO - registered aux handler for topic __end_run__
2024-12-16 15:01:02,476 - AuxRunner - INFO - registered aux handler for topic __do_task__
2024-12-16 15:01:02,545 - CoreCell - INFO - site-2.140eb6cc-3036-4721-a247-9497f8392cdd: created backbone internal connector to tcp://localhost:7367 on parent
2024-12-16 15:01:02,545 - CoreCell - INFO - site-2.140eb6cc-3036-4721-a247-9497f8392cdd: created backbone external connector to grpc://localhost:8002
2024-12-16 15:01:02,545 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE tcp://localhost:7367] is starting
2024-12-16 15:01:02,546 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 ACTIVE grpc://localhost:8002] is starting
2024-12-16 15:01:02,546 - FederatedClient - INFO - Wait for client_runner to be created.
2024-12-16 15:01:02,547 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 127.0.0.1:33150 => 127.0.0.1:7367] is created: PID: 18891
2024-12-16 15:01:02,547 - FederatedClient - INFO - Got client_runner after 0.00031495094299316406 seconds
2024-12-16 15:01:02,547 - FederatedClient - INFO - Got the new primary SP: grpc://localhost:8002
2024-12-16 15:01:02,548 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2024-12-16 15:01:02,551 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002
2024-12-16 15:01:02,552 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => localhost:8002] is created: PID: 18891
2024-12-16 15:01:03,054 - Cell - INFO - broadcast: channel='aux_communication', topic='__sync_runner__', targets=['server.140eb6cc-3036-4721-a247-9497f8392cdd'], timeout=2.0
2024-12-16 15:01:03,059 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: synced to Server Runner in 0.5054581165313721 seconds
2024-12-16 15:01:03,059 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REQUEST
2024-12-16 15:01:03,059 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REPLY
2024-12-16 15:01:03,059 - ReliableMessage - INFO - enabled reliable message: max_request_workers=20 query_interval=2.0
2024-12-16 15:01:04,261 - AuxRunner - INFO - registered aux handler for topic fed.event
2024-12-16 15:01:04,261 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: client runner started
2024-12-16 15:01:04,309 - Communicator - INFO - Received from example_project server. getTask: prepare_bundle size: 696B (696 Bytes) time: 0.047104 seconds
2024-12-16 15:01:04,309 - FederatedClient - INFO - pull_task completed. Task name:prepare_bundle Status:True
2024-12-16 15:01:04,309 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd]: got task assignment: name=prepare_bundle, id=e7202854-1e39-40df-bf79-d0dd55f1dc6c
2024-12-16 15:01:04,309 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: invoking task executor nnUNetExecutor
2024-12-16 15:01:05,275 - PyMAIA - INFO - PyMAIA GPU Available: True
2024-12-16 15:01:05,878 - numexpr.utils - INFO - Note: NumExpr detected 20 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2024-12-16 15:01:05,879 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.
2024-12-16 15:01:08,389 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: finished processing task
2024-12-16 15:01:08,390 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: try #1: sending task result to server
2024-12-16 15:01:08,390 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: checking task ...
2024-12-16 15:01:08,390 - Cell - INFO - broadcast: channel='aux_communication', topic='__task_check__', targets=['server.140eb6cc-3036-4721-a247-9497f8392cdd'], timeout=5.0
2024-12-16 15:01:08,394 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: start to send task result to server
2024-12-16 15:01:08,394 - FederatedClient - INFO - Starting to push execute result.
2024-12-16 15:01:08,398 - Communicator - INFO -  SubmitUpdate size: 918B (918 Bytes). time: 0.003894 seconds
2024-12-16 15:01:08,398 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: task result sent to server
2024-12-16 15:01:08,797 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd]: received request from Server to end current RUN
2024-12-16 15:01:10,400 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: started end-run events sequence
2024-12-16 15:01:10,401 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: ABOUT_TO_END_RUN fired
2024-12-16 15:01:10,402 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: Firing CHECK_END_RUN_READINESS ...
2024-12-16 15:01:10,402 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: END_RUN fired
2024-12-16 15:01:10,403 - ReliableMessage - INFO - ReliableMessage is shutdown
2024-12-16 15:01:10,412 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 18891
2024-12-16 15:01:10,414 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 18891
2024-12-16 15:01:10,415 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]
2024-12-16 15:01:10,416 - FederatedClient - INFO - Shutting down client run: site-2
2024-12-16 15:01:10,670 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: Client is stopping ...
2024-12-16 15:01:11,069 - ReliableMessage - INFO - shutdown reliable message monitor
2024-12-16 15:01:12,176 - MPM - INFO - MPM: Good Bye!
[25]:
job_dir = sess.download_job_result(job_id)

import json
from pathlib import Path

with open(Path(job_dir).joinpath("workspace","run_bundle_prepare","bundle_prepare.json"),"r") as f:
    print(json.load(f))
{'site-2': {'Bundle_Path': '/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-2/startup/../140eb6cc-3036-4721-a247-9497f8392cdd/app_site-2/custom/Spleen_Bundle'}, 'site-1': {'Bundle_Path': '/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-1/startup/../140eb6cc-3036-4721-a247-9497f8392cdd/app_site-1/custom/Spleen_Bundle'}}

Run FL Training#

[ ]:
from pathlib import Path

job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","fl_bundle")))
[64]:
sess.abort_job(job_id)
[72]:
client_id = "site-1"
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
session_inactive

To download the trained models from the clients and upload them to the server, run the following command:

[9]:
job_dir = sess.download_job_result(job_id)
[ ]:
import minio

client = minio.Minio(
            endpoint="<MINIO_URL>",
              access_key="<MINIO_ACCESS_KEY>",
                secret_key="<MINIO_SECRET_KEY>",
                secure=True
)
[ ]:
client.make_bucket("<EXPERIMENT_NAME>")

client.fput_object("<EXPERIMENT_NAME>","Lymphoma_nnUNet_Bundle/models/FL_global_model.pt","FL_global_model.pt")
client.fput_object("<EXPERIMENT_NAME>","Lymphoma_nnUNet_Bundle/models/best_FL_global_model.pt","best_FL_global_model.pt")

Resume FL Training#

To resume the Federated Learning training from an existing checkpoint:

  1. Download the global models from the server (sess.download_job_result(job_id)) and upload them to the MONAI Bundle in the server (<BUNDLE_ROOT>/models)

  2. Add the following parameters to the server configuration file: yaml     nnunet_trainer_def:       pretrained_model: "<BUNDLE_ROOT>/models/FL_global_model.pt"

  3. Update the Experiment Configuration file with the new start_round and num_rounds values.

    start_round: <START_ROUND>
    num_rounds: <INITIAL_ROUNDS - START_ROUND>
    
  4. Re-execute the generate_job_configs script to update the job configurations.

5. MONAI Bundle to nnUNet Conversion#

Once the training is completed, the model can be converted to the nnUNet format. This step is necessary to be able to validate the model using the nnUNet framework.

Since the training is performed using the MONAI Bundle format, the model has to be converted to the nnUNet format. The conversion will be done using the convert_model job, which will convert the MONAI Bundle model to the nnUNet format.

[ ]:
from pathlib import Path
job_id = sess.submit_job(str(Path("<JOB_DIR>").joinpath("jobs","convert_model")))
[ ]:
sess.monitor_job(job_id)
[ ]:
client_id = "KTH-Cloud"
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
print(sess.api.do_command(f"cat server log.txt")['data'][0]['data'])
[ ]:
job_dir = sess.download_job_result(job_id)
[ ]:
from pathlib import Path


with open(Path(job_dir).joinpath("workspace","model_convert","path.txt"),"r") as f:
    print(f.read())

5.1 Fed Model Conversion#

[ ]:
job_id = sess.submit_job(str(Path(JOB_DIR).joinpath("jobs","convert_fed_model")))

6. Validation#

After the model has been converted to the nnUNet format, it can be validated using the nnUNet framework. The validation will be done using the validate job, which will validate the model using the nnUNet framework.

[ ]:
from pathlib import Path
job_id = sess.submit_job(str(Path("<JOB_DIR>").joinpath("jobs","validate")))
[ ]:
print(sess.api.do_command(f"cat {client_id} logfile_validate_out.log")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} logfile_validate_err.log")['data'][0]['data'])
[ ]:
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])

To extract the validation results and display them in a table, run the following commands:

[ ]:
job_dir = sess.download_job_result(job_id)
[ ]:
import json
from pathlib import Path


with open(Path(job_dir).joinpath("workspace","validation","summary.json"),"r") as f:
    summary = json.load(f)
[ ]:
config_dict = {
"label_dict": {"background": 0,
    "Lesion": 1
},
    "label_suffix":".nii.gz"
}
[ ]:
df = []

label_to_name = {v: k for k, v in config_dict["label_dict"].items()}

for case in summary['metric_per_case']:
    for label_id in case['metrics']:
        for metric in case['metrics'][label_id]:
            df.append({
                "Case": Path(case['reference_file']).name[:-len(config_dict["label_suffix"])],
                "Label": label_to_name[int(label_id)],
                "Metric": metric,
                "Value": case['metrics'][label_id][metric]
            })
[ ]:
import pandas as pd
df = pd.DataFrame(df)
[ ]:
import dtale.app as dtale_app
import dtale

dtale_app.JUPYTER_SERVER_PROXY = False

d = dtale.show(df,
               host="127.0.0.1",
              )
[ ]:
DTALE_URL = d._main_url
[ ]:
DTALE_URL

7. Package MONAI Bundle#

After the model has been validated, it can be packaged into a MONAI Bundle. The MONAI Bundle will contain the model weights and the necessary configuration files to run the model. The packaging will be done using the package_monai_bundle job, which will package the model into a MONAI Bundle.

The MONAI Bundle will be stored in the MLFlow server, available for download and deployment.

To combine the MONAI Bundle with an existing MLFlow experiment, run the following command:

nvflare job create -j <JOB_DIR>/jobs/package_monai_bundle -w <JOB_DIR>/package_monai_bundle  -sd nnUNet_NVFlare/ -f package_monai_bundle-client-<CLIENT_NAME>/config_fed_client.conf MLFlow_Run_ID=<MLFLOW_RUN_ID>
[ ]:
from pathlib import Path


job_id = sess.submit_job(str(Path("<JOB_DIR>").joinpath("jobs","package_monai_bundle")))
[ ]:
client_id = "<CLIENT_NAME>"

print(sess.api.do_command(f"cat {client_id} logfile_package_out.log")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} logfile_package_err.log")['data'][0]['data'])
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
[ ]:
job_dir = sess.download_job_result(job_id)

7.1 Package external nnUNet Experiments#

In case the nnUNet experiment has been planned, preprocessed and trained indipendentely from the NVFlare API, the model can be packaged into a MONAI Bundle using the package_monai_bundle job. To package an external nnUNet experiment, run the following command:

nvflare job create -j <JOB_DIR>/jobs/package_monai_bundle -w <JOB_DIR>/package_monai_bundle  -sd nnUNet_NVFlare/ -f package_monai_bundle-client-<CLIENT_NAME>/config_fed_client.conf external_training=True

8. Cross Site Evaluation#

[ ]:
!nvflare job create -j jobs/cross_Site_Evaluation_nnUNet -w cross_Site_Evaluation_nnUNet/  -sd nnUNet_NVFlare/ --force
[ ]:
from pathlib import Path


job_id = sess.submit_job(str(Path("..").joinpath("..","jobs","cross_Site_Evaluation_nnUNet")))
[ ]:
sess.abort_job(job_id)
[ ]:
client_id = "msf"
print(sess.api.do_command(f"cat {client_id} {job_id}/log.txt")['data'][0]['data'])
#print(sess.api.do_command(f"ls {client_id} Cross-Site-Validation/kth-cloud_positive_baseline_v2/extra_files")['data'][0]['data'])
#print(sess.api.do_command(f"ls {client_id} Cross-Site-Validation/bundles/kth-cloud_positive_baseline_v2/models")['data'][0]['data'])
#print(sess.api.do_command(f"ls {client_id} Cross-Site-Validation/inputs")['data'][0]['data'])
[ ]:
job_dir = sess.download_job_result(job_id)
[ ]:
import json
from pathlib import Path


with open(Path(job_dir).joinpath("workspace","validation","summary.json"),"r") as f:
    cross_site_summary = json.load(f)