{ "cells": [ { "cell_type": "markdown", "id": "40543901dbd88c08", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "# nnUNet Experiment in NVFlare\n", "\n", "This tutorial will guide you through the process of running a nnUNet experiment in a Federated Learning context, by using the NVIDIA Flare API.\n", "\n", "In this tutorial, we will use the nnUNet framework to train a model on a dataset that is distributed across multiple sites. The training will be done in a secure and privacy-preserving manner, by using the Federated Learning capabilities of the NVIDIA Flare API.\n", "\n", "### Model Training\n", "\n", "In detail, the following steps will be performed:\n", "1. Dataset Preparation: The dataset in the different sites will be prepared for training, harmonizing the data and creating the necessary files for training according to the nnUNet framework.\n", "2. nnUNet Experiment Planning and Preprocessing: One of the sites will be selected as the main site, where the nnUNet experiment will be planned and the data will be preprocessed. This step has to be done only on one site, as the nnUNet plans will be shared with the other sites.\n", "3. nnUNet Preprocessing: The data will be preprocessed according to the nnUNet plan in all the other sites.\n", "4. nnUNet Training: The model will be trained on the data of all the sites, using the nnUNet framework and aggregating the local gradients from the different sites.\n", "\n", "### Offline Training\n", "Offline training is also made available for the clients, to test the training process and evaluate the model at each individual site, establishing a baseline before moving on to the Federated Learning phase.\n", "In this fase, Data Preparation, Experiment Planning and Preprocessing, Training, Model Conversion, Validation and Packaging will be done on each site separately.\n", "\n", "### Model Deployment\n", "In addition, some specific jobs will be executed to:\n", "- Convert the model from the MONAI Bundle standard to the nnUNet format.\n", "- Validate the model using the nnUNet framework.\n", "- Package the model into a MONAI Bundle, ready for deployment.\n", "- Evaluate the model across different sites.\n", "\n", "### PREREQUISITES\n", "This tutorial assumes that you have already installed the NVIDIA Flare API and have access to a Federated Learning cluster, with ``Lead`` role. If you haven't done so, please refer to the [Getting Started](../README.md#getting-started) section of the Tutorial section.\n", "\n", "Additionally, all the sites should have the necessary data for training ready in a known location ( not necessarily the same location across all sites)." ] }, { "cell_type": "markdown", "id": "95063771", "metadata": {}, "source": [ "## Configure the Federation in POC Mode\n", "\n", "Before starting the tutorial, we need to configure the federation in NVFlare POC (Proof of Concept) mode. This mode allows the federation to be configured within a single site, which is useful for testing and development purposes." ] }, { "cell_type": "code", "execution_count": null, "id": "8421b051", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "\n", "export NVFLARE_POC_WORKSPACE=\"/home/maia-user/Tutorials/NVFlare_POC\"\n", "\n", "/opt/conda/envs/MAIA/bin/nvflare poc prepare" ] }, { "cell_type": "code", "execution_count": null, "id": "897be93f", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "\n", "export NVFLARE_POC_WORKSPACE=\"/home/maia-user/Tutorials/NVFlare_POC\"\n", "\n", "/opt/conda/envs/MAIA/bin/nvflare poc start" ] }, { "cell_type": "markdown", "id": "18f661ff-8a1d-4ab4-94d3-4d51858caebb", "metadata": {}, "source": [ "or" ] }, { "cell_type": "code", "execution_count": null, "id": "f2dde56c-7b66-4a47-9b18-33c84b6561da", "metadata": {}, "outputs": [], "source": [ "from nvflare.tool.poc.poc_commands import _prepare_poc, _start_poc, _stop_poc, _clean_poc\n", "\n", "_prepare_poc(\n", " [\"site-1\",\"site-2\"],\n", " 2,\n", " \"/home/maia-user/Tutorials/NVFlare_POC\",\n", ")\n", "_start_poc(\"/home/maia-user/Tutorials/NVFlare_POC\",[0])" ] }, { "cell_type": "markdown", "id": "a57e085b", "metadata": {}, "source": [ "## Split the Data Across Sites\n", "\n", "After starting the federation in POC mode, we need to split the data across the two different sites. In this tutorial, we will randomly split the Decathlon Spleen dataset into two parts, one for each site.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cfc61419", "metadata": {}, "outputs": [], "source": [ "from PyMAIA.utils.file_utils import subfiles\n", "from random import shuffle\n", "import shutil\n", "from pathlib import Path\n", "image_dir = \"/home/maia-user/Tutorials/MAIA/Task09_Spleen/imagesTr\"\n", "label_dir = \"/home/maia-user/Tutorials/MAIA/Task09_Spleen/labelsTr\"\n", "\n", "# Site 1 will have the \"Decathlon\" dataset format\n", "Path(\"/home/maia-user/Tutorials/NVFlare_POC/data/site-1/imagesTr\").mkdir(parents=True, exist_ok=True)\n", "Path(\"/home/maia-user/Tutorials/NVFlare_POC/data/site-1/labelsTr\").mkdir(parents=True, exist_ok=True)\n", "\n", "# Site 2 will have the \"Subfolders\" dataset format\n", "Path(\"/home/maia-user/Tutorials/NVFlare_POC/data/site-2\").mkdir(parents=True, exist_ok=True)\n", "\n", "images = subfiles(image_dir, suffix=\".nii.gz\")\n", "labels = subfiles(label_dir, suffix=\".nii.gz\")\n", "\n", "print(f\"Images: {len(images)}\")\n", "print(f\"Labels: {len(labels)}\")\n", "\n", "shuffle(images)\n", "\n", "for image in images[:len(images)//2]:\n", " shutil.copy(image, \"/home/maia-user/Tutorials/NVFlare_POC/data/site-1/imagesTr\")\n", " shutil.copy(image.replace(\"imagesTr\", \"labelsTr\"), \"/home/maia-user/Tutorials/NVFlare_POC/data/site-1/labelsTr\")\n", "\n", "for image in images[len(images)//2:]:\n", " id = image.split(\"/\")[-1].split(\".\")[0]\n", " Path(f\"/home/maia-user/Tutorials/NVFlare_POC/data/site-2/{id}\").mkdir(parents=True, exist_ok=True)\n", " shutil.copy(image, f\"/home/maia-user/Tutorials/NVFlare_POC/data/site-2/{id}/{id}_CT.nii.gz\")\n", " shutil.copy(image.replace(\"imagesTr\", \"labelsTr\"), f\"/home/maia-user/Tutorials/NVFlare_POC/data/site-2/{id}/{id}_label.nii.gz\")\n" ] }, { "cell_type": "markdown", "id": "d788842381a2a684", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## FL Cluster Authentication" ] }, { "cell_type": "code", "execution_count": 3, "id": "de790771-2f3b-4a19-9b2d-be689445cdc2", "metadata": { "ExecuteTime": { "end_time": "2024-09-30T18:19:11.262605Z", "start_time": "2024-09-30T18:19:07.990219Z" }, "tags": [] }, "outputs": [], "source": [ "from nvflare.fuel.flare_api.flare_api import new_secure_session\n", "sess = new_secure_session(\n", " \"admin@nvidia.com\",\n", " \"NVFlare_POC/example_project/prod_00/admin@nvidia.com\"\n", ")" ] }, { "cell_type": "code", "execution_count": 4, "id": "b6aa80a4-d5c2-49c7-8054-8c7392bf8fd7", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "SystemInfo\n", "server_info:\n", "status: stopped, start_time: Tue Dec 17 07:16:29 2024\n", "client_info:\n", "site-2(last_connect_time: Tue Dec 17 07:16:36 2024)\n", "site-1(last_connect_time: Tue Dec 17 07:16:37 2024)\n", "job_info:\n", "\n" ] } ], "source": [ "print(sess.get_system_info())" ] }, { "cell_type": "markdown", "id": "2d1b313706f2a9f3", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## List Jobs" ] }, { "cell_type": "code", "execution_count": 5, "id": "997fc41a-3720-4656-b8a8-d0a66dd02d61", "metadata": { "tags": [] }, "outputs": [], "source": [ "jobs = sess.list_jobs()" ] }, { "cell_type": "code", "execution_count": 6, "id": "b52ec84f-b654-4b5c-8fcf-6ae8704c95b2", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'job_id': '6b26b950-4f40-40e0-81ae-27430d5f24a7',\n", " 'job_name': 'fl_bundle',\n", " 'status': 'FINISHED:ABANDONED',\n", " 'submit_time': '2024-12-16T16:36:25.865534+00:00',\n", " 'duration': 'N/A'}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "jobs[-1]" ] }, { "cell_type": "code", "execution_count": 8, "id": "403328c4-4c08-4566-a748-48cb514c934c", "metadata": {}, "outputs": [], "source": [ "job_id = jobs[-1]['job_id']" ] }, { "cell_type": "markdown", "id": "26a852f47f73063e", "metadata": {}, "source": [ "## Terminate Jobs" ] }, { "cell_type": "code", "execution_count": 10, "id": "f1c5b9a823090c1", "metadata": {}, "outputs": [], "source": [ "sess.abort_job(job_id)" ] }, { "cell_type": "markdown", "id": "ec1669dc-daba-439d-8d7c-6c116b463aa3", "metadata": {}, "source": [ "## TODO\n", "\n", "pymaia_nvflare from official PyMAIA\n", "patient_id_in_file_identifier: True #Add Docs" ] }, { "cell_type": "markdown", "id": "fdce8b8961132613", "metadata": {}, "source": [ "## Job Preparation\n", "\n", "The first step in the Federated Learning process is to prepare the job configuration files. The job configuration files contain the necessary information to run the job on the Federated Learning cluster, such as the job type, the resources required, and the parameters for the job execution.\n", "\n", "The job configurations files are automatically generated by the script `generate_job_configs`, which is installed together with this package. The script takes as input client-specific configuration, together with the experiment-specific configuration, and generates the job configuration files for each client.\n", "\n", "\n", "### Client Configuration\n", "The client-specific configuration file should be in the following format:\n", "\n", "```yaml\n", "data_dir: \"\"\n", "modality_dict:\n", " ct: \"\"\n", " label: \"\"\n", "dataset_format: \"\"\n", "patient_id_in_file_identifier: True\n", "nnunet_root_folder: \"\"\n", "Client_Name: \"\"\n", "pymaia_config_file: \"\"\n", "nnunet_model_folder: \"\"\n", "bundle_root: \"\"\n", "```\n", "\n", "`dataset_format` should refer to one of these three different formats, according to the `data_dir` structure:\n", "1. `subfolders`: The dataset is organized in subfolders, where each subfolder corresponds to a subject and contains the images and labels for that subject.\n", "```plaintext\n", " [Dataset_folder]\n", " [Subject_0]\n", " - Subject_0_image0.nii.gz # Subject_0 modality 0\n", " - Subject_0_image1.nii.gz # Subject_0 modality 1\n", " - Subject_0_mask.nii.gz # Subject_0 semantic segmentation mask\n", " [Subject_1]\n", " - Subject_1_image0.nii.gz # Subject_1 modality 0\n", " - Subject_1_image1.nii.gz # Subject_1 modality 1\n", " - Subject_1_mask.nii.gz # Subject_1 semantic segmentation mask\n", " ...\n", "\n", "```\n", "2. `decathlon`: The dataset is organized in the format of the Medical Decathlon challenge, where the images and labels are stored in separate folders.\n", "```plaintext\n", " [Dataset_folder]\n", " [imagesTr]\n", " - Subject_0_image0.nii.gz # Subject_0 modality 0\n", " - Subject_0_image1.nii.gz # Subject_0 modality 1\n", " - Subject_1_image0.nii.gz # Subject_1 modality 0\n", " - Subject_1_image1.nii.gz # Subject_1 modality 1\n", " [labelsTr]\n", " - Subject_1_mask.nii.gz # Subject_1 semantic segmentation mask\n", " - Subject_0_mask.nii.gz # Subject_0 semantic segmentation mask\n", " ...\n", "\n", "```\n", "\n", "3. `nnunet`: The dataset has been already prepared according to the nnUNet framework, with the images and labels stored in separate folders.\n", "```plaintext\n", " [nnUNet_raw]\n", " [DatasetXYZ_TaskName] # THIS IS THE DATASET FOLDER\n", " dataset.json\n", " [imagesTr]\n", " - Subject_0_image0.nii.gz # Subject_0 modality 0\n", " - Subject_0_image1.nii.gz # Subject_0 modality 1\n", " - Subject_1_image0.nii.gz # Subject_1 modality 0\n", " - Subject_1_image1.nii.gz # Subject_1 modality 1\n", " [labelsTr]\n", " - Subject_1_mask.nii.gz # Subject_1 semantic segmentation mask\n", " - Subject_0_mask.nii.gz # Subject_0 semantic segmentation mask\n", " ...\n", "\n", "```\n", "\n", "`nnunet_root_folder` should refer to the root folder used by the nnUnet framework, where the nnUNet experiments are stored.\n", "For the `subfolders` and `decathlon` dataset formats, this folder is created during the dataset preparation step. \n", "For the `nnunet` dataset format, this folder should contain the nnUNet experiments, with the following structure:\n", "```plaintext\n", " [nnunet_root_folder]\n", " [nnUNet_raw]\n", " [DatasetXYZ_TaskName]\n", " dataset.json\n", " [imagesTr]\n", " - Subject_0_image0.nii.gz # Subject_0 modality 0\n", " - Subject_0_image1.nii.gz # Subject_0 modality 1\n", " - Subject_1_image0.nii.gz # Subject_1 modality 0\n", " - Subject_1_image1.nii.gz # Subject_1 modality 1\n", " [labelsTr]\n", " - Subject_1_mask.nii.gz # Subject_1 semantic segmentation mask\n", " - Subject_0_mask.nii.gz # Subject_0 semantic segmentation mask\n", " ...\n", "\n", "```\n", "\n", "`modality_dict` is a dictionary that maps the modality names to the file suffixes. The suffixes are used to identify the files that correspond to the different modalities in the dataset. For example, if the CT images have the suffix `_CT.nii.gz`, the entry in the `modality_dict` should be `ct: \"_CT.nii.gz\"`.\n", "\n", "`pymaia_config_file` is the path to the configuration file generated by PyMAIA. The file can be found in the `results_folder` ( `nnunet_root_folder/TaskName/TaskName_results`) of the PyMAIA experiment, with the name `DatasetXYZ_TaskName.json`.\n", "\n", "`nnunet_model_folder` is the folder where the nnUNet folds and models are stored. This folder is created during the nnUNet experiment run:\n", "`results_folder/DatasetXYZ_TaskName/TrainerClass__PlansName__3d_fullres`\n", "\n", "`bundle_root` is the root folder where the MONAI Bundle will be stored in the client. It can be retrieved at the end of the `prepare_bundle` job.\n", "\n", "### Experiment Configuration\n", "\n", "The experiment-specific configuration file should be in the following format:\n", "\n", "```yaml\n", "task_ID: \n", "Experiment Name: \n", "label_dict:\n", " background: 0\n", " Lesion: 1\n", "MLFlow_Token: \n", "Tracking_URI: \n", "Bundle_File: \n", "num_rounds: 100\n", "start_round: 0\n", "local_epochs: 10\n", "server_bundle_root: \"\"\n", "Minio_Endpoint: \"MINIO_URL\"\n", "Minio_Access_Key: \"ACCESS_KEY\"\n", "Minio_Secret_Key: \"SECRET_KEY\"\n", "Minio_Bucket: \"BUCKET_NAME\"\n", "```\n", "`task_ID` and `Experiment Name` are used as a reference to the nnUNet task ID and name, respectively. These values are used to identify the nnUNet experiment in the nnUNet framework. `Experiment Name` is also used to identify the experiment in the MLFlow server, and to generate the zipped nnUNet model file (as `.zip`).\n", "\n", "`label_dict` is a dictionary that maps the label names to the label values. The label values are used to identify the different classes in the semantic segmentation masks. For example, if the background class is labeled as `0` and the lesion class is labeled as `1`, the entry in the `label_dict` should be `background: 0, Lesion: 1`.\n", "\n", "`MLFlow_Token` and `Tracking_URI` are used to connect to the MLFlow server, where the experiments are logged, and the trained models are uploaded.\n", "\n", "`Bundle_File` is the name of the zipped MONAI Bundle file that contains the python code and the necessary configuration files to run the model training. This file will be uploaded on each site during the training phase, during the execution of the training job.\n", "\n", "`server_bundle_root` is the root folder where the MONAI Bundle will be stored on the server.\n", "\n", "The MinIO parameters are needed to connect to the MinIO server and upload the trained FL models. You can leave these parameters empty if you are not using MinIO.\n", "\n", "\n", "### Prepare the Job Configuration Files\n", "\n", "To prepare the job configuration files, run the following command:\n", "\n", "```bash\n", "generate_job_configs -c ... -e -sd -j \n", "```\n", "where:\n", "- ``, ``, ... are the client-specific configuration files for each client.\n", "- `` is the experiment-specific configuration file.\n", "- `` is the directory where the job scripts and python files are stored [nnUnet_NVFlare](./nnUNet_NVFlare).\n", "- `` is the directory where the job configuration files will be stored." ] }, { "cell_type": "code", "execution_count": 1, "id": "7af82cf7-a0b9-4cf7-ae47-03a7a1d43927", "metadata": {}, "outputs": [], "source": [ "import sys\n", "from pathlib import Path\n", " \n", "ROOT_FOLDER = \"/home/maia-user/shared\"\n", "sys.path.append(ROOT_FOLDER)" ] }, { "cell_type": "code", "execution_count": 16, "id": "1464aecf-f69b-4cd5-b4ab-82e7b748ddf6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting /home/maia-user/shared/Experiments/Spleen.yaml\n" ] } ], "source": [ "%%writefile /home/maia-user/shared/Experiments/Spleen.yaml\n", "\n", "task_ID: \"109\"\n", "Experiment Name: Task09_Spleen\n", "label_dict:\n", " background: 0\n", " Spleen: 1\n", "MLFlow_Token: \"\"\n", "Tracking_URI: \"https://monai-demo.maia.cloud.cbh.kth.se/mlflow\"\n", "Bundle_File: \"Spleen_Bundle.zip\"\n", "num_rounds: 10\n", "start_round: 0\n", "local_epochs: 10\n", "server_bundle_root: \"/home/maia-user/shared/src/Spleen_Bundle\"\n", "Minio_Endpoint: \"\"\n", "Minio_Access_Key: \"\"\n", "Minio_Secret_Key: \"\"\n", "Minio_Bucket: \"\"" ] }, { "cell_type": "code", "execution_count": 65, "id": "ede053d8-280a-4541-8c2d-41c02c54878c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting /home/maia-user/shared/Clients/site-1.yaml\n" ] } ], "source": [ "%%writefile /home/maia-user/shared/Clients/site-1.yaml\n", "\n", "data_dir: \"/home/maia-user/Tutorials/NVFlare_POC/data/site-1\"\n", "modality_dict:\n", " ct: \".nii.gz\"\n", " label: \".nii.gz\"\n", "dataset_format: \"decathlon\"\n", "patient_id_in_file_identifier: True\n", "nnunet_root_folder: \"/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-1\"\n", "Client_Name: \"site-1\"\n", "pymaia_config_file: \"/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-1/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen.json\"\n", "nnunet_model_folder: \"/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-1/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen/nnUNetTrainer__nnUNetPlans_3d_fullres\"\n", "bundle_root: \"/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-1/Spleen_Bundle\"" ] }, { "cell_type": "code", "execution_count": 66, "id": "dfe0bdc6-b7cb-4a0a-9b2f-27c30452e843", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting /home/maia-user/shared/Clients/site-2.yaml\n" ] } ], "source": [ "%%writefile /home/maia-user/shared/Clients/site-2.yaml\n", "\n", "data_dir: \"/home/maia-user/Tutorials/NVFlare_POC/data/site-2\"\n", "modality_dict:\n", " ct: \"_CT.nii.gz\"\n", " label: \"_label.nii.gz\"\n", "dataset_format: \"subfolders\"\n", "patient_id_in_file_identifier: True\n", "nnunet_root_folder: \"/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-2\"\n", "Client_Name: \"site-2\"\n", "pymaia_config_file: \"/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-2/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen.json\"\n", "nnunet_model_folder: \"/home/maia-user/Tutorials/NVFlare_POC/Fed-nnUNet/site-2/Task09_Spleen/Task09_Spleen_results/Dataset109_Task09_Spleen/nnUNetTrainer__nnUNetPlans_3d_fullres\"\n", "bundle_root: \"/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-2/Spleen_Bundle\"" ] }, { "cell_type": "code", "execution_count": 7, "id": "332b56e4-f896-49b8-8721-4a37d43a2078", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Overwriting /home/maia-user/shared/src/Spleen/requirements.txt\n" ] } ], "source": [ "%%writefile /home/maia-user/shared/src/Spleen/requirements.txt\n", "pymaia-learn==1.2.1\n", "nnunetv2==2.5.1\n", "fire\n", "pytorch-ignite==0.4.11\n", "monai[nibabel, skimage, scipy, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, pandas, mlflow, matplotlib, pydicom]\n", "pydicom==2.4.4\n", "minio\n", "numpy==1.24.0\n", "pandas\n", "mlflow\n", "monai-nvflare" ] }, { "cell_type": "code", "execution_count": 67, "id": "91e31d40-9c32-4c44-925e-3542c866c6b5", "metadata": {}, "outputs": [], "source": [ "experiment = \"Spleen\"\n", "clients = [\n", " \"site-1\",\n", " \"site-2\"\n", "]" ] }, { "cell_type": "code", "execution_count": 68, "id": "36832ffc-5057-4a35-bcc6-ac798b9cbe41", "metadata": {}, "outputs": [], "source": [ "%%bash\n", "\n", "rm -r /home/maia-user/shared/Jobs/Spleen/" ] }, { "cell_type": "code", "execution_count": 69, "id": "d18cdd17-8786-4cca-a0f1-93776f7299b3", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "The following are the variables you can change in the template\n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " \n", " job folder: /home/maia-user/shared/Jobs/Spleen/jobs/prepare \n", " \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " file_name var_name value component \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " meta.conf mandatory_clients ['site-1', 'site-2'] \n", " meta.conf min_clients 1 \n", " meta.conf prepare-client-site-1 ['site-1'] \n", " meta.conf prepare-client-site-2 ['site-2'] \n", " meta.conf prepare-server ['server'] \n", "\n", " app: prepare-client-site-1 \n", " \n", " config_fed_client.conf Client_Name site-1 \n", " config_fed_client.conf Experiment Name Task09_Spleen \n", " config_fed_client.conf FileExtension .nii.gz \n", " config_fed_client.conf MLFlow_Token \n", " config_fed_client.conf Seed 12345 \n", " config_fed_client.conf Spleen 1 \n", " config_fed_client.conf Tracking_URI https://monai-demo.maia.cloud.cbh.k \n", " config_fed_client.conf background 0 \n", " config_fed_client.conf components [] \n", " config_fed_client.conf ct .nii.gz \n", " config_fed_client.conf data_dir /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf dataset_format decathlon \n", " config_fed_client.conf label .nii.gz \n", " config_fed_client.conf n_folds 5 \n", " config_fed_client.conf nnunet_root_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf patient_id_in_file_identifier True \n", " config_fed_client.conf task_ID 109 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: prepare-client-site-2 \n", " \n", " config_fed_client.conf Client_Name site-2 \n", " config_fed_client.conf Experiment Name Task09_Spleen \n", " config_fed_client.conf FileExtension .nii.gz \n", " config_fed_client.conf MLFlow_Token \n", " config_fed_client.conf Seed 12345 \n", " config_fed_client.conf Spleen 1 \n", " config_fed_client.conf Tracking_URI https://monai-demo.maia.cloud.cbh.k \n", " config_fed_client.conf background 0 \n", " config_fed_client.conf components [] \n", " config_fed_client.conf ct _CT.nii.gz \n", " config_fed_client.conf data_dir /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf dataset_format subfolders \n", " config_fed_client.conf label _label.nii.gz \n", " config_fed_client.conf n_folds 5 \n", " config_fed_client.conf nnunet_root_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf patient_id_in_file_identifier True \n", " config_fed_client.conf task_ID 109 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: prepare-server \n", " \n", " config_fed_server.conf heart_beat_timeout 600 \n", " config_fed_server.conf min_responses_required 0 \n", " config_fed_server.conf timeout 6000 \n", " config_fed_server.conf wait_time_after_min_received 10 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", "\n", "The following are the variables you can change in the template\n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " \n", " job folder: /home/maia-user/shared/Jobs/Spleen/jobs/plan_and_preprocess \n", " \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " file_name var_name value component \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " meta.conf mandatory_clients ['site-1', 'site-2'] \n", " meta.conf min_clients 1 \n", " meta.conf plan_and_preprocess-client-sit ['site-1'] \n", " meta.conf plan_and_preprocess-client-sit ['site-2'] \n", " meta.conf plan_and_preprocess-server ['server'] \n", "\n", " app: plan_and_preprocess-client-site-2 \n", " \n", " config_fed_client.conf Client_Name site-2 \n", " config_fed_client.conf Experiment Name Task09_Spleen \n", " config_fed_client.conf MLFlow_Token \n", " config_fed_client.conf Tracking_URI https://monai-demo.maia.cloud.cbh.k \n", " config_fed_client.conf components [] \n", " config_fed_client.conf nnunet_root_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf task_ID 109 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: plan_and_preprocess-client-site-1 \n", " \n", " config_fed_client.conf Client_Name site-1 \n", " config_fed_client.conf Experiment Name Task09_Spleen \n", " config_fed_client.conf MLFlow_Token \n", " config_fed_client.conf Tracking_URI https://monai-demo.maia.cloud.cbh.k \n", " config_fed_client.conf components [] \n", " config_fed_client.conf nnunet_root_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf task_ID 109 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: plan_and_preprocess-server \n", " \n", " config_fed_server.conf heart_beat_timeout 600 \n", " config_fed_server.conf min_responses_required 0 \n", " config_fed_server.conf timeout 6000 \n", " config_fed_server.conf wait_time_after_min_received 10 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", "\n", "The following are the variables you can change in the template\n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " \n", " job folder: /home/maia-user/shared/Jobs/Spleen/jobs/preprocess \n", " \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " file_name var_name value component \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " meta.conf mandatory_clients ['site-1', 'site-2'] \n", " meta.conf min_clients 1 \n", " meta.conf preprocess-client-site-1 ['site-1'] \n", " meta.conf preprocess-client-site-2 ['site-2'] \n", " meta.conf preprocess-server ['server'] \n", "\n", " app: preprocess-client-site-2 \n", " \n", " config_fed_client.conf Client_Name site-2 \n", " config_fed_client.conf Experiment Name Task09_Spleen \n", " config_fed_client.conf MLFlow_Token \n", " config_fed_client.conf Tracking_URI https://monai-demo.maia.cloud.cbh.k \n", " config_fed_client.conf components [] \n", " config_fed_client.conf nnunet_root_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf task_ID 109 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: preprocess-client-site-1 \n", " \n", " config_fed_client.conf Client_Name site-1 \n", " config_fed_client.conf Experiment Name Task09_Spleen \n", " config_fed_client.conf MLFlow_Token \n", " config_fed_client.conf Tracking_URI https://monai-demo.maia.cloud.cbh.k \n", " config_fed_client.conf components [] \n", " config_fed_client.conf nnunet_root_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf task_ID 109 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: preprocess-server \n", " \n", " config_fed_server.conf heart_beat_timeout 600 \n", " config_fed_server.conf min_responses_required 0 \n", " config_fed_server.conf timeout 6000 \n", " config_fed_server.conf wait_time_after_min_received 10 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", "\n", "The following are the variables you can change in the template\n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " \n", " job folder: /home/maia-user/shared/Jobs/Spleen/jobs/check_client_packages \n", " \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " file_name var_name value component \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " meta.conf check_client_packages-client-s ['site-1'] \n", " meta.conf check_client_packages-client-s ['site-2'] \n", " meta.conf check_client_packages-server ['server'] \n", " meta.conf mandatory_clients ['site-1', 'site-2'] \n", " meta.conf min_clients 1 \n", "\n", " app: check_client_packages-client-site-1 \n", " \n", " config_fed_client.conf components [] \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: check_client_packages-server \n", " \n", " config_fed_server.conf heart_beat_timeout 600 \n", " config_fed_server.conf min_responses_required 0 \n", " config_fed_server.conf timeout 6000 \n", " config_fed_server.conf wait_time_after_min_received 10 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: check_client_packages-client-site-2 \n", " \n", " config_fed_client.conf components [] \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", "\n", "The following are the variables you can change in the template\n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " \n", " job folder: /home/maia-user/shared/Jobs/Spleen/jobs/prepare_bundle \n", " \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " file_name var_name value component \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " meta.conf mandatory_clients ['site-1', 'site-2'] \n", " meta.conf min_clients 1 \n", " meta.conf prepare_bundle-client-site-1 ['site-1'] \n", " meta.conf prepare_bundle-client-site-2 ['site-2'] \n", " meta.conf prepare_bundle-server ['server'] \n", "\n", " app: prepare_bundle-server \n", " \n", " config_fed_server.conf heart_beat_timeout 600 \n", " config_fed_server.conf min_responses_required 0 \n", " config_fed_server.conf timeout 600000 \n", " config_fed_server.conf wait_time_after_min_received 10 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: prepare_bundle-client-site-2 \n", " \n", " config_fed_client.conf bundle_file Spleen_Bundle.zip \n", " config_fed_client.conf bundle_root /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf components [] \n", " config_fed_client.conf mlflow_experiment_name Task09_Spleen-FL \n", " config_fed_client.conf mlflow_run_name site-2 \n", " config_fed_client.conf nnunet_model_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf pymaia_config_file /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf task_id 109 \n", " config_fed_client.conf tracking_uri https://monai-demo.maia.cloud.cbh.k \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: prepare_bundle-client-site-1 \n", " \n", " config_fed_client.conf bundle_file Spleen_Bundle.zip \n", " config_fed_client.conf bundle_root /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf components [] \n", " config_fed_client.conf mlflow_experiment_name Task09_Spleen-FL \n", " config_fed_client.conf mlflow_run_name site-1 \n", " config_fed_client.conf nnunet_model_folder /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf pymaia_config_file /home/maia-user/Tutorials/NVFlare_P \n", " config_fed_client.conf task_id 109 \n", " config_fed_client.conf tracking_uri https://monai-demo.maia.cloud.cbh.k \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", "\n", "The following are the variables you can change in the template\n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " \n", " job folder: /home/maia-user/shared/Jobs/Spleen/jobs/fl_bundle \n", " \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " file_name var_name value component \n", "---------------------------------------------------------------------------------------------------------------------------------------\n", " meta.conf fl_bundle-server ['server'] \n", " meta.conf fl_bundle-site-1 ['site-1'] \n", " meta.conf fl_bundle-site-2 ['site-2'] \n", " meta.conf mandatory_clients ['site-1', 'site-2'] \n", " meta.conf min_clients 2 \n", "\n", " app: fl_bundle-site-1 \n", " \n", " config_fed_client.conf best_model_filepath models/model.pt MonaiAlgo \n", " config_fed_client.conf bundle_root /home/maia-user/Tutorials/NVFlare_P MonaiAlgo \n", " config_fed_client.conf config_evaluate_filename default MonaiAlgo \n", " config_fed_client.conf config_train_filename configs/train.yaml MonaiAlgo \n", " config_fed_client.conf disable_ckpt_loading True MonaiAlgo \n", " config_fed_client.conf eval_workflow_name train MonaiAlgo \n", " config_fed_client.conf final_model_filepath models/model_final.pt MonaiAlgo \n", " config_fed_client.conf key_metric Val_Dice \n", " config_fed_client.conf local_epochs 10 MonaiAlgo \n", " config_fed_client.conf save_dict_key network_weights MonaiAlgo \n", " config_fed_client.conf send_weight_diff True MonaiAlgo \n", " config_fed_client.conf submit_model_task submit_model \n", " config_fed_client.conf train_task train \n", " config_fed_client.conf validate_task validate \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: fl_bundle-site-2 \n", " \n", " config_fed_client.conf best_model_filepath models/model.pt MonaiAlgo \n", " config_fed_client.conf bundle_root /home/maia-user/Tutorials/NVFlare_P MonaiAlgo \n", " config_fed_client.conf config_evaluate_filename default MonaiAlgo \n", " config_fed_client.conf config_train_filename configs/train.yaml MonaiAlgo \n", " config_fed_client.conf disable_ckpt_loading True MonaiAlgo \n", " config_fed_client.conf eval_workflow_name train MonaiAlgo \n", " config_fed_client.conf final_model_filepath models/model_final.pt MonaiAlgo \n", " config_fed_client.conf key_metric Val_Dice \n", " config_fed_client.conf local_epochs 10 MonaiAlgo \n", " config_fed_client.conf save_dict_key network_weights MonaiAlgo \n", " config_fed_client.conf send_weight_diff True MonaiAlgo \n", " config_fed_client.conf submit_model_task submit_model \n", " config_fed_client.conf train_task train \n", " config_fed_client.conf validate_task validate \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", " app: fl_bundle-server \n", " \n", " config_fed_server.conf best_global_model_file_name best_FL_global_model.pt MonaiBundlePersistor \n", " config_fed_server.conf bundle_root /home/maia-user/shared/src/Spleen_B MonaiBundlePersistor \n", " config_fed_server.conf cleanup_models True CrossSiteModelEval \n", " config_fed_server.conf config_train_filename configs/train.yaml MonaiBundlePersistor \n", " config_fed_server.conf expected_data_kind WEIGHT_DIFF InTimeAccumulateWeightedAggregator \n", " config_fed_server.conf global_model_file_name FL_global_model.pt MonaiBundlePersistor \n", " config_fed_server.conf min_clients {min_clients} \n", " config_fed_server.conf network_def_key network_def MonaiBundlePersistor \n", " config_fed_server.conf num_rounds {num_rounds} \n", " config_fed_server.conf start_round 0 \n", " config_fed_server.conf submit_model_timeout 600 CrossSiteModelEval \n", " config_fed_server.conf train_timeout 0 \n", " config_fed_server.conf validation_timeout 6000 CrossSiteModelEval \n", " config_fed_server.conf wait_time_after_min_received 10 \n", "\n", " nnUNetPlans-checkpoint.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans-checkpoint.json affine True \n", " nnUNetPlans-checkpoint.json batch_dice True \n", " nnUNetPlans-checkpoint.json batch_size 2 \n", " nnUNetPlans-checkpoint.json conv_bias True \n", " nnUNetPlans-checkpoint.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans-checkpoint.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans-checkpoint.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans-checkpoint.json dropout_op None \n", " nnUNetPlans-checkpoint.json dropout_op_kwargs None \n", " nnUNetPlans-checkpoint.json eps 1e-05 \n", " nnUNetPlans-checkpoint.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans-checkpoint.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans-checkpoint.json force_separate_z None \n", " nnUNetPlans-checkpoint.json image_reader_writer SimpleITKIO \n", " nnUNetPlans-checkpoint.json inherits_from 3d_fullres \n", " nnUNetPlans-checkpoint.json inplace True \n", " nnUNetPlans-checkpoint.json is_seg False \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][0] 1 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[0][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[1][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[2][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[3][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[4][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[5][2] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[6][1] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][0] 3 \n", " nnUNetPlans-checkpoint.json kernel_sizes[7][1] 3 \n", " nnUNetPlans-checkpoint.json label_manager LabelManager \n", " nnUNetPlans-checkpoint.json max 1038.0 \n", " nnUNetPlans-checkpoint.json mean 94.82794189453125 \n", " nnUNetPlans-checkpoint.json median 99.0 \n", " nnUNetPlans-checkpoint.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans-checkpoint.json min -620.0 \n", " nnUNetPlans-checkpoint.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans-checkpoint.json n_stages 6 \n", " nnUNetPlans-checkpoint.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans-checkpoint.json next_stage 3d_cascade_fullres \n", " nnUNetPlans-checkpoint.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans-checkpoint.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans-checkpoint.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans-checkpoint.json order 1 \n", " nnUNetPlans-checkpoint.json order_z 0 \n", " nnUNetPlans-checkpoint.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans-checkpoint.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json patch_size [48, 224, 192] \n", " nnUNetPlans-checkpoint.json percentile_00_5 -40.0 \n", " nnUNetPlans-checkpoint.json percentile_99_5 176.0 \n", " nnUNetPlans-checkpoint.json plans_name nnUNetPlans \n", " nnUNetPlans-checkpoint.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans-checkpoint.json previous_stage 3d_lowres \n", " nnUNetPlans-checkpoint.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans-checkpoint.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans-checkpoint.json std 40.52839660644531 \n", " nnUNetPlans-checkpoint.json strides[0][0] 1 \n", " nnUNetPlans-checkpoint.json strides[0][1] 1 \n", " nnUNetPlans-checkpoint.json strides[0][2] 1 \n", " nnUNetPlans-checkpoint.json strides[1][0] 1 \n", " nnUNetPlans-checkpoint.json strides[1][1] 2 \n", " nnUNetPlans-checkpoint.json strides[1][2] 2 \n", " nnUNetPlans-checkpoint.json strides[2][0] 2 \n", " nnUNetPlans-checkpoint.json strides[2][1] 2 \n", " nnUNetPlans-checkpoint.json strides[2][2] 2 \n", " nnUNetPlans-checkpoint.json strides[3][0] 2 \n", " nnUNetPlans-checkpoint.json strides[3][1] 2 \n", " nnUNetPlans-checkpoint.json strides[3][2] 2 \n", " nnUNetPlans-checkpoint.json strides[4][0] 2 \n", " nnUNetPlans-checkpoint.json strides[4][1] 2 \n", " nnUNetPlans-checkpoint.json strides[4][2] 2 \n", " nnUNetPlans-checkpoint.json strides[5][0] 1 \n", " nnUNetPlans-checkpoint.json strides[5][1] 2 \n", " nnUNetPlans-checkpoint.json strides[5][2] 2 \n", " nnUNetPlans-checkpoint.json strides[6][0] 2 \n", " nnUNetPlans-checkpoint.json strides[6][1] 2 \n", " nnUNetPlans-checkpoint.json strides[7][0] 2 \n", " nnUNetPlans-checkpoint.json strides[7][1] 2 \n", " nnUNetPlans-checkpoint.json transpose_backward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json transpose_forward [0, 1, 2] \n", " nnUNetPlans-checkpoint.json use_mask_for_norm [False] \n", "\n", " nnUNetPlans.json _kw_requires_import ['conv_op', 'norm_op', 'dropout_op' \n", " nnUNetPlans.json affine True \n", " nnUNetPlans.json batch_dice True \n", " nnUNetPlans.json batch_size 2 \n", " nnUNetPlans.json conv_bias True \n", " nnUNetPlans.json conv_op torch.nn.modules.conv.Conv3d \n", " nnUNetPlans.json data_identifier nnUNetPlans_3d_fullres \n", " nnUNetPlans.json dataset_name Dataset109_Task09_Spleen \n", " nnUNetPlans.json dropout_op None \n", " nnUNetPlans.json dropout_op_kwargs None \n", " nnUNetPlans.json eps 1e-05 \n", " nnUNetPlans.json experiment_planner_used ExperimentPlanner \n", " nnUNetPlans.json features_per_stage [32, 64, 128, 256, 320, 320] \n", " nnUNetPlans.json force_separate_z None \n", " nnUNetPlans.json image_reader_writer SimpleITKIO \n", " nnUNetPlans.json inherits_from 3d_fullres \n", " nnUNetPlans.json inplace True \n", " nnUNetPlans.json is_seg False \n", " nnUNetPlans.json kernel_sizes[0][0] 1 \n", " nnUNetPlans.json kernel_sizes[0][1] 3 \n", " nnUNetPlans.json kernel_sizes[0][2] 3 \n", " nnUNetPlans.json kernel_sizes[1][0] 3 \n", " nnUNetPlans.json kernel_sizes[1][1] 3 \n", " nnUNetPlans.json kernel_sizes[1][2] 3 \n", " nnUNetPlans.json kernel_sizes[2][0] 3 \n", " nnUNetPlans.json kernel_sizes[2][1] 3 \n", " nnUNetPlans.json kernel_sizes[2][2] 3 \n", " nnUNetPlans.json kernel_sizes[3][0] 3 \n", " nnUNetPlans.json kernel_sizes[3][1] 3 \n", " nnUNetPlans.json kernel_sizes[3][2] 3 \n", " nnUNetPlans.json kernel_sizes[4][0] 3 \n", " nnUNetPlans.json kernel_sizes[4][1] 3 \n", " nnUNetPlans.json kernel_sizes[4][2] 3 \n", " nnUNetPlans.json kernel_sizes[5][0] 3 \n", " nnUNetPlans.json kernel_sizes[5][1] 3 \n", " nnUNetPlans.json kernel_sizes[5][2] 3 \n", " nnUNetPlans.json kernel_sizes[6][0] 3 \n", " nnUNetPlans.json kernel_sizes[6][1] 3 \n", " nnUNetPlans.json kernel_sizes[7][0] 3 \n", " nnUNetPlans.json kernel_sizes[7][1] 3 \n", " nnUNetPlans.json label_manager LabelManager \n", " nnUNetPlans.json max 1038.0 \n", " nnUNetPlans.json mean 94.82794189453125 \n", " nnUNetPlans.json median 99.0 \n", " nnUNetPlans.json median_image_size_in_voxels [112.0, 512.0, 512.0] \n", " nnUNetPlans.json min -620.0 \n", " nnUNetPlans.json n_conv_per_stage [2, 2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_conv_per_stage_decoder [2, 2, 2, 2, 2] \n", " nnUNetPlans.json n_stages 6 \n", " nnUNetPlans.json network_class_name dynamic_network_architectures.archi \n", " nnUNetPlans.json next_stage 3d_cascade_fullres \n", " nnUNetPlans.json nonlin torch.nn.LeakyReLU \n", " nnUNetPlans.json norm_op torch.nn.modules.instancenorm.Insta \n", " nnUNetPlans.json normalization_schemes ['CTNormalization'] \n", " nnUNetPlans.json order 1 \n", " nnUNetPlans.json order_z 0 \n", " nnUNetPlans.json original_median_shape_after_tr [83, 512, 512] \n", " nnUNetPlans.json original_median_spacing_after_ [5.0, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json patch_size [48, 224, 192] \n", " nnUNetPlans.json percentile_00_5 -40.0 \n", " nnUNetPlans.json percentile_99_5 176.0 \n", " nnUNetPlans.json plans_name nnUNetPlans \n", " nnUNetPlans.json preprocessor_name DefaultPreprocessor \n", " nnUNetPlans.json previous_stage 3d_lowres \n", " nnUNetPlans.json resampling_fn_data resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_probabilities resample_data_or_seg_to_shape \n", " nnUNetPlans.json resampling_fn_seg resample_data_or_seg_to_shape \n", " nnUNetPlans.json spacing [2.5, 0.7929689884185791, 0.7929689 \n", " nnUNetPlans.json std 40.52839660644531 \n", " nnUNetPlans.json strides[0][0] 1 \n", " nnUNetPlans.json strides[0][1] 1 \n", " nnUNetPlans.json strides[0][2] 1 \n", " nnUNetPlans.json strides[1][0] 1 \n", " nnUNetPlans.json strides[1][1] 2 \n", " nnUNetPlans.json strides[1][2] 2 \n", " nnUNetPlans.json strides[2][0] 2 \n", " nnUNetPlans.json strides[2][1] 2 \n", " nnUNetPlans.json strides[2][2] 2 \n", " nnUNetPlans.json strides[3][0] 2 \n", " nnUNetPlans.json strides[3][1] 2 \n", " nnUNetPlans.json strides[3][2] 2 \n", " nnUNetPlans.json strides[4][0] 2 \n", " nnUNetPlans.json strides[4][1] 2 \n", " nnUNetPlans.json strides[4][2] 2 \n", " nnUNetPlans.json strides[5][0] 1 \n", " nnUNetPlans.json strides[5][1] 2 \n", " nnUNetPlans.json strides[5][2] 2 \n", " nnUNetPlans.json strides[6][0] 2 \n", " nnUNetPlans.json strides[6][1] 2 \n", " nnUNetPlans.json strides[7][0] 2 \n", " nnUNetPlans.json strides[7][1] 2 \n", " nnUNetPlans.json transpose_backward [0, 1, 2] \n", " nnUNetPlans.json transpose_forward [0, 1, 2] \n", " nnUNetPlans.json use_mask_for_norm [False] \n", "\n", "---------------------------------------------------------------------------------------------------------------------------------------\n" ] } ], "source": [ "from generate_jobs_config import generate_config\n", "\n", "generate_config(\n", " [str(Path(ROOT_FOLDER).joinpath(\"Clients\",client+\".yaml\")) for client in clients],\n", " str(Path(ROOT_FOLDER).joinpath(\"Experiments\",experiment+\".yaml\")),\n", " str(Path(ROOT_FOLDER).joinpath(\"src\",experiment)),\n", " str(Path(ROOT_FOLDER).joinpath(\"Jobs\",experiment)),\n", " nvflare_exec=\"/home/maia-user/.conda/envs/NVFlare/bin/nvflare\",\n", " pymaia_nvflare_exec=\"pymaia_nvflare\"\n", " \n", ")" ] }, { "cell_type": "code", "execution_count": 70, "id": "77f57247-c564-4703-bcc7-4e6f12e82e22", "metadata": {}, "outputs": [], "source": [ "JOB_DIR=str(Path(ROOT_FOLDER).joinpath(\"Jobs\",experiment))" ] }, { "cell_type": "code", "execution_count": 60, "id": "378d1865-5d37-41df-a5c0-ab6ebb88511a", "metadata": {}, "outputs": [], "source": [ "client_id = \"site-2\"" ] }, { "cell_type": "markdown", "id": "d0d6a222a5a6fc31", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "### Create and Modify Job Configuration\n", "\n", "Jobs configuration files are automatically created, but can be recreated or modified using the following NVFlare command:\n", "\n", "```bash\n", "nvflare job create -j / -w //\n", "```\n", "\n", "```bash\n", "nvflare job create -j / -w // var_name=new_value\n", "```" ] }, { "cell_type": "markdown", "id": "9545a4e6-3c6c-4e2c-99f8-3b5ce4b0239c", "metadata": {}, "source": [ "## 0. Check Python Packages" ] }, { "cell_type": "code", "execution_count": 25, "id": "9d504722-79a7-4968-9f3e-d837c91dd396", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"check_client_packages\")))" ] }, { "cell_type": "code", "execution_count": 26, "id": "3e5da18f-0440-4266-8fff-7ac855fe2f6c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sess.monitor_job(job_id)" ] }, { "cell_type": "code", "execution_count": 46, "id": "2350fa0a-5c79-4068-a6c5-1b47d7412347", "metadata": {}, "outputs": [], "source": [ "client_id = \"site-2\"" ] }, { "cell_type": "code", "execution_count": 31, "id": "50524574-e08d-46e2-9de1-01e4ba45ec07", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Mon Dec 16 13:23:30 2024 \n", "+---------------------------------------------------------------------------------------+\n", "| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |\n", "|-----------------------------------------+----------------------+----------------------+\n", "| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n", "| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n", "| | | MIG M. |\n", "|=========================================+======================+======================|\n", "| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |\n", "| 30% 35C P8 26W / 350W | 261MiB / 24576MiB | 0% Default |\n", "| | | N/A |\n", "+-----------------------------------------+----------------------+----------------------+\n", " \n", "+---------------------------------------------------------------------------------------+\n", "| Processes: |\n", "| GPU GI CI PID Type Process name GPU Memory |\n", "| ID ID Usage |\n", "|=======================================================================================|\n", "+---------------------------------------------------------------------------------------+\n" ] } ], "source": [ "#print(sess.api.do_command(f\"cat server log.txt\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} nvidia-smi.log\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} init_logfile_out.log\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} init_logfile_err.log\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": 32, "id": "25774b07-99d3-4229-92fd-9bff414e21bf", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": 33, "id": "3d4aa553-a00e-45b3-ae0c-08054ef59bfa", "metadata": {}, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"package_report\",\"package_report.json\"),\"r\") as f:\n", " package_report = json.load(f)" ] }, { "cell_type": "code", "execution_count": 34, "id": "79f8448b-aeab-4e7c-b21c-ce4e0e80bc4a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'site-2': {'nvflare': 'nvflare 2.5.2 is installed.',\n", " 'pymaia-learn': 'pymaia-learn 1.2.1 is installed.',\n", " 'torch': 'torch 2.5.1 is installed.',\n", " 'monai': 'monai 1.4.0 is installed.',\n", " 'numpy': 'numpy 1.24.0 is installed.',\n", " 'nnunetv2': 'nnunetv2 2.5.1 is installed.',\n", " 'MONAI': '1.4.0',\n", " 'Numpy': '1.24.0',\n", " 'Pytorch': '',\n", " 'Pytorch Ignite': '0.4.11',\n", " 'ITK': '5.4.0',\n", " 'Nibabel': '5.3.2',\n", " 'scikit-image': '0.25.0',\n", " 'scipy': '1.14.1',\n", " 'Pillow': '11.0.0',\n", " 'Tensorboard': '2.18.0',\n", " 'gdown': '5.2.0',\n", " 'TorchVision': '0.20.1',\n", " 'tqdm': '4.67.1',\n", " 'lmdb': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'psutil': '6.1.0',\n", " 'pandas': '2.2.3',\n", " 'einops': '0.8.0',\n", " 'transformers': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'mlflow': '2.19.0',\n", " 'pynrrd': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'clearml': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'Num GPUs': '1',\n", " 'cuDNN enabled': 'True',\n", " 'GPU 0 Name': 'NVIDIA GeForce RTX 3090',\n", " 'GPU 0 Total memory GB ': '23.7',\n", " 'CPU_Cores': 20,\n", " 'Total Memory': 62.38420104980469,\n", " 'Memory Used': 24.1},\n", " 'site-1': {'nvflare': 'nvflare 2.5.2 is installed.',\n", " 'pymaia-learn': 'pymaia-learn 1.2.1 is installed.',\n", " 'torch': 'torch 2.5.1 is installed.',\n", " 'monai': 'monai 1.4.0 is installed.',\n", " 'numpy': 'numpy 1.24.0 is installed.',\n", " 'nnunetv2': 'nnunetv2 2.5.1 is installed.',\n", " 'MONAI': '1.4.0',\n", " 'Numpy': '1.24.0',\n", " 'Pytorch': '',\n", " 'Pytorch Ignite': '0.4.11',\n", " 'ITK': '5.4.0',\n", " 'Nibabel': '5.3.2',\n", " 'scikit-image': '0.25.0',\n", " 'scipy': '1.14.1',\n", " 'Pillow': '11.0.0',\n", " 'Tensorboard': '2.18.0',\n", " 'gdown': '5.2.0',\n", " 'TorchVision': '0.20.1',\n", " 'tqdm': '4.67.1',\n", " 'lmdb': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'psutil': '6.1.0',\n", " 'pandas': '2.2.3',\n", " 'einops': '0.8.0',\n", " 'transformers': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'mlflow': '2.19.0',\n", " 'pynrrd': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'clearml': 'NOT INSTALLED or UNKNOWN VERSION.',\n", " 'Num GPUs': '1',\n", " 'cuDNN enabled': 'True',\n", " 'GPU 0 Name': 'NVIDIA GeForce RTX 3090',\n", " 'GPU 0 Total memory GB ': '23.7',\n", " 'CPU_Cores': 20,\n", " 'Total Memory': 62.38420104980469,\n", " 'Memory Used': 24.1}}" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "package_report" ] }, { "cell_type": "code", "execution_count": null, "id": "148b7f21-ae9a-44ac-bd9c-1bf052ef58ac", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "6e28d47fb25446e1", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## 1. Prepare Dataset\n", "\n", "In this step, the dataset in the different sites will be prepared for training, harmonizing the data and creating the necessary files for training according to the nnUNet framework. The following python packages will be installed during the execution of this step:\n", "\n", "```text\n", "pymaia-learn==1.1\n", "nnunetv2==2.5.1\n", "fire\n", "pytorch-ignite==0.4.11\n", "monai[nibabel, skimage, scipy, pillow, tensorboard, gdown, ignite, torchvision, itk, tqdm, pandas, mlflow, matplotlib, pydicom]\n", "``` " ] }, { "cell_type": "markdown", "id": "b42596bfcf2aea4c", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "### Submit the Job" ] }, { "cell_type": "code", "execution_count": 35, "id": "5775e1b39443362d", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"prepare\")))" ] }, { "cell_type": "code", "execution_count": 36, "id": "e33d4bcf9051775b", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sess.monitor_job(job_id)" ] }, { "cell_type": "markdown", "id": "98df7413cdbf4afd", "metadata": {}, "source": [ "To monitor the job, or print the logs from either the server or the client side, you can use the following commands:" ] }, { "cell_type": "code", "execution_count": null, "id": "e71f849b-e09f-4cda-bf01-ee36db5cd4ef", "metadata": { "scrolled": true }, "outputs": [], "source": [ "#print(sess.api.do_command(f\"cat server log.txt\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} init_logfile_out.log\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} init_logfile_err.log\")['data'][0]['data'])\n", "\n", "#print(sess.api.do_command(f\"cat {client_id} logfile_datadir.log\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} logfile_storage.log\")['data'][0]['data'])\n", "\n", "#print(sess.api.do_command(f\"cat {client_id} logfile_prepare_out.log\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} logfile_prepare_err.log\")['data'][0]['data'])" ] }, { "cell_type": "markdown", "id": "a900685dc5fc0604", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "### Download the Job Results\n", "When the job is completed, you can download the results, which, for the Prepare Dataset job, will contain the dataset.json file, containing the information about the dataset. In detail, for each client, the dataset.json file will list the dataset files and wheter all the files are valid or not." ] }, { "cell_type": "code", "execution_count": 37, "id": "830fded9516c0d6a", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": 38, "id": "f77cb85fa5dc8c82", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"prepare_dataset\",\"data_prepare.json\"),\"r\") as f:\n", " dataset_dict = json.load(f)" ] }, { "cell_type": "markdown", "id": "3f4dffceb081fef1", "metadata": {}, "source": [ "To display the PyMAIA Experiment Configuration, run the following command:" ] }, { "cell_type": "code", "execution_count": 42, "id": "13d2fec7-b9c2-4783-a305-d31904745d41", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "['spleen_49_0000.nii.gz',\n", " 'spleen_8_0000.nii.gz',\n", " 'spleen_16_0000.nii.gz',\n", " 'spleen_31_0000.nii.gz',\n", " 'spleen_52_0000.nii.gz',\n", " 'spleen_56_0000.nii.gz',\n", " 'spleen_40_0000.nii.gz',\n", " 'spleen_61_0000.nii.gz',\n", " 'spleen_62_0000.nii.gz',\n", " 'spleen_22_0000.nii.gz',\n", " 'spleen_9_0000.nii.gz',\n", " 'spleen_45_0000.nii.gz',\n", " 'spleen_59_0000.nii.gz',\n", " 'spleen_53_0000.nii.gz',\n", " 'spleen_17_0000.nii.gz',\n", " 'spleen_41_0000.nii.gz',\n", " 'spleen_13_0000.nii.gz',\n", " 'spleen_60_0000.nii.gz',\n", " 'spleen_3_0000.nii.gz',\n", " 'spleen_46_0000.nii.gz',\n", " 'spleen_28_0000.nii.gz',\n", " 'spleen_21_0000.nii.gz',\n", " 'spleen_14_0000.nii.gz',\n", " 'spleen_10_0000.nii.gz',\n", " 'spleen_47_0000.nii.gz',\n", " 'spleen_26_0000.nii.gz',\n", " 'spleen_19_0000.nii.gz',\n", " 'spleen_32_0000.nii.gz',\n", " 'spleen_33_0000.nii.gz',\n", " 'spleen_29_0000.nii.gz',\n", " 'spleen_6_0000.nii.gz']" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset_dict[client_id]['nnUNet_Dataset']['imagesTr']#['pymaia_config']\n", "#dataset_dict[client_id]['pymaia_config']" ] }, { "cell_type": "code", "execution_count": 43, "id": "df9f7c87-4609-43e7-b2c1-3b206ed185d8", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'imagesTr': ['spleen_49_0000.nii.gz',\n", " 'spleen_8_0000.nii.gz',\n", " 'spleen_16_0000.nii.gz',\n", " 'spleen_31_0000.nii.gz',\n", " 'spleen_52_0000.nii.gz',\n", " 'spleen_56_0000.nii.gz',\n", " 'spleen_40_0000.nii.gz',\n", " 'spleen_61_0000.nii.gz',\n", " 'spleen_62_0000.nii.gz',\n", " 'spleen_22_0000.nii.gz',\n", " 'spleen_9_0000.nii.gz',\n", " 'spleen_45_0000.nii.gz',\n", " 'spleen_59_0000.nii.gz',\n", " 'spleen_53_0000.nii.gz',\n", " 'spleen_17_0000.nii.gz',\n", " 'spleen_41_0000.nii.gz',\n", " 'spleen_13_0000.nii.gz',\n", " 'spleen_60_0000.nii.gz',\n", " 'spleen_3_0000.nii.gz',\n", " 'spleen_46_0000.nii.gz',\n", " 'spleen_28_0000.nii.gz',\n", " 'spleen_21_0000.nii.gz',\n", " 'spleen_14_0000.nii.gz',\n", " 'spleen_10_0000.nii.gz',\n", " 'spleen_47_0000.nii.gz',\n", " 'spleen_26_0000.nii.gz',\n", " 'spleen_19_0000.nii.gz',\n", " 'spleen_32_0000.nii.gz',\n", " 'spleen_33_0000.nii.gz',\n", " 'spleen_29_0000.nii.gz',\n", " 'spleen_6_0000.nii.gz'],\n", " 'labelsTr': ['spleen_9.nii.gz',\n", " 'spleen_31.nii.gz',\n", " 'spleen_28.nii.gz',\n", " 'spleen_59.nii.gz',\n", " 'spleen_10.nii.gz',\n", " 'spleen_47.nii.gz',\n", " 'spleen_41.nii.gz',\n", " 'spleen_45.nii.gz',\n", " 'spleen_21.nii.gz',\n", " 'spleen_26.nii.gz',\n", " 'spleen_13.nii.gz',\n", " 'spleen_33.nii.gz',\n", " 'spleen_6.nii.gz',\n", " 'spleen_8.nii.gz',\n", " 'spleen_53.nii.gz',\n", " 'spleen_52.nii.gz',\n", " 'spleen_22.nii.gz',\n", " 'spleen_16.nii.gz',\n", " 'spleen_49.nii.gz',\n", " 'spleen_17.nii.gz',\n", " 'spleen_29.nii.gz',\n", " 'spleen_56.nii.gz',\n", " 'spleen_62.nii.gz',\n", " 'spleen_14.nii.gz',\n", " 'spleen_46.nii.gz',\n", " 'spleen_19.nii.gz',\n", " 'spleen_3.nii.gz',\n", " 'spleen_61.nii.gz',\n", " 'spleen_40.nii.gz',\n", " 'spleen_32.nii.gz',\n", " 'spleen_60.nii.gz']}" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset_dict[client_id]['nnUNet_Dataset']" ] }, { "cell_type": "markdown", "id": "cb243c580fb4c54d", "metadata": {}, "source": [ "And to inspect the dataset, run the following command:" ] }, { "cell_type": "code", "execution_count": 47, "id": "8bc9091f580cdb56", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Error: /home/maia-user/Tutorials/NVFlare_POC/data/site-2/.ipynb_checkpoints/.ipynb_checkpoints_CT.nii.gz is not a valid file!\n", "Error: /home/maia-user/Tutorials/NVFlare_POC/data/site-2/.ipynb_checkpoints/.ipynb_checkpoints_label.nii.gz is not a valid file!\n" ] } ], "source": [ "verified = True\n", "\n", "for case in dataset_dict[client_id][\"train\"]:\n", " for key in case:\n", " if key.endswith(\"_is_file\") and not case[key]:\n", " file = case[key[:-len(\"_is_file\")]]\n", " print(f\"Error: {file} is not a valid file!\")\n", " verified = False\n", "if verified:\n", " print(f\"Dataset succesfully verified for client {client_id}\")" ] }, { "cell_type": "code", "execution_count": 48, "id": "de23b6d829e777c3", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "25\n" ] } ], "source": [ "print(len(dataset_dict[client_id][\"train\"]))" ] }, { "cell_type": "markdown", "id": "54e0f76dbaa30073", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## 2. Plan and Preprocess\n", "\n", "After the dataset has been prepared, the nnUNet experiment has to be planned and the data preprocessed. This step has to be done only on one site, as the nnUNet plans will be shared with the other sites.\n", "\n", "The steps to plan and preprocess the nnUNet experiment are the following:\n", "1. Run the `plan_and_preprocess` job on the chosen site.\n", "2. Extract the nnUNet plans from the job results.\n", "3. Share the nnUNet plans with the other sites.\n", "4. Run the `preprocess` job on the other sites." ] }, { "cell_type": "markdown", "id": "2aee7d58-d8f9-4bb7-99c6-9b4d4fd8641a", "metadata": {}, "source": [ "### Submit the Job" ] }, { "cell_type": "code", "execution_count": 55, "id": "c38972ad-b061-4963-8327-806d9ab378c7", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"plan_and_preprocess\")))" ] }, { "cell_type": "code", "execution_count": 56, "id": "901df284-f6ac-41cc-b580-cd32ed2b9381", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 56, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sess.monitor_job(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "649a99ae-271d-4ca9-85a3-0d64c59e5c55", "metadata": { "scrolled": true }, "outputs": [], "source": [ "#print(sess.api.do_command(f\"tail server log.txt\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile_plan_and_preprocess_out.log\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile_plan_and_preprocess_err.log\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": 57, "id": "302cfe9c-269e-4164-9e14-cb7a316a691e", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "markdown", "id": "eff74355-fc9d-446b-8352-08cf3e9a0fd1", "metadata": {}, "source": [ "### Inspect nnUNetPlans.json\n", "\n", "The nnUNet plans are stored in the `nnUNetPlans.json` file, which contains the configuration for the nnUNet experiment. The file can be found in the `workspace/nnUNet_preprocessing` folder of the job results." ] }, { "cell_type": "code", "execution_count": 58, "id": "54e3868d-1f0e-4649-8031-79732ad6ad95", "metadata": { "tags": [] }, "outputs": [], "source": [ "import json\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"nnUNet_preprocessing\",\"nnUNetPlans.json\"),\"r\") as f:\n", " nnunet_plans = json.load(f)" ] }, { "cell_type": "code", "execution_count": 59, "id": "2e8e246e-c7bf-4861-a6cd-b3f35156aa24", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"site-1\": {\n", " \"dataset_name\": \"Dataset109_Task09_Spleen\",\n", " \"plans_name\": \"nnUNetPlans\",\n", " \"original_median_spacing_after_transp\": [\n", " 5.0,\n", " 0.7929689884185791,\n", " 0.7929689884185791\n", " ],\n", " \"original_median_shape_after_transp\": [\n", " 83,\n", " 512,\n", " 512\n", " ],\n", " \"image_reader_writer\": \"SimpleITKIO\",\n", " \"transpose_forward\": [\n", " 0,\n", " 1,\n", " 2\n", " ],\n", " \"transpose_backward\": [\n", " 0,\n", " 1,\n", " 2\n", " ],\n", " \"configurations\": {\n", " \"2d\": {\n", " \"data_identifier\": \"nnUNetPlans_2d\",\n", " \"preprocessor_name\": \"DefaultPreprocessor\",\n", " \"batch_size\": 12,\n", " \"patch_size\": [\n", " 512,\n", " 512\n", " ],\n", " \"median_image_size_in_voxels\": [\n", " 512.0,\n", " 512.0\n", " ],\n", " \"spacing\": [\n", " 0.7929689884185791,\n", " 0.7929689884185791\n", " ],\n", " \"normalization_schemes\": [\n", " \"CTNormalization\"\n", " ],\n", " \"use_mask_for_norm\": [\n", " false\n", " ],\n", " \"resampling_fn_data\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_seg\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_data_kwargs\": {\n", " \"is_seg\": false,\n", " \"order\": 3,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"resampling_fn_seg_kwargs\": {\n", " \"is_seg\": true,\n", " \"order\": 1,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"resampling_fn_probabilities\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_probabilities_kwargs\": {\n", " \"is_seg\": false,\n", " \"order\": 1,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"architecture\": {\n", " \"network_class_name\": \"dynamic_network_architectures.architectures.unet.PlainConvUNet\",\n", " \"arch_kwargs\": {\n", " \"n_stages\": 8,\n", " \"features_per_stage\": [\n", " 32,\n", " 64,\n", " 128,\n", " 256,\n", " 512,\n", " 512,\n", " 512,\n", " 512\n", " ],\n", " \"conv_op\": \"torch.nn.modules.conv.Conv2d\",\n", " \"kernel_sizes\": [\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3\n", " ]\n", " ],\n", " \"strides\": [\n", " [\n", " 1,\n", " 1\n", " ],\n", " [\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2\n", " ]\n", " ],\n", " \"n_conv_per_stage\": [\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " \"n_conv_per_stage_decoder\": [\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " \"conv_bias\": true,\n", " \"norm_op\": \"torch.nn.modules.instancenorm.InstanceNorm2d\",\n", " \"norm_op_kwargs\": {\n", " \"eps\": 1e-05,\n", " \"affine\": true\n", " },\n", " \"dropout_op\": null,\n", " \"dropout_op_kwargs\": null,\n", " \"nonlin\": \"torch.nn.LeakyReLU\",\n", " \"nonlin_kwargs\": {\n", " \"inplace\": true\n", " }\n", " },\n", " \"_kw_requires_import\": [\n", " \"conv_op\",\n", " \"norm_op\",\n", " \"dropout_op\",\n", " \"nonlin\"\n", " ]\n", " },\n", " \"batch_dice\": true\n", " },\n", " \"3d_lowres\": {\n", " \"data_identifier\": \"nnUNetPlans_3d_lowres\",\n", " \"preprocessor_name\": \"DefaultPreprocessor\",\n", " \"batch_size\": 2,\n", " \"patch_size\": [\n", " 64,\n", " 192,\n", " 192\n", " ],\n", " \"median_image_size_in_voxels\": [\n", " 102,\n", " 292,\n", " 292\n", " ],\n", " \"spacing\": [\n", " 2.7318175000000005,\n", " 1.390475921094405,\n", " 1.390475921094405\n", " ],\n", " \"normalization_schemes\": [\n", " \"CTNormalization\"\n", " ],\n", " \"use_mask_for_norm\": [\n", " false\n", " ],\n", " \"resampling_fn_data\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_seg\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_data_kwargs\": {\n", " \"is_seg\": false,\n", " \"order\": 3,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"resampling_fn_seg_kwargs\": {\n", " \"is_seg\": true,\n", " \"order\": 1,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"resampling_fn_probabilities\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_probabilities_kwargs\": {\n", " \"is_seg\": false,\n", " \"order\": 1,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"architecture\": {\n", " \"network_class_name\": \"dynamic_network_architectures.architectures.unet.PlainConvUNet\",\n", " \"arch_kwargs\": {\n", " \"n_stages\": 6,\n", " \"features_per_stage\": [\n", " 32,\n", " 64,\n", " 128,\n", " 256,\n", " 320,\n", " 320\n", " ],\n", " \"conv_op\": \"torch.nn.modules.conv.Conv3d\",\n", " \"kernel_sizes\": [\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ]\n", " ],\n", " \"strides\": [\n", " [\n", " 1,\n", " 1,\n", " 1\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 1,\n", " 2,\n", " 2\n", " ]\n", " ],\n", " \"n_conv_per_stage\": [\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " \"n_conv_per_stage_decoder\": [\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " \"conv_bias\": true,\n", " \"norm_op\": \"torch.nn.modules.instancenorm.InstanceNorm3d\",\n", " \"norm_op_kwargs\": {\n", " \"eps\": 1e-05,\n", " \"affine\": true\n", " },\n", " \"dropout_op\": null,\n", " \"dropout_op_kwargs\": null,\n", " \"nonlin\": \"torch.nn.LeakyReLU\",\n", " \"nonlin_kwargs\": {\n", " \"inplace\": true\n", " }\n", " },\n", " \"_kw_requires_import\": [\n", " \"conv_op\",\n", " \"norm_op\",\n", " \"dropout_op\",\n", " \"nonlin\"\n", " ]\n", " },\n", " \"batch_dice\": false,\n", " \"next_stage\": \"3d_cascade_fullres\"\n", " },\n", " \"3d_fullres\": {\n", " \"data_identifier\": \"nnUNetPlans_3d_fullres\",\n", " \"preprocessor_name\": \"DefaultPreprocessor\",\n", " \"batch_size\": 2,\n", " \"patch_size\": [\n", " 48,\n", " 224,\n", " 192\n", " ],\n", " \"median_image_size_in_voxels\": [\n", " 112.0,\n", " 512.0,\n", " 512.0\n", " ],\n", " \"spacing\": [\n", " 2.5,\n", " 0.7929689884185791,\n", " 0.7929689884185791\n", " ],\n", " \"normalization_schemes\": [\n", " \"CTNormalization\"\n", " ],\n", " \"use_mask_for_norm\": [\n", " false\n", " ],\n", " \"resampling_fn_data\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_seg\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_data_kwargs\": {\n", " \"is_seg\": false,\n", " \"order\": 3,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"resampling_fn_seg_kwargs\": {\n", " \"is_seg\": true,\n", " \"order\": 1,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"resampling_fn_probabilities\": \"resample_data_or_seg_to_shape\",\n", " \"resampling_fn_probabilities_kwargs\": {\n", " \"is_seg\": false,\n", " \"order\": 1,\n", " \"order_z\": 0,\n", " \"force_separate_z\": null\n", " },\n", " \"architecture\": {\n", " \"network_class_name\": \"dynamic_network_architectures.architectures.unet.PlainConvUNet\",\n", " \"arch_kwargs\": {\n", " \"n_stages\": 6,\n", " \"features_per_stage\": [\n", " 32,\n", " 64,\n", " 128,\n", " 256,\n", " 320,\n", " 320\n", " ],\n", " \"conv_op\": \"torch.nn.modules.conv.Conv3d\",\n", " \"kernel_sizes\": [\n", " [\n", " 1,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ],\n", " [\n", " 3,\n", " 3,\n", " 3\n", " ]\n", " ],\n", " \"strides\": [\n", " [\n", " 1,\n", " 1,\n", " 1\n", " ],\n", " [\n", " 1,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " [\n", " 1,\n", " 2,\n", " 2\n", " ]\n", " ],\n", " \"n_conv_per_stage\": [\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " \"n_conv_per_stage_decoder\": [\n", " 2,\n", " 2,\n", " 2,\n", " 2,\n", " 2\n", " ],\n", " \"conv_bias\": true,\n", " \"norm_op\": \"torch.nn.modules.instancenorm.InstanceNorm3d\",\n", " \"norm_op_kwargs\": {\n", " \"eps\": 1e-05,\n", " \"affine\": true\n", " },\n", " \"dropout_op\": null,\n", " \"dropout_op_kwargs\": null,\n", " \"nonlin\": \"torch.nn.LeakyReLU\",\n", " \"nonlin_kwargs\": {\n", " \"inplace\": true\n", " }\n", " },\n", " \"_kw_requires_import\": [\n", " \"conv_op\",\n", " \"norm_op\",\n", " \"dropout_op\",\n", " \"nonlin\"\n", " ]\n", " },\n", " \"batch_dice\": true\n", " },\n", " \"3d_cascade_fullres\": {\n", " \"inherits_from\": \"3d_fullres\",\n", " \"previous_stage\": \"3d_lowres\"\n", " }\n", " },\n", " \"experiment_planner_used\": \"ExperimentPlanner\",\n", " \"label_manager\": \"LabelManager\",\n", " \"foreground_intensity_properties_per_channel\": {\n", " \"0\": {\n", " \"max\": 1038.0,\n", " \"mean\": 94.82794189453125,\n", " \"median\": 99.0,\n", " \"min\": -620.0,\n", " \"percentile_00_5\": -40.0,\n", " \"percentile_99_5\": 176.0,\n", " \"std\": 40.52839660644531\n", " }\n", " }\n", " }\n", "}\n" ] } ], "source": [ "print(json.dumps(nnunet_plans,indent=4))" ] }, { "cell_type": "markdown", "id": "30195c6c-f9ba-41d6-a060-8518bb5f7994", "metadata": {}, "source": [ "### Copy nnUNetPlans into Transfer Folder\n", "\n", "To share the nnUNet plans with the other sites, copy the `nnUNetPlans.json` file into the `src/Spleen/pymaia_nvflare` Folder." ] }, { "cell_type": "code", "execution_count": 61, "id": "771b3750-e06e-4579-a6bc-daa90a57ca04", "metadata": { "tags": [] }, "outputs": [], "source": [ "with open(Path(\"/home/maia-user/shared/src/Spleen/pymaia_nvflare/nnUNetPlans.json\"),\"w\") as f:\n", " json.dump(nnunet_plans,f)" ] }, { "cell_type": "markdown", "id": "5a71ce9bbbdd895", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "## 3. Preprocess\n", "\n", "After the nnUNet plans have been shared with the other sites, the data has to be preprocessed according to the nnUNet plans. This step has to be done on all the sites, except the one where the nnUNet experiment has been planned." ] }, { "cell_type": "markdown", "id": "506c02f9-0260-43c7-a52a-464b945c6ee6", "metadata": {}, "source": [ "### Submit Job" ] }, { "cell_type": "code", "execution_count": 34, "id": "5d00082ded26cd86", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"preprocess\")))" ] }, { "cell_type": "code", "execution_count": 35, "id": "dd04d7cc-8986-4c8e-bfc8-c5839651125c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sess.monitor_job(job_id)" ] }, { "cell_type": "code", "execution_count": 98, "id": "d957fb19-834d-4ab8-824a-64d7baa108ad", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dataset109_Task09_Spleen\n", "Preprocessing dataset Dataset109_Task09_Spleen\n", "Configuration: 3d_fullres...\n", "2024-12-16 14:13:18,552 - worker_process - INFO - Worker_process started.\n", "2024-12-16 14:13:18,584 - AuxRunner - INFO - registered aux handler for topic __end_run__\n", "2024-12-16 14:13:18,584 - AuxRunner - INFO - registered aux handler for topic __do_task__\n", "2024-12-16 14:13:18,652 - CoreCell - INFO - site-2.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce: created backbone internal connector to tcp://localhost:7367 on parent\n", "2024-12-16 14:13:18,653 - CoreCell - INFO - site-2.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce: created backbone external connector to grpc://localhost:8002\n", "2024-12-16 14:13:18,653 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE tcp://localhost:7367] is starting\n", "2024-12-16 14:13:18,653 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 ACTIVE grpc://localhost:8002] is starting\n", "2024-12-16 14:13:18,654 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 127.0.0.1:40002 => 127.0.0.1:7367] is created: PID: 10699\n", "2024-12-16 14:13:18,654 - FederatedClient - INFO - Wait for client_runner to be created.\n", "2024-12-16 14:13:18,655 - FederatedClient - INFO - Got client_runner after 0.0007114410400390625 seconds\n", "2024-12-16 14:13:18,656 - FederatedClient - INFO - Got the new primary SP: grpc://localhost:8002\n", "2024-12-16 14:13:18,656 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'\n", "2024-12-16 14:13:18,657 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002\n", "2024-12-16 14:13:18,657 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => localhost:8002] is created: PID: 10699\n", "2024-12-16 14:13:19,162 - Cell - INFO - broadcast: channel='aux_communication', topic='__sync_runner__', targets=['server.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce'], timeout=2.0\n", "2024-12-16 14:13:19,183 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: synced to Server Runner in 0.5214066505432129 seconds\n", "2024-12-16 14:13:19,184 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REQUEST\n", "2024-12-16 14:13:19,184 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REPLY\n", "2024-12-16 14:13:19,185 - ReliableMessage - INFO - enabled reliable message: max_request_workers=20 query_interval=2.0\n", "2024-12-16 14:13:20,389 - AuxRunner - INFO - registered aux handler for topic fed.event\n", "2024-12-16 14:13:20,389 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: client runner started\n", "2024-12-16 14:13:20,445 - Communicator - INFO - Received from example_project server. getTask: preprocess size: 688B (688 Bytes) time: 0.054908 seconds\n", "2024-12-16 14:13:20,445 - FederatedClient - INFO - pull_task completed. Task name:preprocess Status:True \n", "2024-12-16 14:13:20,445 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: got task assignment: name=preprocess, id=aa5312dd-d56b-4077-8606-a51f1919b20e\n", "2024-12-16 14:13:20,445 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: invoking task executor nnUNetExecutor\n", "2024-12-16 14:13:21,330 - PyMAIA - INFO - PyMAIA GPU Available: True\n", "2024-12-16 14:13:21,835 - numexpr.utils - INFO - Note: NumExpr detected 20 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 16.\n", "2024-12-16 14:13:21,835 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.\n", "2024-12-16 14:17:33,179 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: finished processing task\n", "2024-12-16 14:17:33,179 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: try #1: sending task result to server\n", "2024-12-16 14:17:33,179 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: checking task ...\n", "2024-12-16 14:17:33,179 - Cell - INFO - broadcast: channel='aux_communication', topic='__task_check__', targets=['server.aadd4d58-ad6c-4e6f-acba-ae0df79d03ce'], timeout=5.0\n", "2024-12-16 14:17:33,183 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: start to send task result to server\n", "2024-12-16 14:17:33,183 - FederatedClient - INFO - Starting to push execute result.\n", "2024-12-16 14:17:33,187 - Communicator - INFO - SubmitUpdate size: 4.9KB (4932 Bytes). time: 0.003701 seconds\n", "2024-12-16 14:17:33,187 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, task_name=preprocess, task_id=aa5312dd-d56b-4077-8606-a51f1919b20e]: task result sent to server\n", "2024-12-16 14:17:33,406 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce, peer=example_project, peer_run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: received request from Server to end current RUN\n", "2024-12-16 14:17:35,189 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: started end-run events sequence\n", "2024-12-16 14:17:35,189 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: ABOUT_TO_END_RUN fired\n", "2024-12-16 14:17:35,189 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: Firing CHECK_END_RUN_READINESS ...\n", "2024-12-16 14:17:35,190 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: END_RUN fired\n", "2024-12-16 14:17:35,190 - ReliableMessage - INFO - ReliableMessage is shutdown\n", "2024-12-16 14:17:35,192 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 10699\n", "2024-12-16 14:17:35,193 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 10699\n", "2024-12-16 14:17:35,193 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]\n", "2024-12-16 14:17:35,193 - FederatedClient - INFO - Shutting down client run: site-2\n", "2024-12-16 14:17:35,422 - ReliableMessage - INFO - shutdown reliable message monitor\n", "2024-12-16 14:17:36,016 - ClientRunner - INFO - [identity=site-2, run=aadd4d58-ad6c-4e6f-acba-ae0df79d03ce]: Client is stopping ...\n", "2024-12-16 14:17:37,524 - MPM - INFO - MPM: Good Bye!\n", "2024-12-16 14:13:25 - PyMAIA - INFO - PyMAIA GPU Available: True\n", "\n", " 0%| | 0/24 [00:00/jobs/bundle_training -w /bundle_training -sd nnUNet_NVFlare/ -f bundle_training-client-/config_fed_client.conf resume_training=epoch_number\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "26f07f91-32de-4a34-864d-cecc898a5be1", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"bundle_training\")))" ] }, { "cell_type": "code", "execution_count": null, "id": "c7b8c5c5-6a67-4684-958e-79de02e028c4", "metadata": {}, "outputs": [], "source": [ "job_id = \"e5072797-f93b-4d43-9bbd-6d947f894856\"\n", "\n", "sess.monitor_job(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "82b89e29-8ab4-4fde-a46d-d0f6112cdf5e", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "93278c9b-1736-4ac7-8cb2-45e201b4c1b7", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"run_bundle_train\",\"cmd.sh\"),\"r\") as f:\n", " print(f.read())" ] }, { "cell_type": "markdown", "id": "463011e88cf642ec", "metadata": {}, "source": [ "To check the training logs, run the following command:" ] }, { "cell_type": "code", "execution_count": null, "id": "f224b65a-af93-4982-87ce-fe63d8297f59", "metadata": { "scrolled": true }, "outputs": [], "source": [ "\n", "#bundle_name = \"\" #AutoPETLymphoma_nnUNet\n", "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile.log\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile_train_out.log\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile_train_err.log\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"ls {client_id} {bundle_name}/models\")['data'][0]['data'])" ] }, { "cell_type": "markdown", "id": "8c9bb767", "metadata": {}, "source": [ "## 4.1 Federated Learning Training\n", "\n", "After the individual training at each site, the Federated Learning training can be started. The Federated Learning training will aggregate the local gradients from the different sites and update the global model. The Federated Learning training will be conducted in rounds, with each round consisting of a number of local epochs. The Federated Learning training will be conducted in a secure and privacy-preserving manner, with the data remaining on the client side and only the local gradients being shared with the server." ] }, { "cell_type": "markdown", "id": "5a50b0c4", "metadata": {}, "source": [ "Prior to starting the FL training, follow the steps below to correctly configure the federation:\n", "\n", "1. Copy the `/src` folder to the src folder of the NVFlare jobs.\n", "\n", "2. Upload `requirements.txt` and the MONAI Bundle to the server. \n", "\n", "3. Upload the `plans.json` and `dataset.json` files to the server (in `/models`).\n", " `dataset.json` can be in the form:\n", " ```json\n", " {\n", " \"task\": \"Dataset109_Task09_Spleen\",\n", " \"dim\": 3,\n", " \"test_labels\": true,\n", " \"tensorImageSize\": \"4D\",\n", " \"channel_names\": {\"0\": \"ct\"},\n", " \"labels\": {\"background\": 0, \"Spleen\": 1},\n", " \"numTraining\": 0,\n", " \"numTest\": 0,\n", " \"training\": [],\n", " \"test\": [],\n", " \"file_ending\": \".nii.gz\"\n", " }\n", " ```\n", "4. Specify `BUNDLE_ROOT` in the server bundle configuration file (`/configs/train.yaml`).\n", "\n", "5. Install the required packages on the server:\n", " ```bash\n", " pip install -r requirements.txt\n", " ```\n", "\n", "6. Reconfigure the clients accordingly, adding:\n", " ```yaml\n", " pymaia_config_file: \"/Dataset_.json\"\n", " nnunet_model_folder: \"/Dataset_/___3d_fullres\"\n", " bundle_root: \"\"\n", " ```\n", "7. Specify the following parameters in the `Experiment` configuration:\n", " ```yaml\n", " num_rounds: 100\n", " server_bundle_root: \"/workspace/Spleen_Bundle\"\n", " start_round: 0\n", " local_epochs: 10\n", " Bundle_File: \"\"\n", " ```\n", "\n", " \n", "8. Copy the nnUNet Bundle into the `src/` folder.\n", "\n", "9. Re-execute the `generate_job_configs` script to update the job configurations." ] }, { "cell_type": "markdown", "id": "cb860d83", "metadata": {}, "source": [ "### Prepare Bundle for FL Training\n", "\n", "The MONAI Bundle has to be prepared for the Federated Learning training. In this step, the MONAI Bundle will be extracted on client, and the train and evaluation bundle parameters will be overwritten to match the Federated Learning training configuration.\n", "\n", "To prepare the MONAI Bundle for the Federated Learning training, run the following command:\n" ] }, { "cell_type": "code", "execution_count": 22, "id": "b4285b49", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"prepare_bundle\")))" ] }, { "cell_type": "code", "execution_count": 23, "id": "7a28aaf7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sess.monitor_job(job_id)" ] }, { "cell_type": "code", "execution_count": 24, "id": "40117eda", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2024-12-16 15:01:02,444 - worker_process - INFO - Worker_process started.\n", "2024-12-16 15:01:02,476 - AuxRunner - INFO - registered aux handler for topic __end_run__\n", "2024-12-16 15:01:02,476 - AuxRunner - INFO - registered aux handler for topic __do_task__\n", "2024-12-16 15:01:02,545 - CoreCell - INFO - site-2.140eb6cc-3036-4721-a247-9497f8392cdd: created backbone internal connector to tcp://localhost:7367 on parent\n", "2024-12-16 15:01:02,545 - CoreCell - INFO - site-2.140eb6cc-3036-4721-a247-9497f8392cdd: created backbone external connector to grpc://localhost:8002\n", "2024-12-16 15:01:02,545 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE tcp://localhost:7367] is starting\n", "2024-12-16 15:01:02,546 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00002 ACTIVE grpc://localhost:8002] is starting\n", "2024-12-16 15:01:02,546 - FederatedClient - INFO - Wait for client_runner to be created.\n", "2024-12-16 15:01:02,547 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 127.0.0.1:33150 => 127.0.0.1:7367] is created: PID: 18891\n", "2024-12-16 15:01:02,547 - FederatedClient - INFO - Got client_runner after 0.00031495094299316406 seconds\n", "2024-12-16 15:01:02,547 - FederatedClient - INFO - Got the new primary SP: grpc://localhost:8002\n", "2024-12-16 15:01:02,548 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'\n", "2024-12-16 15:01:02,551 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - created secure channel at localhost:8002\n", "2024-12-16 15:01:02,552 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 N/A => localhost:8002] is created: PID: 18891\n", "2024-12-16 15:01:03,054 - Cell - INFO - broadcast: channel='aux_communication', topic='__sync_runner__', targets=['server.140eb6cc-3036-4721-a247-9497f8392cdd'], timeout=2.0\n", "2024-12-16 15:01:03,059 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: synced to Server Runner in 0.5054581165313721 seconds\n", "2024-12-16 15:01:03,059 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REQUEST\n", "2024-12-16 15:01:03,059 - AuxRunner - INFO - registered aux handler for topic RM.RELIABLE_REPLY\n", "2024-12-16 15:01:03,059 - ReliableMessage - INFO - enabled reliable message: max_request_workers=20 query_interval=2.0\n", "2024-12-16 15:01:04,261 - AuxRunner - INFO - registered aux handler for topic fed.event\n", "2024-12-16 15:01:04,261 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: client runner started\n", "2024-12-16 15:01:04,309 - Communicator - INFO - Received from example_project server. getTask: prepare_bundle size: 696B (696 Bytes) time: 0.047104 seconds\n", "2024-12-16 15:01:04,309 - FederatedClient - INFO - pull_task completed. Task name:prepare_bundle Status:True \n", "2024-12-16 15:01:04,309 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd]: got task assignment: name=prepare_bundle, id=e7202854-1e39-40df-bf79-d0dd55f1dc6c\n", "2024-12-16 15:01:04,309 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: invoking task executor nnUNetExecutor\n", "2024-12-16 15:01:05,275 - PyMAIA - INFO - PyMAIA GPU Available: True\n", "2024-12-16 15:01:05,878 - numexpr.utils - INFO - Note: NumExpr detected 20 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 16.\n", "2024-12-16 15:01:05,879 - numexpr.utils - INFO - NumExpr defaulting to 16 threads.\n", "2024-12-16 15:01:08,389 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: finished processing task\n", "2024-12-16 15:01:08,390 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: try #1: sending task result to server\n", "2024-12-16 15:01:08,390 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: checking task ...\n", "2024-12-16 15:01:08,390 - Cell - INFO - broadcast: channel='aux_communication', topic='__task_check__', targets=['server.140eb6cc-3036-4721-a247-9497f8392cdd'], timeout=5.0\n", "2024-12-16 15:01:08,394 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: start to send task result to server\n", "2024-12-16 15:01:08,394 - FederatedClient - INFO - Starting to push execute result.\n", "2024-12-16 15:01:08,398 - Communicator - INFO - SubmitUpdate size: 918B (918 Bytes). time: 0.003894 seconds\n", "2024-12-16 15:01:08,398 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd, task_name=prepare_bundle, task_id=e7202854-1e39-40df-bf79-d0dd55f1dc6c]: task result sent to server\n", "2024-12-16 15:01:08,797 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd, peer=example_project, peer_run=140eb6cc-3036-4721-a247-9497f8392cdd]: received request from Server to end current RUN\n", "2024-12-16 15:01:10,400 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: started end-run events sequence\n", "2024-12-16 15:01:10,401 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: ABOUT_TO_END_RUN fired\n", "2024-12-16 15:01:10,402 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: Firing CHECK_END_RUN_READINESS ...\n", "2024-12-16 15:01:10,402 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: END_RUN fired\n", "2024-12-16 15:01:10,403 - ReliableMessage - INFO - ReliableMessage is shutdown\n", "2024-12-16 15:01:10,412 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 18891\n", "2024-12-16 15:01:10,414 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00003 Not Connected] is closed PID: 18891\n", "2024-12-16 15:01:10,415 - nvflare.fuel.f3.drivers.grpc_driver.GrpcDriver - INFO - CLIENT: finished connection [CN00003 Not Connected]\n", "2024-12-16 15:01:10,416 - FederatedClient - INFO - Shutting down client run: site-2\n", "2024-12-16 15:01:10,670 - ClientRunner - INFO - [identity=site-2, run=140eb6cc-3036-4721-a247-9497f8392cdd]: Client is stopping ...\n", "2024-12-16 15:01:11,069 - ReliableMessage - INFO - shutdown reliable message monitor\n", "2024-12-16 15:01:12,176 - MPM - INFO - MPM: Good Bye!\n" ] } ], "source": [ "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": 25, "id": "956a1545", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'site-2': {'Bundle_Path': '/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-2/startup/../140eb6cc-3036-4721-a247-9497f8392cdd/app_site-2/custom/Spleen_Bundle'}, 'site-1': {'Bundle_Path': '/home/maia-user/Tutorials/NVFlare_POC/example_project/prod_00/site-1/startup/../140eb6cc-3036-4721-a247-9497f8392cdd/app_site-1/custom/Spleen_Bundle'}}\n" ] } ], "source": [ "job_dir = sess.download_job_result(job_id)\n", "\n", "import json\n", "from pathlib import Path\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"run_bundle_prepare\",\"bundle_prepare.json\"),\"r\") as f:\n", " print(json.load(f))" ] }, { "cell_type": "markdown", "id": "039eaf71", "metadata": {}, "source": [ "### Run FL Training" ] }, { "cell_type": "code", "execution_count": null, "id": "20744dcf", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"fl_bundle\")))" ] }, { "cell_type": "code", "execution_count": 64, "id": "c240462e-d098-49c9-aa1e-a50ef6c9d61b", "metadata": {}, "outputs": [], "source": [ "sess.abort_job(job_id)" ] }, { "cell_type": "code", "execution_count": 72, "id": "59e8d70c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "session_inactive\n" ] } ], "source": [ "client_id = \"site-1\"\n", "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])" ] }, { "cell_type": "markdown", "id": "8b65df78", "metadata": {}, "source": [ "To download the trained models from the clients and upload them to the server, run the following command:" ] }, { "cell_type": "code", "execution_count": 9, "id": "86caea60", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "058c0c82", "metadata": {}, "outputs": [], "source": [ "import minio\n", "\n", "client = minio.Minio(\n", " endpoint=\"\",\n", " access_key=\"\",\n", " secret_key=\"\",\n", " secure=True\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "ee88be98", "metadata": {}, "outputs": [], "source": [ "client.make_bucket(\"\")\n", "\n", "client.fput_object(\"\",\"Lymphoma_nnUNet_Bundle/models/FL_global_model.pt\",\"FL_global_model.pt\")\n", "client.fput_object(\"\",\"Lymphoma_nnUNet_Bundle/models/best_FL_global_model.pt\",\"best_FL_global_model.pt\")\n" ] }, { "cell_type": "markdown", "id": "d2f1d609", "metadata": { "jp-MarkdownHeadingCollapsed": true }, "source": [ "## Resume FL Training\n", "\n", "To resume the Federated Learning training from an existing checkpoint:\n", "\n", "1. Download the global models from the server (``` sess.download_job_result(job_id)```) and upload them to the MONAI Bundle in the server (`/models`)\n", "\n", "2. Add the following parameters to the server configuration file:\n", " ```yaml\n", " nnunet_trainer_def:\n", " pretrained_model: \"/models/FL_global_model.pt\"\n", " ```\n", "\n", "3. Update the Experiment Configuration file with the new start_round and num_rounds values.\n", "\n", " ```yaml\n", " start_round: \n", " num_rounds: \n", " ```\n", "4. Re-execute the `generate_job_configs` script to update the job configurations." ] }, { "cell_type": "markdown", "id": "4cf2a900-9d16-4424-a603-6f638aee1199", "metadata": {}, "source": [ "## 5. MONAI Bundle to nnUNet Conversion\n", "\n", "Once the training is completed, the model can be converted to the nnUNet format. This step is necessary to be able to validate the model using the nnUNet framework.\n", "\n", "Since the training is performed using the MONAI Bundle format, the model has to be converted to the nnUNet format. The conversion will be done using the `convert_model` job, which will convert the MONAI Bundle model to the nnUNet format." ] }, { "cell_type": "code", "execution_count": null, "id": "5d8c39c7-dda9-435b-bdb6-20f13e7ed7c2", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "job_id = sess.submit_job(str(Path(\"\").joinpath(\"jobs\",\"convert_model\")))" ] }, { "cell_type": "code", "execution_count": null, "id": "07542c94-3645-4a00-8f9e-bd85c09d2e9f", "metadata": {}, "outputs": [], "source": [ "sess.monitor_job(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "8fd89092-9878-4a87-82b2-45b0fe93784a", "metadata": { "scrolled": true }, "outputs": [], "source": [ "client_id = \"KTH-Cloud\"\n", "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat server log.txt\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": null, "id": "00dd968f-8aaa-400f-a24a-db06cba7c7c2", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "3454dba2-952e-4889-82bf-e289a5c5c744", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"model_convert\",\"path.txt\"),\"r\") as f:\n", " print(f.read())" ] }, { "cell_type": "markdown", "id": "754be7e4", "metadata": {}, "source": [ "## 5.1 Fed Model Conversion" ] }, { "cell_type": "code", "execution_count": null, "id": "0cefe56f", "metadata": {}, "outputs": [], "source": [ "job_id = sess.submit_job(str(Path(JOB_DIR).joinpath(\"jobs\",\"convert_fed_model\")))" ] }, { "cell_type": "markdown", "id": "9c397366-c673-420d-9873-f544f3d4394f", "metadata": {}, "source": [ "## 6. Validation\n", "\n", "After the model has been converted to the nnUNet format, it can be validated using the nnUNet framework. The validation will be done using the `validate` job, which will validate the model using the nnUNet framework." ] }, { "cell_type": "code", "execution_count": null, "id": "2f363ebf-1936-42f1-aa12-01f42252ee28", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "job_id = sess.submit_job(str(Path(\"\").joinpath(\"jobs\",\"validate\")))" ] }, { "cell_type": "code", "execution_count": null, "id": "b2b16dea-3683-47c1-b788-b37bca9dd62d", "metadata": {}, "outputs": [], "source": [ "print(sess.api.do_command(f\"cat {client_id} logfile_validate_out.log\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile_validate_err.log\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": null, "id": "68794a97-7983-45d5-bae8-a6e438754152", "metadata": {}, "outputs": [], "source": [ "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])" ] }, { "cell_type": "markdown", "id": "76fa7f39434db9ba", "metadata": {}, "source": [ "To extract the validation results and display them in a table, run the following commands:" ] }, { "cell_type": "code", "execution_count": null, "id": "c9c0636c-a12e-405c-b9f1-ce0611b614b8", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "7d12d8dd-6130-4067-a34b-13db68df78c8", "metadata": {}, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"validation\",\"summary.json\"),\"r\") as f:\n", " summary = json.load(f)" ] }, { "cell_type": "code", "execution_count": null, "id": "6af0e8ea-e31c-4512-ad45-6ec34ab10a98", "metadata": {}, "outputs": [], "source": [ "config_dict = {\n", "\"label_dict\": {\"background\": 0,\n", " \"Lesion\": 1\n", "},\n", " \"label_suffix\":\".nii.gz\"\n", "}" ] }, { "cell_type": "code", "execution_count": null, "id": "21679bb7-ae68-4496-a42b-ae01f9c59d30", "metadata": {}, "outputs": [], "source": [ "df = []\n", "\n", "label_to_name = {v: k for k, v in config_dict[\"label_dict\"].items()}\n", "\n", "for case in summary['metric_per_case']:\n", " for label_id in case['metrics']:\n", " for metric in case['metrics'][label_id]:\n", " df.append({\n", " \"Case\": Path(case['reference_file']).name[:-len(config_dict[\"label_suffix\"])],\n", " \"Label\": label_to_name[int(label_id)],\n", " \"Metric\": metric,\n", " \"Value\": case['metrics'][label_id][metric]\n", " })" ] }, { "cell_type": "code", "execution_count": null, "id": "359b55b2-d0cf-4c87-bb78-0c96bd01ed5e", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "df = pd.DataFrame(df)" ] }, { "cell_type": "code", "execution_count": null, "id": "eff2422a-48f1-423a-b4dd-cc37bf0f237d", "metadata": {}, "outputs": [], "source": [ "import dtale.app as dtale_app\n", "import dtale\n", "\n", "dtale_app.JUPYTER_SERVER_PROXY = False\n", "\n", "d = dtale.show(df,\n", " host=\"127.0.0.1\",\n", " )" ] }, { "cell_type": "code", "execution_count": null, "id": "d1d97895-8d81-4a71-bcac-ccdcd30dea99", "metadata": {}, "outputs": [], "source": [ "DTALE_URL = d._main_url" ] }, { "cell_type": "code", "execution_count": null, "id": "dfb8b569-0e13-4797-907d-b47d05d1e0ee", "metadata": {}, "outputs": [], "source": [ "DTALE_URL" ] }, { "cell_type": "markdown", "id": "74bdc310-70e5-4f0e-b624-60493c1aaba0", "metadata": {}, "source": [ "## 7. Package MONAI Bundle\n", "\n", "\n", "After the model has been validated, it can be packaged into a MONAI Bundle. The MONAI Bundle will contain the model weights and the necessary configuration files to run the model. The packaging will be done using the `package_monai_bundle` job, which will package the model into a MONAI Bundle.\n", "\n", "The MONAI Bundle will be stored in the MLFlow server, available for download and deployment.\n", "\n", "To combine the MONAI Bundle with an existing MLFlow experiment, run the following command:\n", "```bash\n", "nvflare job create -j /jobs/package_monai_bundle -w /package_monai_bundle -sd nnUNet_NVFlare/ -f package_monai_bundle-client-/config_fed_client.conf MLFlow_Run_ID=\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "6f218edd-8bfd-48e8-aa4b-383bfc9484c4", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "\n", "job_id = sess.submit_job(str(Path(\"\").joinpath(\"jobs\",\"package_monai_bundle\")))" ] }, { "cell_type": "code", "execution_count": null, "id": "4006bc866cae27ec", "metadata": {}, "outputs": [], "source": [ "client_id = \"\"\n", "\n", "print(sess.api.do_command(f\"cat {client_id} logfile_package_out.log\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} logfile_package_err.log\")['data'][0]['data'])\n", "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": null, "id": "24b14b1d-079a-4814-b1d9-2415aac0d3e0", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "markdown", "id": "70e96b71fef7f42d", "metadata": {}, "source": [ "### 7.1 Package external nnUNet Experiments\n", "\n", "In case the nnUNet experiment has been planned, preprocessed and trained indipendentely from the NVFlare API, the model can be packaged into a MONAI Bundle using the `package_monai_bundle` job.\n", "To package an external nnUNet experiment, run the following command:\n", "```bash\n", "nvflare job create -j /jobs/package_monai_bundle -w /package_monai_bundle -sd nnUNet_NVFlare/ -f package_monai_bundle-client-/config_fed_client.conf external_training=True\n", "```" ] }, { "cell_type": "markdown", "id": "fa295670-a27b-4226-b844-8d6878f0c760", "metadata": {}, "source": [ "## 8. Cross Site Evaluation" ] }, { "cell_type": "code", "execution_count": null, "id": "6a29118e-0e19-436a-a4a9-9dee4f747f21", "metadata": { "scrolled": true }, "outputs": [], "source": [ "!nvflare job create -j jobs/cross_Site_Evaluation_nnUNet -w cross_Site_Evaluation_nnUNet/ -sd nnUNet_NVFlare/ --force " ] }, { "cell_type": "code", "execution_count": null, "id": "63536c39-c0e8-45c6-a4c4-5041f35f66a7", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "\n", "job_id = sess.submit_job(str(Path(\"..\").joinpath(\"..\",\"jobs\",\"cross_Site_Evaluation_nnUNet\")))" ] }, { "cell_type": "code", "execution_count": null, "id": "8648dfba-fdaa-4930-a816-6ea3cc5a0d6e", "metadata": {}, "outputs": [], "source": [ "sess.abort_job(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "6e06f5a0-2fc4-40b0-a495-efd22d6f3a99", "metadata": {}, "outputs": [], "source": [ "client_id = \"msf\"\n", "print(sess.api.do_command(f\"cat {client_id} {job_id}/log.txt\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"ls {client_id} Cross-Site-Validation/kth-cloud_positive_baseline_v2/extra_files\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"ls {client_id} Cross-Site-Validation/bundles/kth-cloud_positive_baseline_v2/models\")['data'][0]['data'])\n", "#print(sess.api.do_command(f\"ls {client_id} Cross-Site-Validation/inputs\")['data'][0]['data'])" ] }, { "cell_type": "code", "execution_count": null, "id": "7c67623b-cffe-408b-94e1-f4d3e9299421", "metadata": {}, "outputs": [], "source": [ "job_dir = sess.download_job_result(job_id)" ] }, { "cell_type": "code", "execution_count": null, "id": "818b9372-2e47-458a-afa8-d1f5c4b8d673", "metadata": {}, "outputs": [], "source": [ "import json\n", "from pathlib import Path\n", "\n", "\n", "with open(Path(job_dir).joinpath(\"workspace\",\"validation\",\"summary.json\"),\"r\") as f:\n", " cross_site_summary = json.load(f)" ] } ], "metadata": { "kernelspec": { "display_name": "NVFlare", "language": "python", "name": "nvflare" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.16" } }, "nbformat": 4, "nbformat_minor": 5 }