Covid-19 classification pipeline

This is not for diagnostic use and no clinical decision should be made using the contents shared in this post. The model used for Covid-19 classification is not FDA approved for clinical use.

The purpose of this post is purely educational for the understanding of how an end-end clinical workflow can be produced using various components.

The full cycle shown here is as follows:

PACS (Orthanc) initiates an covid-19 classification request to NVIDIA Clara which does the inference and sends back the results back to PACS. Additionally, 3DSlicer is integrated with PACS to extract results for visualization purposes.

This flow is going to use following components:

Nvidia clara covid-19 pipeline is using pre-built models to create segmentation and classification output. Go through the pipeline from the above link to have a detailed understanding of the flow.

This post is going to extend the pipeline further to integrate it with an Orthanc PACS. Orthanc PACS is used as a DICOM server. An AI analysis can be triggered from the Orthanc PACS with a DICOM series and after the completion of the pipeline, inferences will be automatically fed back to Orthanc.

A desktop based 3DSlicer tool is integrated with the Orthanc PACS to directly get the inferences from the Orthanc PACS to 3DSlicer for visualization.

Orthanc PACS has Stone viewer which is utilized for reading the pipeline generated observation pdf files - covid_lung_ratio and covid probability.

I am using an AWS g4dn.2xlarge instance with 8 vCPUs and 1 GPU. I am on the Ubuntu 18.04 server. Minimum 6 vCPUs are required to execute this pipeline.

The DICOM series provided by the clara covid-19 classification pipeline is utilized for this post.

3DSlicer visualization of the input series

Output of the complete pipeline:

I could not open the DICOM SEG Modality file generated by clara-seg-writer operator (refer to https://ngc.nvidia.com/catalog/resources/nvidia:clara:clara_ai_covid19_pipeline ) in 3Dslicer DICOM viewer. I got an error while opening the file (ITK exception info: error in unknown: Could not create IO object for reading file) I need to further investigate the reason.

But the Clara render server (localhost:8080) could open the SEG Modality segmentations. Below two segmentations are created by the pipeline. One is for the lung and the other is for lung lesions.

Let’s walkthrough the steps taken to create the outputs.

Step1: Create the Clara Deploy pipeline

Follow the setup process described in the following link

https://ngc.nvidia.com/catalog/resources/nvidia:clara:clara_ai_covid19_pipeline/setup

This will create a pipeline

Step2: Configure the Clara DICOM Adapter for PACS to Pipeline communication

These references are valuable when using Clara DICOM Adapter -

For setting up the adapter

https://nvidia.github.io/clara-dicom-adapter/setup/setup.html

To create and delete AETitles for clara, sources and destinations use the APIs-

https://nvidia.github.io/clara-dicom-adapter/api/rest/config.html

The adapter will do few things for us-

  • It will receive DICOM series from PACS (Orthanc)

  • It will call the Clara pipeline to initiate the inference process

  • It will send the inference results back to PACS (Orthanc)

Clara Adapter will work as a DICOM device in this case. Everytime a DICOM device calls another device there are few important parameters exchanged for verification.

  • A “ Calling AETTitle” - AETitle that initiates the call.

  • A “Called AETitle” - AETitle that receives the call.

  • IP and Port information

A good source of basic understanding in PACS and networking between devices is here - https://www.orthanc-server.com/resources/2015-02-09-emsy-tutorial/index.html

2.1 Instal Orthanc open source PACS

Install Orthanc in a host machine. I am using the same host machine and using the Orthanc Docker container to install Orthanc. It is straightforward process and explained in - https://book.orthanc-server.com/users/docker.html

Few things to keep in mind-

Run the container in detached mode. Make sure to create a persistent volume of the DICOM database and map it to the local host. Create with restart option as “unless-stopped” to start the docker container automatically after restart of host. Use the orthanc image with plugins. Below Docker run command should help:

docker run -d \

-p 4242:4242 -p 8042:8042 \

--restart unless-stopped \

-v /orthanc/orthanc.json:/etc/orthanc/orthanc.json:ro \

-v /orthanc/orthanc-db/:/var/lib/orthanc/db/ \

jodogne/orthanc-plugins

I have created a folder /orthanc for storing the orthanc config and the database. These folders are mapped to the Docker container. Orthanc uses /etc/orthanc/orthan.json file as config source which is mapped to host /orthanc/orthanc.json. We will make necessary changes in Orthanc by making changes in the /orthanc/orthanc.json and recreating the docker container.

Use port number 8042 to open the Orthanc PACS. Default id/pwd is orthanc/orthanc

After logging in, upload a DICOM series using an Orthanc upload user interface. You can use the same series available with the clara model

After you upload you will find a DICOM series available in the studies

2.1 For PACS(Orthanc) to send DICOM series to Clara Adapter:

2.1.1 Orthanc has AETitle as “ORTHANC”

At the Clara Adapter side create a source for the Orthanc using the clara CLI command:

clara dicom create src -a ORTHANC -i 10.244.0.1

For me the Orthanc host ip is 10.244.0.1 and this will create a source name AETitle “ORTHANC” at clara adapter.

You can check the source by calling clara adapter API

There are 3 sources listed with one for AETitle ORTHANC.

2.1.2 Next, let’s create another AETitle which will trigger the Clara pipeline job.

For this run the Clara CLI command

clara dicom create aetitle -a COVIDAET pipeline-covid=<PIPELINE-ID>

Here replace the PIPELINE-ID with the pipeline-id created in step-1 above. This will create a AETitle, “COVIDAET”, which, when called, will trigger a pipeline job indicated by the pipeline-id.

You can check this clara AETitle by calling API:

curl –location –request GET ‘http://localhost:5000/api/config/claraaetitle/COVIDAET’

2.1.3 Install DCMTK

Install DCMTK to proxy the DICOM service class user (SCU) and service class provider (SCP) functionality

Refer to this doc for understanding SCP and SCU -

https://www.orthanc-server.com/resources/2015-02-09-emsy-tutorial/index.html

Install DCMTK by executing -

sudo apt-get install dcmtk

If you like you can create a local Dicom source called “MYPACS” in the host machine and issue a storescu command from the folder where input dicom source is stored to trigger the COVIDAET AETitle and associated pipelinejob.

The command you will use is -

storescu -v +sd +r -xb -aet “MYPACS” -aec “COVIDAET” AA.BB.CC.DD 104 ./

AA.BB.CC.DD is local host ip

This should complete the pipeline job and create an output file in Clara console to download form.

2.1.4 Update PACS server to include Clara AETitle (i.e “COVIDAET)

We need to include COVIDAET as known AETitle in Orthanc for the PACS to send a DICOM series to COVIDAET. We already created a source called “ORTHANC” in clara adapter and therefore clara adapter knows how to handle a request from PACS.

We will add a DICOM Modality for in orthanc config file for COVIDAET

2.1.5. Run a Storescu to associate source AETitle “ORTHANC” with Clara AETitle “COVIDAET”

Run the following command

storescu -v +sd +r -xb -aet “ORTHANC” -aec “COVIDAET” AA.BB.CC.DD 104 ./

AA.BB.CC.DD is host ip address, 104 is port

2.1.6 Run an incomplete loop by sending DICOM series from Orthanc

At this time you will be able to trigger COVIDAET and its associated pipeline job from Orthanc PACS. Select the DICOM study you uploaded and then select send to DICOM modality followed by “orthanc”. This should start a job

You can check the status in the Clara Console.

You may also see whether Orthanc has successfully passed the DICOM series to COVIDAET in Orthanc Jobs tab

If there is something wrong with the process check the kubernetes pods to understand reasons for failures.

Specially clara-adapter-* and clara-clara-platformapiserver-* will come handy in the debug process

After successful completion of the job in the console, you may download the output payload for offline analysis and read.

At this time, Clara render server (port 8080) will be updated with the segmentations as well.

Step 3: Configure Clara Adapter for sending inference back to PACS

If you want to send the results back to PACS instead (which is the most typical scenario), we will update the Clara adapter and PACS config to enable that communication.

In the Clara pipeline, there are steps to register results with outside AETitles.

Clara adapter has the default SCU AETitle as “ClaraSCU”. Orthanc PACS has AETitle “ORTHANC” and uses DICOM Port as 4242

3.1 Create a destination AETitle in Clara Adapter for Orthanc PACS

Use following command to create destination

clara dicom create dest -n orthanc -a ORTHANC -i 10.244.0.1 -p 4242

The `-n orthanc` is important as the same name will be used in the pipeline yaml file.

Change the pipeline yaml to reflect the Orthanc PACS as destination by indicating (`orthanc`). There are multiple places in the pipeline where the result got registered.

Changing pipeline yaml will require creation of a new pipeline.

Create a new pipeline

Delete the existing aetitle called COVIDAET and create a new one with the same name (COVIDAET) using the pipeline id from above

To delete clara adapter AETitles use the APIs available at https://nvidia.github.io/clara-dicom-adapter/api/rest/config.html

3.2 Add `ClaraSCU` in Orthanc PACS for Orthanc to accept results from Clara Adapter

Update Orthanc config file and restart orthanc

To restart docker, stop and delete docker container and start a new container. The data should persist as we are mapping the DICOM database in the host machine.

With this change `ClaraSCU`should appear in the DICOM modalities of Orthanc PACS

3.3 Start a job for starting a job from Orthanc PACS and getting the inference back in the PACS (Full cycle)

Dicom study loaded in Orthanc

Send to DICOM modality - `orthanc`

This should start a job on Orthanc

Clara console will show the progress of the job

`kubectl get pods` will show the working pods

Job completed

Success! The job is complete as shown in the dicom adapter.

New studies sent back to PACS

That completes the full cycle of sending DICOM series from PACS to getting the inference back in the PACS.

The Stone web viewer plugin shows the reports

I could not get the segmentation to open in the stone viewer, but used the Clara render server (port 8080) to show the segmentation.

At this point connections in PACS should look like this

Step 4: Connecting PACS with 3DSlicer for visualization

Install 3DSlicer in your visualization server, for me it is in my local machine.

For this we need to add your local machine ip and port in Orthanc PACS config dicom modalities.

And we need to initiate a network connection from 3dslicer with Orthanc ip and its dicom port (4242). It should import the study and the inferences in the local 3dslicer database.

I could not get the inferred segmentation visible in 3dslicer. The DICOM SEG file could not load. But the output segmentation was visible in the Clara render server.