How to get FLUENT through workbench to use the GPU?

tyw7

As subject. I am using Fluent through workbench on a HPC. How to get FLUENT to use the GPU?

tsiriaks

What is the job scheduler on the cluster ?

What is the OS version ?

What is the graphics card on this node ?

Normally, you may have use stand-alone Fluent to achieve that but it also depends on the info that i asked above.

tyw7

Redhat linux. It's a direct submission using qsub and mobaxterm.

tsiriaks

I forgot to ask about ANSYS/Fluent version.

Can you provide a sample command (and script, if exist) that you are using to submit the job ?

Please also provide the output of the following commands

cat /etc/*release

lspci -vnn | grep VGA -A 12

glxinfo | grep OpenGL

tyw7

Version 2020R2.

I have attached the GPU / Fluent / Workbench scripts.

The GPU one is pretty generic and doesn't show how to submit Fluent / Workbench to use the GPU

Submit script.zip

tyw7

Output.txt

I have attached the output of the strings you asked

tsiriaks

Sorry, ANSYS employee are not allowed to download file attachment.

Could you provide that in screenshots inline with text or perhaps copy the content and post as text here ?

tyw7

[[username]@crescent-login1 ~]$ cat /etc/*release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.7 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.7"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.7 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.7"
Red Hat Enterprise Linux Server release 7.7 (Maipo)
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[[username]@crescent-login1 ~]$ lspci -vnn | grep VGA -A 12
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation Device [10de:11b3]
        Physical Slot: 2-2
        Flags: bus master, fast devsel, latency 0, IRQ 206, NUMA node 0
        Memory at d1000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 1000 [size=128]
        [virtual] Expansion ROM at d2000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia

--
0a:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) [102b:0522] (rev 05) (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device [8086:0103]
        Flags: bus master, fast devsel, latency 0, IRQ 19, NUMA node 0
        Memory at f3000000 (32-bit, prefetchable) [size=16M]
        Memory at d3010000 (32-bit, non-prefetchable) [size=16K]
        Memory at d2800000 (32-bit, non-prefetchable) [size=8M]
        Expansion ROM at d3000000 [disabled] [size=64K]
        Capabilities: <access denied>
        Kernel driver in use: mgag200
        Kernel modules: mgag200

7f:08.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 0 [8086:3c80] (rev 07)
        Subsystem: Intel Corporation Device [8086:3582]
[[username]@crescent-login1 ~]$ glxinfo | grep OpenGL
OpenGL vendor string: Mesa Project
OpenGL renderer string: Software Rasterizer
OpenGL version string: 1.4 (2.1 Mesa 10.5.4)
OpenGL extensions:

tyw7

#!/bin/bash

##

## ANSYS submission script for PBS on CRESCENT

## -------------------------------------------

##

## Follow the 6 steps below to configure your job

## 

## STEP 1:

##

## Enter a job name after the -N on the line below:

##

#PBS -N fluent_example

##

## STEP 2:

##

## Select the number of cpus/cores required by modifying the #PBS -l select line below

##

## Normally you select cpus in chunks of 16 cpus

## The Maximum value for ncpus is 16 and mpiprocs MUST be the same value as ncpus.

##

## If more than 16 cpus are required then select multiple chunks of 16

## e.g. 16 CPUs: select=1:ncpus=16:mpiprocs=16

## 32 CPUs: select=2:ncpus=16:mpiprocs=16

## 48 CPUs: select=3:ncpus=16:mpiprocs=16

## ..etc..

##

#PBS -l select=1:ncpus=16:mpiprocs=16

##

## STEP 3:

##

## Select the correct queue by modifying the #PBS -q line below

##

## half_hour - 30 minutes

## one_hour -  1 hour

## three_hour  -  3 hours

## six_hour   -  6 hours

## half_day - 12 hours

## one_day - 24 hours

## two_day - 48 hours

## five_day - 120 hours

## ten_day - 240 hours (by special arrangement)

##

#PBS -q half_hour

##

## STEP 4:

##

## Replace the [email protected] email address

## with your Cranfield email address on the #PBS -M line below:

## Your email address is NOT your username

##

#PBS -m abe 

#PBS -M [email protected]

##

## ====================================

## DO NOT CHANGE THE LINES BETWEEN HERE

## ====================================

#PBS -l application=ansys

#PBS -j oe

#PBS -W sandbox=PRIVATE

#PBS -k n

ln -s $PWD $PBS_O_WORKDIR/$PBS_JOBID

## Change to working directory

cd $PBS_O_WORKDIR

# Get node list from PBS and format for ANSYS:

ANSYS_NODES=`uniq -c ${PBS_NODEFILE} | awk -F. '{ print $1 }' | awk '{print $2 ":" $1}' | paste -s -d ':'`

## load module required by ANSYS/16.2

module load libGLU/9.0.0-foss-2016b

## ========

## AND HERE

## ========

##

## STEP 5: 

## 

## Load the default application environment

## For a specific version add the version number, e.g.

## module load ANSYS/17.1

##

module load ANSYS/16.2

##

## STEP 6: 

## 

## Put the correct parameters in the ansys execution line

## below to run the ANSYS launcher

## Unless you are sure do not change the default parameters

##

## The main parameters to modify are -

## application name - must match the version loaded

## input filename - change input.jou to your own filename

ansys162 -dis -machines $ANSYS_NODES -dir $PBS_O_WORKDIR -b -i input.inp

## Tidy up the log directory

## DO NOT CHANGE THE LINE BELOW

## ============================

rm $PBS_O_WORKDIR/$PBS_JOBID

#

tyw7

#!/bin/bash

##

## GPU submission script for PBS on CRESCENT

## -----------------------------------------

##

## Follow the 6 steps below to configure your job

## 

## STEP 1:

##

## Enter a job name after the -N on the line below:

##

#PBS -N gpu_example

##

## STEP 2:

##

## Select the number of cpus/cores and GPUs required by modifying the #PBS -l select line below

##

## The Maximum value for ncpus is 8 and mpiprocs MUST be the same value as ncpus.

## The Maximum value for ngpus is 1 

## e.g. 1 GPU and 8 CPUs : select=1:ncpus=8:mpiprocs=8;ngpus=1

##

#PBS -l select=1:ncpus=8:mpiprocs=8:ngpus=1

##

## STEP 3:

##

## There is only one queue for GPU jobs as defined in the #PBS -q line below

##

#PBS -q gpu

##

## The default walltime in the gpu queue is one day(24 hours)

## The maximum walltime in the gpu queue is five days(120 hours)

## In order to increase the walltime modify the #PBS -l walltime line below

## and remove one of the leading # characters 

##

##PBS -l walltime=24:00:00

##

## STEP 4:

##

## Replace the [email protected] email address

## with your Cranfield email address on the #PBS -M line below:

## Your email address is NOT your username

##

#PBS -m abe 

#PBS -M [email protected]

##

## ====================================

## DO NOT CHANGE THE LINES BETWEEN HERE

## ====================================

#PBS -j oe

#PBS -v "CUDA_VISIBLE_DEVICES="

#PBS -W sandbox=PRIVATE

#PBS -k n

ln -s $PWD $PBS_O_WORKDIR/$PBS_JOBID

## Allocated gpu(s)

echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES

## Change to working directory

cd $PBS_O_WORKDIR

## Calculate number of CPUs and GPUs

export cpus=`cat $PBS_NODEFILE | wc -l`

export gpus=`echo $CUDA_VISIBLE_DEVICES|awk -F"," '{print NF}'`

## ========

## AND HERE

## ========

##

## STEP 5: 

## 

## Load the default application environment

## For a specific version add the version number, e.g.

## module load CUDA/8.0.44

##

module load CUDA

##

## STEP 6: 

## 

## Run gpu application 

##

## Put correct parameters and cuda application in the line below:

##

/apps/software/CUDA/9.0.176/extras/demo_suite/deviceQuery

## Tidy up the log directory

## DO NOT CHANGE THE LINE BELOW

## ============================

rm $PBS_O_WORKDIR/$PBS_JOBID

#

tyw7

#!/bin/bash

##

## Workbench submission script for PBS on CRESCENT

## -----------------------------------------------

##

## Follow the 6 steps below to configure your job

## 

## STEP 1:

##

## Enter a job name after the -N on the line below:

##

#PBS -N workbench-example

##

## STEP 2:

##

## Select the number of cpus/cores required by modifying the #PBS -l select line below

##

## Normally you select cpus in chunks of 16 cpus

## The Maximum value for ncpus is 16 and mpiprocs MUST be the same value as ncpus.

##

## If more than 16 cpus are required then select multiple chunks of 16

## e.g. 16 CPUs: select=1:ncpus=16:mpiprocs=16

## 32 CPUs: select=2:ncpus=16:mpiprocs=16

## 48 CPUs: select=3:ncpus=16:mpiprocs=16

## ..etc..

##

#PBS -l select=1:ncpus=16:mpiprocs=16

##

## STEP 3:

##

## Select the correct queue by modifying the #PBS -q line below

##

## half_hour - 30 minutes

## one_hour -  1 hour

## half_day - 12 hours

## three_hour  -  3 hours

## six_hour   -  6 hours

## one_day - 24 hours

## two_day - 48 hours

## five_day - 120 hours

## ten_day - 240 hours (by special arrangement)

##

#PBS -q half_hour

##

## STEP 4:

##

## Replace the [email protected] email address

## with your Cranfield email address on the #PBS -M line below:

## Your email address is NOT your username

##

#PBS -m abe 

#PBS -M [email protected]

##

## ====================================

## DO NOT CHANGE THE LINES BETWEEN HERE

## ====================================

#PBS -l application=ansys

#PBS -j oe

#PBS -W sandbox=PRIVATE

#PBS -k n

ln -s $PWD $PBS_O_WORKDIR/$PBS_JOBID

## Change to working directory

cd $PBS_O_WORKDIR

## Calculate number of CPUs

export cpus=`cat $PBS_NODEFILE | wc -l`

# Get node list from PBS and format for ANSYS

ANSYS_NODES=`uniq -c ${PBS_NODEFILE} | awk -F. '{ print $1 }' | awk '{print $2 ":" $1}' | paste -s -d ':'`

## load module required by ANSYS

module load libXt libxcb

## Set dummy Display

export DISPLAY=hpcxserv-1.central.cranfield.ac.uk:0.0

## ========

## AND HERE

## ========

##

## STEP 5: 

## 

## Load the default application environment

## For a specific version add the version number, e.g.

## module load ANSYS/17.1

##

module load ANSYS

##

## STEP 6: 

## 

## Put correct parameters in workbench execution line

## below (i.e. replace wbsimulation.wbjn with your journal filename):

## see wbsimulation.wbjn in /apps/example/pbs for format of workbench journal file

##

runwb2 -b -r wbsimulation.wbjn

## Tidy up the log directory

## DO NOT CHANGE THE LINE BELOW

## ============================

rm $PBS_O_WORKDIR/$PBS_JOBID

#

mbhide

Hello

If all you need to do is run solvers then explore configuring up ANSYS RSM to PBS or running Fluent under a scheduler per the documentation.

The graphics card model may not work for GPU acceleration and may not offer much benefit.

Running Workbench in batch may not be necessary unless other things are also being done.