How to get FLUENT through workbench to use the GPU?

As subject. I am using Fluent through workbench on a HPC. How to get FLUENT to use the GPU?

Answers

  • tsiriakstsiriaks 3240 El Camino Real #290, Irvine, CA 92602Forum Coordinator

    What is the job scheduler on the cluster ?

    What is the OS version ?

    What is the graphics card on this node ?

    Normally, you may have use stand-alone Fluent to achieve that but it also depends on the info that i asked above.

  • tyw7tyw7 Member

    Redhat linux. It's a direct submission using qsub and mobaxterm.

  • tsiriakstsiriaks 3240 El Camino Real #290, Irvine, CA 92602Forum Coordinator

    I forgot to ask about ANSYS/Fluent version.

    Can you provide a sample command (and script, if exist) that you are using to submit the job ?

    Please also provide the output of the following commands

    cat /etc/*release

    lspci -vnn | grep VGA -A 12

    glxinfo | grep OpenGL

  • tyw7tyw7 Member

    Version 2020R2.


    I have attached the GPU / Fluent / Workbench scripts.


    The GPU one is pretty generic and doesn't show how to submit Fluent / Workbench to use the GPU


  • tyw7tyw7 Member

    I have attached the output of the strings you asked

  • tsiriakstsiriaks 3240 El Camino Real #290, Irvine, CA 92602Forum Coordinator

    Sorry, ANSYS employee are not allowed to download file attachment.

    Could you provide that in screenshots inline with text or perhaps copy the content and post as text here ?

  • [[username]@crescent-login1 ~]$ cat /etc/*release
    NAME="Red Hat Enterprise Linux Server"
    VERSION="7.7 (Maipo)"
    ID="rhel"
    ID_LIKE="fedora"
    VARIANT="Server"
    VARIANT_ID="server"
    VERSION_ID="7.7"
    PRETTY_NAME="Red Hat Enterprise Linux Server 7.7 (Maipo)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server"
    HOME_URL="https://www.redhat.com/"
    BUG_REPORT_URL="https://bugzilla.redhat.com/"
    
    REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
    REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
    REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
    REDHAT_SUPPORT_PRODUCT_VERSION="7.7"
    Red Hat Enterprise Linux Server release 7.7 (Maipo)
    Red Hat Enterprise Linux Server release 7.7 (Maipo)
    [[username]@crescent-login1 ~]$ lspci -vnn | grep VGA -A 12
    08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2000] [10de:1c30] (rev a1) (prog-if 00 [VGA controller])
            Subsystem: NVIDIA Corporation Device [10de:11b3]
            Physical Slot: 2-2
            Flags: bus master, fast devsel, latency 0, IRQ 206, NUMA node 0
            Memory at d1000000 (32-bit, non-prefetchable) [size=16M]
            Memory at e0000000 (64-bit, prefetchable) [size=256M]
            Memory at f0000000 (64-bit, prefetchable) [size=32M]
            I/O ports at 1000 [size=128]
            [virtual] Expansion ROM at d2000000 [disabled] [size=512K]
            Capabilities: <access denied>
            Kernel driver in use: nvidia
            Kernel modules: nouveau, nvidia
    
    --
    0a:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) [102b:0522] (rev 05) (prog-if 00 [VGA controller])
            Subsystem: Intel Corporation Device [8086:0103]
            Flags: bus master, fast devsel, latency 0, IRQ 19, NUMA node 0
            Memory at f3000000 (32-bit, prefetchable) [size=16M]
            Memory at d3010000 (32-bit, non-prefetchable) [size=16K]
            Memory at d2800000 (32-bit, non-prefetchable) [size=8M]
            Expansion ROM at d3000000 [disabled] [size=64K]
            Capabilities: <access denied>
            Kernel driver in use: mgag200
            Kernel modules: mgag200
    
    7f:08.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 0 [8086:3c80] (rev 07)
            Subsystem: Intel Corporation Device [8086:3582]
    [[username]@crescent-login1 ~]$ glxinfo | grep OpenGL
    OpenGL vendor string: Mesa Project
    OpenGL renderer string: Software Rasterizer
    OpenGL version string: 1.4 (2.1 Mesa 10.5.4)
    OpenGL extensions:
    


  • #!/bin/bash

    ##

    ## ANSYS submission script for PBS on CRESCENT

    ## -------------------------------------------

    ##

    ## Follow the 6 steps below to configure your job

    ## 

    ## STEP 1:

    ##

    ## Enter a job name after the -N on the line below:

    ##

    #PBS -N fluent_example

    ##

    ## STEP 2:

    ##

    ## Select the number of cpus/cores required by modifying the #PBS -l select line below

    ##

    ## Normally you select cpus in chunks of 16 cpus

    ## The Maximum value for ncpus is 16 and mpiprocs MUST be the same value as ncpus.

    ##

    ## If more than 16 cpus are required then select multiple chunks of 16

    ## e.g. 16 CPUs: select=1:ncpus=16:mpiprocs=16

    ## 32 CPUs: select=2:ncpus=16:mpiprocs=16

    ## 48 CPUs: select=3:ncpus=16:mpiprocs=16

    ## ..etc..

    ##

    #PBS -l select=1:ncpus=16:mpiprocs=16

    ##

    ## STEP 3:

    ##

    ## Select the correct queue by modifying the #PBS -q line below

    ##

    ## half_hour - 30 minutes

    ## one_hour -  1 hour

    ## three_hour  -  3 hours

    ## six_hour   -  6 hours

    ## half_day - 12 hours

    ## one_day - 24 hours

    ## two_day - 48 hours

    ## five_day - 120 hours

    ## ten_day - 240 hours (by special arrangement)

    ##

    #PBS -q half_hour

    ##

    ## STEP 4:

    ##

    ## Replace the [email protected] email address

    ## with your Cranfield email address on the #PBS -M line below:

    ## Your email address is NOT your username

    ##

    #PBS -m abe 

    #PBS -M [email protected]

    ##

    ## ====================================

    ## DO NOT CHANGE THE LINES BETWEEN HERE

    ## ====================================

    #PBS -l application=ansys

    #PBS -j oe

    #PBS -W sandbox=PRIVATE

    #PBS -k n

    ln -s $PWD $PBS_O_WORKDIR/$PBS_JOBID

    ## Change to working directory

    cd $PBS_O_WORKDIR

    # Get node list from PBS and format for ANSYS:

    ANSYS_NODES=`uniq -c ${PBS_NODEFILE} | awk -F. '{ print $1 }' | awk '{print $2 ":" $1}' | paste -s -d ':'`

    ## load module required by ANSYS/16.2

    module load libGLU/9.0.0-foss-2016b

    ## ========

    ## AND HERE

    ## ========

    ##

    ## STEP 5: 

    ## 

    ## Load the default application environment

    ## For a specific version add the version number, e.g.

    ## module load ANSYS/17.1

    ##

    module load ANSYS/16.2

    ##

    ## STEP 6: 

    ## 

    ## Put the correct parameters in the ansys execution line

    ## below to run the ANSYS launcher

    ## Unless you are sure do not change the default parameters

    ##

    ## The main parameters to modify are -

    ## application name - must match the version loaded

    ## input filename - change input.jou to your own filename


    ansys162 -dis -machines $ANSYS_NODES -dir $PBS_O_WORKDIR -b -i input.inp


    ## Tidy up the log directory

    ## DO NOT CHANGE THE LINE BELOW

    ## ============================

    rm $PBS_O_WORKDIR/$PBS_JOBID

    #

  • #!/bin/bash

    ##

    ## GPU submission script for PBS on CRESCENT

    ## -----------------------------------------

    ##

    ## Follow the 6 steps below to configure your job

    ## 

    ## STEP 1:

    ##

    ## Enter a job name after the -N on the line below:

    ##

    #PBS -N gpu_example

    ##

    ## STEP 2:

    ##

    ## Select the number of cpus/cores and GPUs required by modifying the #PBS -l select line below

    ##

    ## The Maximum value for ncpus is 8 and mpiprocs MUST be the same value as ncpus.

    ## The Maximum value for ngpus is 1 

    ## e.g. 1 GPU and 8 CPUs : select=1:ncpus=8:mpiprocs=8;ngpus=1

    ##

    #PBS -l select=1:ncpus=8:mpiprocs=8:ngpus=1

    ##

    ## STEP 3:

    ##

    ## There is only one queue for GPU jobs as defined in the #PBS -q line below

    ##

    #PBS -q gpu

    ##

    ## The default walltime in the gpu queue is one day(24 hours)

    ## The maximum walltime in the gpu queue is five days(120 hours)

    ## In order to increase the walltime modify the #PBS -l walltime line below

    ## and remove one of the leading # characters 

    ##

    ##PBS -l walltime=24:00:00

    ##

    ## STEP 4:

    ##

    ## Replace the [email protected] email address

    ## with your Cranfield email address on the #PBS -M line below:

    ## Your email address is NOT your username

    ##

    #PBS -m abe 

    #PBS -M [email protected]

    ##

    ## ====================================

    ## DO NOT CHANGE THE LINES BETWEEN HERE

    ## ====================================

    #PBS -j oe

    #PBS -v "CUDA_VISIBLE_DEVICES="

    #PBS -W sandbox=PRIVATE

    #PBS -k n

    ln -s $PWD $PBS_O_WORKDIR/$PBS_JOBID

    ## Allocated gpu(s)

    echo CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES

    ## Change to working directory

    cd $PBS_O_WORKDIR

    ## Calculate number of CPUs and GPUs

    export cpus=`cat $PBS_NODEFILE | wc -l`

    export gpus=`echo $CUDA_VISIBLE_DEVICES|awk -F"," '{print NF}'`

    ## ========

    ## AND HERE

    ## ========

    ##

    ## STEP 5: 

    ## 

    ## Load the default application environment

    ## For a specific version add the version number, e.g.

    ## module load CUDA/8.0.44

    ##

    module load CUDA

    ##

    ## STEP 6: 

    ## 

    ## Run gpu application 

    ##

    ## Put correct parameters and cuda application in the line below:

    ##

    /apps/software/CUDA/9.0.176/extras/demo_suite/deviceQuery


    ## Tidy up the log directory

    ## DO NOT CHANGE THE LINE BELOW

    ## ============================

    rm $PBS_O_WORKDIR/$PBS_JOBID

    #

  • #!/bin/bash

    ##

    ## Workbench submission script for PBS on CRESCENT

    ## -----------------------------------------------

    ##

    ## Follow the 6 steps below to configure your job

    ## 

    ## STEP 1:

    ##

    ## Enter a job name after the -N on the line below:

    ##

    #PBS -N workbench-example

    ##

    ## STEP 2:

    ##

    ## Select the number of cpus/cores required by modifying the #PBS -l select line below

    ##

    ## Normally you select cpus in chunks of 16 cpus

    ## The Maximum value for ncpus is 16 and mpiprocs MUST be the same value as ncpus.

    ##

    ## If more than 16 cpus are required then select multiple chunks of 16

    ## e.g. 16 CPUs: select=1:ncpus=16:mpiprocs=16

    ## 32 CPUs: select=2:ncpus=16:mpiprocs=16

    ## 48 CPUs: select=3:ncpus=16:mpiprocs=16

    ## ..etc..

    ##

    #PBS -l select=1:ncpus=16:mpiprocs=16

    ##

    ## STEP 3:

    ##

    ## Select the correct queue by modifying the #PBS -q line below

    ##

    ## half_hour - 30 minutes

    ## one_hour -  1 hour

    ## half_day - 12 hours

    ## three_hour  -  3 hours

    ## six_hour   -  6 hours

    ## one_day - 24 hours

    ## two_day - 48 hours

    ## five_day - 120 hours

    ## ten_day - 240 hours (by special arrangement)

    ##

    #PBS -q half_hour

    ##

    ## STEP 4:

    ##

    ## Replace the [email protected] email address

    ## with your Cranfield email address on the #PBS -M line below:

    ## Your email address is NOT your username

    ##

    #PBS -m abe 

    #PBS -M [email protected]

    ##

    ## ====================================

    ## DO NOT CHANGE THE LINES BETWEEN HERE

    ## ====================================

    #PBS -l application=ansys

    #PBS -j oe

    #PBS -W sandbox=PRIVATE

    #PBS -k n

    ln -s $PWD $PBS_O_WORKDIR/$PBS_JOBID

    ## Change to working directory

    cd $PBS_O_WORKDIR

    ## Calculate number of CPUs

    export cpus=`cat $PBS_NODEFILE | wc -l`

    # Get node list from PBS and format for ANSYS

    ANSYS_NODES=`uniq -c ${PBS_NODEFILE} | awk -F. '{ print $1 }' | awk '{print $2 ":" $1}' | paste -s -d ':'`

    ## load module required by ANSYS

    module load libXt libxcb

    ## Set dummy Display

    export DISPLAY=hpcxserv-1.central.cranfield.ac.uk:0.0

    ## ========

    ## AND HERE

    ## ========

    ##

    ## STEP 5: 

    ## 

    ## Load the default application environment

    ## For a specific version add the version number, e.g.

    ## module load ANSYS/17.1

    ##

    module load ANSYS

    ##

    ## STEP 6: 

    ## 

    ## Put correct parameters in workbench execution line

    ## below (i.e. replace wbsimulation.wbjn with your journal filename):

    ## see wbsimulation.wbjn in /apps/example/pbs for format of workbench journal file

    ##


    runwb2 -b -r wbsimulation.wbjn


    ## Tidy up the log directory

    ## DO NOT CHANGE THE LINE BELOW

    ## ============================

    rm $PBS_O_WORKDIR/$PBS_JOBID

    #

  • mbhidembhide Forum Coordinator
    edited September 17

    Hello

    If all you need to do is run solvers then explore configuring up ANSYS RSM to PBS or running Fluent under a scheduler per the documentation.

    The graphics card model may not work for GPU acceleration and may not offer much benefit.

    Running Workbench in batch may not be necessary unless other things are also being done.

Sign In or Register to comment.