Learning Forum

[chenyuanming]

Dear all,

we have bought the lsdyna hpc license installed on the cluster. But with some problems, I have a simulation, which was ran on the single computer with 16 cores before for 6 hours, but on the cluster, running the same input file needs 9 hours using more than 40 cores. I think the hyperthreading on the cluster is a problem, does anybody know how to run the simulation on the physical cores?

the second issue is that we have a simulation using lsdyna implicit with more than 10 million elements, we allocate more than 3 nodes, each node have 80 cores and 378GB system memory. but lsdyna seems always put the model into the head node, which means the simulation will run out of memory, but we have enough memory on other nodes, does anybody know how to fix this?

[Reno Genest]

Hyperthreading is not the issue; if you run with 40 cores on an 80 physical core machine, LS-DYNA will use 40 physical cores. How many elements do you have in the model? If the model is too small, then it is possible to running with 40 cores is slower than running with 16. You should have 10-20k elements per core for an efficient solution. If you have fewer elements per core, each core does not have enough work and the communication between the cores becomes the bottleneck. Also, are the cores in the 16 core machine the same as the cluster? Is it possible that the 16 core machine has newer and faster cpus? You can trying binding the MPI process to cores and see if you get performance gains:
To run on different nodes, you can create an appfile to specify how many cores to use on each node. See the following article:
Reno.

[chenyuanming]

Dear Reno thank you sincerely !
I think in order ro run on different nodes, wo just need to select number of nodes, then mpirun from many search results from google; I have tried the link of instructions on how to run across nodes, but it didn't work, can you give me more instructions on running across the nodes?

[Reno Genest]

Have you tried with an appfile? It is explained in the link above.
"Q4: How to run MPP job across nodes of a cluster?

A4:
(a) Create a file with commandline options. As you know for one node, it was straight-forward "mpirun -np $NCPU ./mppdyna i=input.key jobid=my_job_name memory=$memory memory2=$memory2". For multiple nodes, you will put this commandlines into a file, let's call this file appfile. Let us say, you have 4 nodes to use, with names host1, host2, host3 and host4. Each node has 12cores, and you want to use all 12 cores. You appfile will contain the following lines:

-h host1 -np 12 /path/to/solver/mppdyna i=input_file.key jobid=my_job_name memory=$memory memory2=$memory2
-h host2 -np 12 /path/to/solver/mppdyna
-h host3 -np 12 /path/to/solver/mppdyna
-h host4 -np 12 /path/to/solver/mppdyna

As you may have noticed, LS-DYNA binary should be hosted on a network drive where all the hosts have access to.
(b) Once the appfile is ready, you simply issue the following command from head node (host1) -

/opt/platform_mpi/bin/mpirun -prot -ibv -f ./appfile
"
Also, which MPI are you using? The command line arguments will change based on the MPI you are using. For example, you can find information on how to run Intel MPI here:
Reno.

[chenyuanming]

Hi
I have tried this file, it has no difference with select more nodes, and mpirun;
Besides, we bought lsdyna, but they installed the R10 version on the campus cluster, very old, the latest one is R13, I think. I don't know whether they can upgrade it
for the hyperthreading, our cluster has 80 cores for each node, 80 is actually virtual processors, only 40 physical cores for each nodes.
so I don't know if I can run on the physical nodes.

[Reno Genest]

LS-DYNA will use the physical cores. But, if you think hyperthreading is causing an issue, you can disable it. Google how to disable hyperthreading on your system.
I think you have all the information you need to run on multiple nodes. Just follow the instructions in the link above.
Good luck.

Reno.