March 7, 2023 at 11:04 pm
mkhademi
Subscriber

Thank you so much for your replies Rob and Mangesh,

I got some information from our IT administration. Here is the information they provided:

1) The cluster is running RHEL 8.6. The nodes have dual AMD EPYC 7763 64-Core Processor with 512 GiB of RAM
2) The compute nodes have HDR100 Infiniband interconnects
3) The scratch filesystem is BeeGFS 7.3.0
4) Scheduler is Slurm 22.05.07

I ran a simulation both with my Laptop and cluster with both using 8 cores. I got some information with your instructions after running a common unsteady case with 100 iterations:

The cluster gave me this information:

> (proc-stats)
 
------------------------------------------------------------------------------
       | Virtual Mem Usage (GB)   | Resident Mem Usage(GB)   |              
ID     | Current      Peak        | Current      Peak        | Page Faults  
------------------------------------------------------------------------------
host   | 0.591854     0.592873    | 0.174324     0.174324    | 591        
n0     | 0.818371     0.826019    | 0.175053     0.182316    | 259        
n1     | 0.814728     0.82201     | 0.167213     0.173817    | 214        
n2     | 0.815907     0.822296    | 0.165989     0.171776    | 255        
n3     | 0.81435      0.821392    | 0.168133     0.174389    | 175        
n4     | 0.814678     0.82206     | 0.165474     0.172218    | 168        
n5     | 0.814739     0.822018    | 0.164593     0.171059    | 252        
n6     | 0.814575     0.821838    | 0.165981     0.172546    | 253        
n7     | 0.815868     0.822704    | 0.164452     0.1703      | 258        
------------------------------------------------------------------------------
Total  | 7.11507      7.17321     | 1.51121      1.56274     | 2425       
------------------------------------------------------------------------------
 
------------------------------------------------------------------------------------------------
                    | Virtual Mem Usage (GB)    | Resident Mem Usage(GB)    | System Mem (GB)          
Hostname            | Current      Peak         | Current      Peak         |                          
------------------------------------------------------------------------------------------------
compute-b6-21.zarata| 7.11507      7.17321      | 1.51121      1.56274      | 503.141      
------------------------------------------------------------------------------------------------
Total               | 7.11507      7.17321      | 1.51121      1.56274      |           
------------------------------------------------------------------------------------------------
()
 
 
>  (sys-stats)
 
---------------------------------------------------------------------------------------
                    | CPU                                  | System Mem (GB)                  
Hostname            | Sock x Core x HT  Clock (MHz) Load   | Total        Available  
---------------------------------------------------------------------------------------
compute-b6-21.zarata| 2 x 64 x 1        0           58.77  | 503.141      362.15     
---------------------------------------------------------------------------------------
Total               | 128               -           -      | 503.141      362.15     
---------------------------------------------------------------------------------------
()
 
 
>  (show-affinity)
999999: 0 22 23 24 72 73 74 75 
0: 0 
1: 22 23 24 
2: 72 73 74 75 
3: 22 23 24 
4: 22 23 24 
5: 72 73 74 75 
6: 72 73 74 75 
7: 72 73 74 75 
 
999999: 0 22 23 24 72 73 74 75 
0: 0 22 23 24 
1: 0 22 23 24 
2: 0 22 23 24 
3: 0 22 23 24 
4: 72 73 74 75 
5: 72 73 74 75 
6: 72 73 74 75 
7: 72 73 74 75 
 
999999: 0 22 23 24 72 73 74 75 
0: 0 22 23 24 
1: 0 22 23 24 
2: 0 22 23 24 
3: 0 22 23 24 
4: 72 73 74 75 
5: 72 73 74 75 
6: 72 73 74 75 
7: 72 73 74 75 
  • The alptop gave me this information for the same case and 8 cores for the same number of iterations:

> (proc-stats)

 

----------------------------------------------

| Virtual Mem Usage (GB)|

ID | Current Peak | Page Faults

----------------------------------------------

host | 0.112896 0.144833 | 7.166e+04

n0 | 0.0862007 0.153492 | 2.659e+06

n1 | 0.0807686 0.134022 | 1.793e+06

n2 | 0.0791664 0.135555 | 3.148e+06

n3 | 0.0825119 0.134621 | 2.002e+06

n4 | 0.0780334 0.134251 | 3.438e+06

n5 | 0.0720711 0.130085 | 1.969e+06

n6 | 0.0833321 0.135273 | 1.328e+06

n7 | 0.0814285 0.135128 | 1.962e+06

----------------------------------------------

Total | 0.756409 1.23726 | 1.837e+07

----------------------------------------------

 

-----------------------------------------------------------------

| Virtual Mem Usage (GB) | System Mem (GB)

Hostname | Current Peak |

-----------------------------------------------------------------

DESKTOP-C6GP8UV | 0.75642 1.23726 | 15.7449

-----------------------------------------------------------------

Total | 0.75642 1.23726 |

-----------------------------------------------------------------

()

> (sys-stats)

 

---------------------------------------------------------------------------------------

| CPU | System Mem (GB)

Hostname | Sock x Core x HT Clock (MHz) Load (%)| Total Available

---------------------------------------------------------------------------------------

DESKTOP-C6GP8UV | 1 x 8 x 2 2304 14.8064| 15.7449 3.93291

---------------------------------------------------------------------------------------

Total | 16 - - | 15.7449 3.93291

---------------------------------------------------------------------------------------

()

> (show-affinity)

 

999999: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

0: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

1: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

2: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

3: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

4: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

5: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

6: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

7: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

()