K-Scale Cluster - Revision history

Ben: /* SLURM Commands */

2024-05-25T00:12:26Z

‎SLURM Commands

2024-05-25T00:10:45Z

‎SLURM Commands

2024-05-25T00:07:19Z

‎SLURM Commands

2024-05-25T00:06:27Z

‎SLURM Commands

2024-05-24T23:55:00Z

‎Andromeda Cluster

2024-05-24T23:51:44Z

‎Andromeda Cluster

2024-05-16T00:26:13Z

2024-05-16T00:25:53Z

2024-05-16T00:24:34Z

2024-05-07T05:08:49Z

@@ Line 124: / Line 124: @@
 <syntaxhighlight lang="bash">
-$ sbatch test.sh
+$ sbatch --gpus 16 test.sh
 Submitted batch job 461
 </syntaxhighlight>

@@ Line 135: / Line 135: @@
    compute  test.sh      ben  R       0:37      1 compute-permanent-node-285
 </syntaxhighlight>
 ==== Reserving a GPU ====

@@ Line 119: / Line 119: @@
 ... truncated
-</syntaxhiglight>
+</syntaxhighlight>
 Alternatively, I can queue the job using <code>sbatch</code>, which gives me the following result:

@@ Line 88: / Line 88: @@
 * There are 8 nodes of that type, all currently in <code>idle</code> state
 * The node names are things like <code>compute-permanent-node-68</code>
 ==== Reserving a GPU ====

← Older revision		Revision as of 23:55, 24 May 2024
Line 54:		Line 54:

	Don't do anything computationally expensive on the main node or you will crash it for everyone. Instead, when you need to run some experiments, reserve a GPU (see below).		Don't do anything computationally expensive on the main node or you will crash it for everyone. Instead, when you need to run some experiments, reserve a GPU (see below).
		+
		+	==== SLURM Commands ====
		+
		+	Show all currently running jobs:
		+
		+	<syntaxhighlight lang="bash">
		+	squeue
		+	</syntaxhighlight>
		+
		+	Show your own running jobs:
		+
		+	<syntaxhighlight lang="bash">
		+	squeue --me
		+	</syntaxhighlight>
		+
		+	Show the available partitions on the cluster:
		+
		+	<syntaxhighlight lang="bash">
		+	sinfo
		+	</syntaxhighlight>
		+
		+	You'll see something like this:
		+
		+	<syntaxhighlight lang="bash">
		+	$ sinfo
		+	PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
		+	compute* up infinite 8 idle compute-permanent-node-[68,285,493,580,625-626,749,801]
		+	</syntaxhighlight>
		+
		+	This means:
		+
		+	* There is one compute node type, called <code>compute</code>
		+	* There are 8 nodes of that type, all currently in <code>idle</code> state
		+	* The node names are things like <code>compute-permanent-node-68</code>

	==== Reserving a GPU ====		==== Reserving a GPU ====

← Older revision		Revision as of 23:51, 24 May 2024
Line 52:		Line 52:

	The Andromeda cluster is a different cluster which uses Slurm for job management. Authentication is different from the Lambda cluster - Ben will provide instructions directly.		The Andromeda cluster is a different cluster which uses Slurm for job management. Authentication is different from the Lambda cluster - Ben will provide instructions directly.
		+
		+	Don't do anything computationally expensive on the main node or you will crash it for everyone. Instead, when you need to run some experiments, reserve a GPU (see below).

	==== Reserving a GPU ====		==== Reserving a GPU ====

@@ Line 85: / Line 85: @@
 You might get an error like this: <code>groups: cannot find name for group ID 1506</code>. But things should still run fine. Check with <code>nvidia-smi</code>.
-=== Useful Commands ===
+==== Useful Commands ====
 Set a node state back to normal:

@@ Line 4: / Line 4: @@
 To get onboarded, you should send us the public key that you want to use and maybe your preferred username.
 After being onboarded, you should receive the following information:
@@ Line 47: / Line 49: @@
 * You should avoid storing data files and model checkpoints to your root directory. Instead, use the <code>/ephemeral</code> directory. Your home directory should come with a symlink to a subdirectory which you have write access to.
-== Reserving a GPU ==
+=== Andromeda Cluster ===
 Here is a script you can use for getting an interactive node through Slurm.

← Older revision		Revision as of 05:08, 7 May 2024
Line 94:		Line 94:

	You might get an error like this: <code>groups: cannot find name for group ID 1506</code>. But things should still run fine. Check with <code>nvidia-smi</code>.		You might get an error like this: <code>groups: cannot find name for group ID 1506</code>. But things should still run fine. Check with <code>nvidia-smi</code>.
		+
		+	=== Useful Commands ===
		+
		+	Set a node state back to normal:
		+
		+	<syntaxhighlight lang="bash">
		+	sudo scontrol update nodename='nodename' state=resume
		+	</syntaxhighlight>

	[[Category:K-Scale]]		[[Category:K-Scale]]