- grex-pi - University of Manitoba Mailing Lists

Announcing 2025 Spring HPC and Cloud computing workshop at UM! May 21-23
by Grigory Shamov 01 May '25

01 May '25

Hi, (You are receiving this email because you are an active Principal Investigator at UMan, with access to the Alliance and Grex HPC and Cloud computing resources through the CCDB portal.) I want to announce our upcoming HPC and Cloud workshop! This year, after many years of online-only workshops, our group is excited to offer it in-person, and with hands-on exercises. In case you‘d have anyone in your research group interested, please find the Workshop ad and RSVP link below! ========= Learn to Use Canadian High-Performance Computing (HPC) and Cloud Resources Spring 2025 Workshop – Free Registration Join us for our free, hands-on Spring 2025 workshop, open to all researchers from Manitoba institutions! Held May 21–23, 2025 at the University of Manitoba, this in-person event will introduce participants to Linux, SLURM, Open OnDemand, Globus, and OpenStack. Requirements: * A laptop is required for hands-on exercises. * An active CCDB account is optional. Audience: The workshop is designed for beginner and intermediate users. Content is spread across three days, and attendees are encouraged to RSVP through Eventbrite: Eventbrite Registration Link<https://www.eventbrite.ca/e/spring-2025-manitoba-high-performance-computing…> . (https://shorturl.at/8vnoM). Participants may register for and attend individual days or talks. However, attending sessions in order is recommended, as later material builds on earlier sessions. Workshop Schedule: Day 1: May 21, 2025 * Using the Linux BASH shell * Data transfer with SSH and Globus * Working with HPC software and LMod Day 2: May 22, 2025 * Using the Open OnDemand web portal for HPC * Running jobs with the SLURM scheduler * Introduction to using OpenStack Cloud Day 3: May 23, 2025 * Using containers in HPC: Singularity and Podman * Using Python for AI/ML on HPC systems Thank you for considering attending our Spring Workshop! ========= Thank you very much! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Last call for local HPC (Grex) allocations!
by Grigory Shamov 26 Nov '24

26 Nov '24

Hi, This is a reminder that of 2024-25 call for Grex resource allocations is due in two days, by 4:30 pm on November 28, 2024. Thank you if you have already applied! If your group needs local HPC compute and storage resources exceeding the "default" allocations, and you have not yet applied, applying may be a good idea. Thank you! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Recall: Re: Time to renew Grex HPC Allocations! Local RAC call
by Chuang Deng 14 Nov '24

14 Nov '24

Chuang.Deng(a)umanitoba.ca would like to recall the message, "Re: Time to renew Grex HPC Allocations! Local RAC call ".

1 0

Time to renew Grex HPC Allocations! Local RAC call
by Advanced Research Computing 08 Nov '24

08 Nov '24

Dear Grex / HPC Users, Thank you all for using our HPC resources! Another year passed since our last call. Our local HPC system, Grex, just got a major hardware and datacentre renewal in 2024. It is time to renew the local resource allocations for 2025 Please find the Local Resource Allocation Call 2025 forms and conditions attached Your HPC Team. ==== Local HPC (Grex) resource allocation call description Overview of the HPC resources Grex is the High-Performance Computing (HPC) system at the University of Manitoba. In addition to its compute capacity, which includes Intel Skylake compute nodes purchased in 2020 and 2021, over 5,000 new AMD Genoa compute cores were added as part of the HPCC SISF 2023 project. There is also a large, partially community-funded project storage on Grex. This call is for renewing and updating the allocation of resources on the Grex HPC system, specifically CPU time (in Core Years, CY) and storage (in TBs). The generally available hardware is as follows: * 30 of 192-core AMD Genoa Zen4 9654 nodes, each with 192 cores and 4 GB or 8 GB of memory per core. * 12 nodes with 40-core Intel Cascade Lake CPUs and 42 nodes with 52-core Intel Cascade Lake CPUs, each with 3GB or 8 GB of memory per core. * Two nodes with 4x NVIDIA V100 GPUs, Intel 5218 Cascade Lake CPUs, and 6 GB memory per core. The total number of allocatable CPU cores is 8,580. * General GPU nodes are available on a first-come, first-served basis. There are no dedicated GPU allocations, as the number of GPUs (eight V100s) is limited. Please indicate if you anticipate needing GPUs in your proposal. * Contributed resources: A significant number of contributed GPUs (34 total, comprising A30 and V100 models) and CPU cores (1,800 AMD Genoa) are available. These resources are owned by contributors and are not allocatable but can be used opportunistically when not in use by their owners. The Project storage, allocatable per research group, has a total capacity of 2 PB, with 1.7 PB designated for the contributing Faculties of Agriculture, Engineering, and Science. The /project space is allocated by default between 5 and 40 TB per group. Larger storage allocations are possible through this call, especially for research groups from contributing Faculties. Research groups from these Faculties will be given priority for /project storage according to their contributions. There is no charge for using Grex for University of Manitoba academic PIs. The request for ARC resources must come from a UManitoba PI, and the unit/owner of a resource allocation is the PI’s research group. Categories of Resource Requests There are two categories of resource requests: 1. Default allocations or “Rapid Access Service” (RAS) is limited to under 60 core-years, 4 GB memory per core, and 5 to 40 TB of Project storage per research group. RAS requests do not require a full proposal. For information and access, please email arc(a)umanitoba.ca. 2. Resource Allocation Call (RAC): Users who need more core-years or storage than the RAS limits above for 2025 must request resources via an RAC proposal as part of this call. Proposals must be submitted in the format described below. ________________________________ Proposal Format and Submission Timelines Please use the attached 2024 RAC Request Template for your proposal. You may remove all explanations (text in italics) if desired. This template is intended as a guide. Expected lengths for each section are included in the template, but your proposal may use more, or less space, depending on the size and complexity of the request. Proposals for the use of Grex resources will be reviewed by the Advanced Research Computing (ARC) Committee and Grex technical staff to ensure resources are used efficiently. Based on available Grex resources and the total request volume, requested allocations may be scaled to fit within the system’s capacity. Proposals must be sent by email to ARC(a)umanitoba.ca<mailto:ARC@umanitoba.ca> by 4:30 pm on November 28, 2024.

1 0

Updates on the Multi Factor Authentication
by Grigory Shamov 09 Jan '24

09 Jan '24

Dear Grex/HPC User, As you might have heard, the University is switching from Duo to Microsoft Entra MFA solution. The Digital Research Alliance of Canada, and our UManitoba HPC system, Grex, continue Cisco Duo as the multifactor authentication (MFA) second factor. As of today (January 9, 2024) enrollment in Duo is required to access Grex, and optional but highly recommended to access the Alliance systems such as Cedar, Graham, and Narval. On Grex and Alliance systems, the following second factors are enabled: * Duo smartphone app (Android and iOS) * Yubico Yubikey cybersecurity USB key device * 10 one-time codes (recommended as a backup 2FA in addition to the primary device). The one-time codes require another 2nd factor registered first. Enrollment into the Alliance Duo is through CCDB: https://ccdb.alliancecan.ca<https://ccdb.alliancecan.ca/>. A successful Enrollment enables the MFA requirement on every SSH login on both Grex and Alliance systems like Cedar, Graham, Narval, or Niagara. The following Grex Documentation page explains the process of the device enrollment with screenshots: https://um-grex.github.io/grex-docs/connecting/mfa/ Our staff will be available for support in case you have difficulty enrolling or using MFA. If you have any questions, please let us know (by an email to support(a)tech.alliancecan.ca<mailto:support@tech.alliancecan.ca>, mentioning “Grex” in the subject line). Thank you for your attention to this message! -- Your Grex HPC team.

1 0

Notice for initial MFA Enforcement on Grex - Dec 2023
by Grigory Shamov 29 Nov '23

29 Nov '23

Dear Grex/HPC User, The Digital Research Alliance of Canada, and our UManitoba HPC system, Grex, are proceeding with the ongoing implementation of a multifactor authentication (MFA) system. MFA adds an additional layer of security to the traditional password-based (or SSH keys based) authentication by requiring a second factor, known as "something you have." Grex is using the Cisco Duo instance from the Alliance as the second factor authentication. We have successfully completed the first phase of MFA testing for staff and early adopter users. As of now, every Alliance and Grex user has an option to register a device in CCDB. As our next stage in MFA adoption, we are going to temporarily enforce the use of MFA for all GREX users between Dec 04 and 11, 2023 . All authentication attempts to Grex during this week will require the second factor. We would like to make MFA mandatory for all GREX users in January 2024. Note that the second factors used by the Alliance are not entirely the same as those used by the University of Manitoba for platforms such as UM Intranet and Exchange. On Grex and Alliance systems, the following factors are enabled: ● Duo smartphone app (Android and iOS) ● Yubico Yubikey cybersecurity USB key device ● 10 one-time codes (recommended as a backup 2FA in addition to the primary device) Enrollment into the Alliance Duo is through CCDB: https://ccdb.alliancecan.ca<https://ccdb.alliancecan.ca/> . A successful Enrollment enables the MFA requirement on every SSH login on both Grex and Alliance systems like Cedar, Graham, Narval or Niagara. The following Grex Documentation page explains the process of the device enrollment with screenshots: https://um-grex.github.io/grex-docs/connecting/mfa/ Our staff will be available for support in case you have a difficulty to enroll or use MFA.. If you have any questions, please let us know (by an email to support(a)tech.alliancecan.ca<mailto:support@tech.alliancecan.ca> , mentioning “Grex” in the subject line). Thank you for your patience and thank you for your attention to this message! -- Your Grex HPC team.

1 0

Reminder: Grex local RAC deadline comes Nov 22 (this Wednesday )
by Grigory Shamov 20 Nov '23

20 Nov '23

Dear Grex PI, In case your group was interested in applying to our local Resource allocation call: the deadline for it comes this Wednesday, November 22. Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada From: Advanced Research Computing <ARC(a)umanitoba.ca> Date: Wednesday, November 1, 2023 at 9:35 AM To: "grex-pi(a)lists.umanitoba.ca" <grex-pi(a)lists.umanitoba.ca> Subject: [grex-pi] Time to renew Grex HPC resource allocations ! Local RAC call starts today. Dear Grex / HPC Users, Thank you all for using our HPC resources! Another year passed since our last call. It is time to renew the local resource allocations for 2023/2024. Please find the Local Resource Allocation Call 2023/24 forms and conditions attached Your HPC Team. =========== Overview Grex is a High-Performance Computing system of the University of Manitoba. Its current compute capacity consists of three parts: legacy compute nodes, new compute nodes purchased in 2020 and 2021, and researcher-contributed nodes. There is also now partially community-funded storage on Grex. This call is for renewing and updating the allocation of the resources of the Grex HPC system (CPU time in Core Years (CY) and storage in TBs. The generally available hardware is as follows: New compute nodes added in 2020 and 2021 * two 4x NVIDIA V100 GPU nodes, Intel 5218 Cascade Lake, 192GB RAM * one 16x NVIDIA V100 HGX-2 node 48 core Intel CPU, 1.5TB of RAM * 12 of 40 core Intel CascadeLake 6248 2.5GHz compute nodes, 384GB RAM * 42 of 52 core Intel CascadeLake 6230R 2.1GHz compute nodes, 192GB RAM Legacy compute nodes from 2011 * About 200 remaining original compute nodes, 12 cores 2.67GHz Nehalem processors each with 48GB RAM The Project storage, allocatable per research group, has total space of 2 PB, with 1.7TB targeted for the contributing Faculties of Agriculture, Engineering, and Science. * The /project space is allocated by default 5 top 20 TB per group, with larger allocations possible through this Call. Research groups from the contributor Faculties will be guaranteed a preference in the allocation of the /project storage quota according to their contributions. In this round of resource allocations, we call for proposals to allocate the current compute nodes (a total of 2716 CPU cores in 55 CascadeLake nodes) and current 2PB Project storage. * General GP GPU nodes are available on a first-come, first-served basis. Please indicate if you might need GPUs in your proposal. * The legacy 316 compute nodes will likely be decommissioned in the near future. Until that, they are available on a first-come, first-served basis. Please indicate if you need these nodes in your proposal. * Contributed GPU and CPU nodes are not allocatable, but they are available for opportunistic, preemptible use when not used by their contributors. Proposals for the use of Grex resources will be reviewed by the Advanced Research Computing (ARC) Committee. They will also be reviewed by Grex technical staff to ensure that Grex resources will be used appropriately and efficiently. Categories of Resource Requests The request for ARC resources must come from a UManitoba PI. There are two (2) categories of resource requests: 1. Rapid Access Service (RAS) is limited to less than 50 core-years, 2 GB memory/core, and 5 to 20 TB of Project storage per research group. RAS requests do not require a proposal For information and access please email arc(a)umanitoba.ca<mailto:arc@umanitoba.ca> 2. Resource Allocation Call (RAC). For the annual allocation of Grex resources, we ask users who need more core years or storage than the RAS limits above to request the resources via an RAC proposal. The proposals must be submitted in the format described below. Proposal Format Please use the attached 2023 RAC Request Template for your proposal. You may remove all explanations (text in italics) if desired. This template attached is intended as a guide. Please remove any section or sub-section that does not apply to your application. Maximum lengths for each section are included in the template, but your proposal may use (much) less space depending on the size and complexity of the request. Proposals must be sent by email to ARC(a)umanitoba.ca<mailto:ARC@umanitoba.ca> by 4:30 pm November 22, 2023.

1 0

Time to renew Grex HPC resource allocations ! Local RAC call starts today.
by Advanced Research Computing 01 Nov '23

01 Nov '23

Dear Grex / HPC Users, Thank you all for using our HPC resources! Another year passed since our last call. It is time to renew the local resource allocations for 2023/2024. Please find the Local Resource Allocation Call 2023/24 forms and conditions attached Your HPC Team. =========== Overview Grex is a High-Performance Computing system of the University of Manitoba. Its current compute capacity consists of three parts: legacy compute nodes, new compute nodes purchased in 2020 and 2021, and researcher-contributed nodes. There is also now partially community-funded storage on Grex. This call is for renewing and updating the allocation of the resources of the Grex HPC system (CPU time in Core Years (CY) and storage in TBs. The generally available hardware is as follows: New compute nodes added in 2020 and 2021 * two 4x NVIDIA V100 GPU nodes, Intel 5218 Cascade Lake, 192GB RAM * one 16x NVIDIA V100 HGX-2 node 48 core Intel CPU, 1.5TB of RAM * 12 of 40 core Intel CascadeLake 6248 2.5GHz compute nodes, 384GB RAM * 42 of 52 core Intel CascadeLake 6230R 2.1GHz compute nodes, 192GB RAM Legacy compute nodes from 2011 * About 200 remaining original compute nodes, 12 cores 2.67GHz Nehalem processors each with 48GB RAM The Project storage, allocatable per research group, has total space of 2 PB, with 1.7TB targeted for the contributing Faculties of Agriculture, Engineering, and Science. * The /project space is allocated by default 5 top 20 TB per group, with larger allocations possible through this Call. Research groups from the contributor Faculties will be guaranteed a preference in the allocation of the /project storage quota according to their contributions. In this round of resource allocations, we call for proposals to allocate the current compute nodes (a total of 2716 CPU cores in 55 CascadeLake nodes) and current 2PB Project storage. * General GP GPU nodes are available on a first-come, first-served basis. Please indicate if you might need GPUs in your proposal. * The legacy 316 compute nodes will likely be decommissioned in the near future. Until that, they are available on a first-come, first-served basis. Please indicate if you need these nodes in your proposal. * Contributed GPU and CPU nodes are not allocatable, but they are available for opportunistic, preemptible use when not used by their contributors. Proposals for the use of Grex resources will be reviewed by the Advanced Research Computing (ARC) Committee. They will also be reviewed by Grex technical staff to ensure that Grex resources will be used appropriately and efficiently. Categories of Resource Requests The request for ARC resources must come from a UManitoba PI. There are two (2) categories of resource requests: 1. Rapid Access Service (RAS) is limited to less than 50 core-years, 2 GB memory/core, and 5 to 20 TB of Project storage per research group. RAS requests do not require a proposal For information and access please email arc(a)umanitoba.ca<mailto:arc@umanitoba.ca> 2. Resource Allocation Call (RAC). For the annual allocation of Grex resources, we ask users who need more core years or storage than the RAS limits above to request the resources via an RAC proposal. The proposals must be submitted in the format described below. Proposal Format Please use the attached 2023 RAC Request Template for your proposal. You may remove all explanations (text in italics) if desired. This template attached is intended as a guide. Please remove any section or sub-section that does not apply to your application. Maximum lengths for each section are included in the template, but your proposal may use (much) less space depending on the size and complexity of the request. Proposals must be sent by email to ARC(a)umanitoba.ca<mailto:ARC@umanitoba.ca> by 4:30 pm November 22, 2023.

1 0

One week left to register UManitoba Autumn 2023 - High-Performance Computing workshop Nov 6, 7
by Grigory Shamov 31 Oct '23

31 Oct '23

Dear UM Faculty, We would like to issue a second announcement for our semi-annual HPC workshop, that your group members may be interested in. ====== “Updates about High-Performance Computing resources, and beginner HPC user workshop for UManitoba users” Date: Nov 6-7, 2023 Time: 10:30 AM - 2:30 PM, Winnipeg Time Place: Online with UManitoba Zoom (meeting link provided after registration) Join us for the Research Computing online Workshop, where we'll provide the latest news about, and insights on how to effectively use the National and local High-Performance Computing. This workshop will consist of two sessions on Monday, Nov 6th, Tuesday, Nov 7. These two sessions will be followed by another session of two hours on Wednesday Nov. 8th where we can answer all your questions and/or discuss any other topic that may help you to start using the resources efficiently. It's open to all UManitoba users and will be held online via Zoom. To attend, please RSVP using the Zoom link provided below. https://umanitoba.zoom.us/meeting/register/u5UsdO2qpzIjG9NgXwjl7J-Xqk1mFMhh… Our team is excited to bring you the programme outlined below. Thank you for your interest, and we look forward to seeing you at the workshop! Sincerely, Your local HPC team. Programme: November 6: 1. Introduction, available resources, RAC process (30 min) 2. Getting an account, how to connect, using Duo MFA (40 min) 3. Basics of Linux Shell (40 min) 4. Running jobs with SLURM (60 min) 5. Using Grex with Open OnDemand portal (40 min) November 7: 1. Using HPC software stacks (60 min) 2. Using containers in HPC ( 40 min) 3. Basics of using OpenStack cloud (60 min) 4. Advanced HPC usage, Getting most of HPC systems(40 min) November 8: Online Office hours/ QA (10:30 AM until Noon) ======= (You are receiving the message because at some point you have registered in CCDB and you are UManitoba Faculty whose group may have used Grex HPC system. If you do not want to receive the emails about Grex or HPC at Umanitoba, please let us know. Thanks!) -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

1 0

Announcement of Duo MFA for Grex and DRAC
by Grigory Shamov 20 Sep '23

20 Sep '23

Introducing Multi Factor Authentication implementation on Grex and DRAC systems Dear PI, As you may know, both Digital Research Alliance and our UManitoba HPC system, Grex, are proceeding with the ongoing implementation of a multifactor authentication (MFA) system. MFA adds an additional layer of security to the traditional password-based and SSH keys authentication by requiring a second factor, known as "something you have". The Cisco Duo product was chosen as the second factor authentication. Grex is using the Duo instance from the Alliance. We have successfully completed the first phase of MFA testing for staff and early adopter users. Some of you might have been invited to enroll from the Alliance side already. Now, we would like to extend the invitation to every Grex user to enroll in the new MFA system! Please let us know if you would like to get enrolled. Note that the second factors used by the Alliance are not entirely the same as those used by the University of Manitoba for platforms such as UM Intranet and Exchange. On Grex and Alliance systems, the following factors are enabled: * Duo smartphone app (Android and iOS) * Yubico Yubikey cybersecurity USB key device * One-time codes (recommended as a backup MFA) Enrollment into the Alliance Duo is through CCDB: https://ccdb.alliancecan.ca<https://ccdb.alliancecan.ca/> . Enrollment enables the MFA requirement on every SSH login on both Grex and Alliance systems like Cedar, Graham, Beluga, Narval or Niagara. After we enable MFA for your account, you must follow these steps to complete the enrollment: * login into CCDB with your credentials * from the top menu choose "My Account" ⇒ "Multifactor Authentication Management" * register a new device * OPTIONAL BUT RECOMMENDED, at the bottom of the same page, use the "Generate 10 codes" button to generate 10 rescue codes (you must print/save these codes in a safe location and never disclose them to anyone) The following Grex Documentation page explains it better with screenshots: https://um-grex.github.io/grex-docs/connecting/mfa/ You can also find additional information on the Alliance official wiki: https://docs.alliancecan.ca/wiki/Multifactor_authentication As of now, the enrollment is voluntary: we kindly ask you to let us know (by an email to support(a)tech.alliancecan.ca<mailto:support@tech.alliancecan.ca> ) if you / your group members would like to enroll, and we will then enable the MFA option under your CCDB account. Eventually enrollment would become mandatory. IMPORTANT NOTICES We plan for MFA to become mandatory for all Grex users in October 2023. We appreciate your attention to this matter and encourage you to enroll at your earliest convenience to ensure a smooth transition. Should you have any questions or require further assistance, please don't hesitate to reach out to your local HPC team! However, If some of your work relies on unattended connections or automations that may be disrupted by the MFA, we ask that you do not enroll yet, and instead ask that you contact us for technical support, so that we can work with you on a solution. Also, please note that enrolling in MFA for Grex, makes MFA active and enforced also on the National systems as well, and vice versa. If this does not work for you, please contact us (by an email to support(a)tech.alliancecan.ca<mailto:support@tech.alliancecan.ca> ).

1 0