-
Notifications
You must be signed in to change notification settings - Fork 377
feat(gsoc): spack upstreaming of key4HEP package recipes #1818
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
2c6f8b1
feat(gsoc): spack debuggable installs
wdconinc a13169e
fix: UM -> umanitoba
wdconinc 5c99f61
feat(gsoc): spack upstreaming of key4hep package recipes
wdconinc 456a7c7
fix: introduce HEP abbreviation
wdconinc acbd53f
fix: MacOS -> macOS
wdconinc 1366f89
fix: add key4HEP links
wdconinc 25ae1e9
add me as mentor
vvolkl File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| --- | ||
| title: "University of Manitoba" | ||
| author: "Wouter Deconinck" | ||
| layout: default | ||
| organization: umanitoba | ||
| logo: UManitoba-logo.png | ||
| description: | | ||
| The University of Manitoba is a Canadian public research university in the province of Manitoba, | ||
| located on original lands of Anishinaabeg, Cree, Oji-Cree, Dakota, and Dene peoples, and on the | ||
| homeland of the Métis Nation. | ||
| --- | ||
|
|
||
| {% include gsoc_proposal.ext %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| project: Spack | ||
| layout: default | ||
| logo: spack-logo-220-LLNL.png | ||
| description: | | ||
| [Spack](https://spack.io) is a flexible package manager designed to support multiple versions, configurations, platforms, and compilers. It is widely used in high-performance computing (HPC) environments to manage complex software stacks. | ||
| --- | ||
|
|
||
| {% include gsoc_project.ext %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| --- | ||
| title: Debuggable Installations for Spack Packages | ||
| layout: gsoc_proposal | ||
| project: Spack | ||
| year: 2026 | ||
| organization: | ||
| - umanitoba | ||
| difficulty: medium | ||
| duration: 175 | ||
| mentor_avail: June-October | ||
| project_mentors: | ||
| - email: wouter.deconinck@umanitoba.ca | ||
| first_name: Wouter | ||
| last_name: Deconinck | ||
| organization: umanitoba | ||
| is_preferred_contact: yes | ||
| --- | ||
|
|
||
| ## Description | ||
|
|
||
| Spack is a flexible package manager widely used in high-performance computing (HPC) to manage complex software stacks. It is commonly used in scientific computing environments, including particle physics research. For example, the key4HEP project uses Spack to manage software dependencies for high-energy physics applications. | ||
|
|
||
| Spack builds packages from scratch in a staging directory, which is then installed into a Spack-managed prefix. However, this approach can make debugging installation issues challenging, as the build artifacts are not easily accessible after installation. Symbols in the installed binaries may not correspond to the original source files, complicating debugging efforts. | ||
|
|
||
| There are some workarounds to ensuring debuggable installs in Spack, such as using `spack dev-build` or `spack develop`, but these methods have limitations and do not fully address the issue in a streamlined manner. This project will focus on ensuring that the source trees used for compilation can be installed in the prefix alongside the built binaries, allowing for easier debugging and symbol resolution. This project will have to ensure that temporary build products are still cleaned up properly to avoid bloating the installation prefix. | ||
|
|
||
| ## Task Ideas | ||
|
|
||
| - Assess the limitations of the current approaches for installing source trees alongside built binaries in Spack | ||
| - Develop a robust solution to enable debuggable installs for Spack packages with out-of-tree build systems | ||
| - Test the implementation with a variety of packages and build systems, with a focus on high-energy physics software | ||
|
|
||
| As a stretch goal, in case of rapid progress, we may consider a next step which involves installing debug symbols in a separate location for access with `debuginfod`. This would allow installed binaries to be stripped of debug symbols, reducing their size and increasing performance, while still providing access to the symbols when needed for debugging. | ||
|
|
||
| ## Expected Results and Milestones | ||
|
|
||
| - Familiarization with Spack's build and install processes | ||
| - Summarization of current limitations and potential solutions | ||
| - Design of a solution for debuggable installs | ||
| - Analyze design for anticipated challenges | ||
| - Implementation of the solution | ||
| - Testing and validation with various packages | ||
| - Documentation and integration into Spack | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python programming skills | ||
| - Packaging and build system knowledge (in particular CMake) | ||
| - Interest in scientific software stacks and high-energy physics | ||
|
|
||
| ## AI Policy | ||
|
|
||
| AI assistance is allowed for this contribution. The applicant takes full responsibility for all code and results, disclosing AI use for non-routine tasks (algorithm design, architecture, complex problem-solving). Routine tasks (grammar, formatting, style) do not require disclosure. | ||
|
|
||
| ## How to Apply | ||
|
|
||
| Email mentors with a brief background and interest in scientific software stacks and high-energy physics. Please include "gsoc26" in the subject line. Mentors will provide an evaluation task after submission. | ||
|
|
||
| ## Resources | ||
|
|
||
| - [Spack](https://spack.io/) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| --- | ||
| title: Expanding User-Facing High-Energy Physics Spack Packages | ||
| layout: gsoc_proposal | ||
| project: Spack | ||
| year: 2026 | ||
| organization: | ||
| - umanitoba | ||
| - CERN | ||
| difficulty: medium | ||
| duration: 175 | ||
| mentor_avail: June-October | ||
| project_mentors: | ||
| - email: wouter.deconinck@umanitoba.ca | ||
| first_name: Wouter | ||
| last_name: Deconinck | ||
| organization: umanitoba | ||
| is_preferred_contact: yes | ||
| - email: valentin.volkl@cern.ch | ||
| first_name: Valentin | ||
| last_name: Volkl | ||
| organization: CERN | ||
| --- | ||
|
|
||
| ## Description | ||
|
|
||
| Spack is a flexible package manager widely used in high-performance computing (HPC) to manage complex software stacks. It is commonly used in scientific computing environments, including particle physics research. For example, the key4HEP project uses Spack to manage complex software dependencies for high-energy physics (HEP) applications. | ||
|
|
||
| The key4HEP project has for several years maintained and curated an extensive selection of high-energy physics software packages in Spack, through the third-party `key4hep-spack` package repository. This selection of packages has formed the basis of large deployments of the key4HEP software stack on CVMFS. | ||
|
|
||
| However, changing directions for the deployment strategy of the key4HEP stack are affecting the sustainability of central support for the package collection. The package collection will therefore benefit from transitioning to an end-user community-support model by upstreaming the package recipes into the main Spack repository. This transition process is expected to play out over the course of several months, during the GSoC period. | ||
|
|
||
| ## Task Ideas | ||
|
|
||
| - Enumerate and categorize the existing key4HEP Spack packages by usage and maintenance status (latest release, actively maintained, breadth of user-base, etc.) | ||
| - Identify discrepancies between key4HEP packaging practices and current Spack best practices | ||
| - Rank the list of packages to be upstreamed based on the previous analyses | ||
| - Upstream the highest priority packages into Spack, following Spack best practices | ||
| - Integrate new packages into Spack's continuous integration and testing infrastructure (HEP stack) | ||
|
|
||
| As a stretch goal, in case of rapid progress, we may consider as a next step the expansion of the continuous integration and testing infrastructure to include macOS as a target platform for the HEP stack. This would allow Spack users on macOS to more confidently rely on the functionality of high-energy physics packages. | ||
|
|
||
| ## Expected Results and Milestones | ||
|
|
||
| - Familiarization with Spack's packaging and contribution processes | ||
| - Summarization of current package recipes in the key4HEP Spack repository | ||
| - Identification of plan of work for upstreaming packages | ||
| - Upstreaming of individual (or cohesive groups of) packages in multiple pull requests | ||
| - Testing and validation of the upstreamed packages | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python programming skills | ||
| - Packaging and build system knowledge | ||
| - Some experience with continuous integration on GitLab | ||
| - Interest in scientific software stacks and high-energy physics | ||
|
|
||
| ## AI Policy | ||
|
|
||
| AI assistance is allowed for this contribution. The applicant takes full responsibility for all code and results, disclosing AI use for non-routine tasks (algorithm design, architecture, complex problem-solving). Routine tasks (grammar, formatting, style) do not require disclosure. | ||
|
|
||
| ## How to Apply | ||
|
|
||
| Email mentors with a brief background and interest in scientific software stacks and high-energy physics. Please include "gsoc26" in the subject line. Mentors will provide an evaluation task after submission. | ||
|
|
||
| ## Resources | ||
|
|
||
| - [Spack](https://spack.io/) | ||
| - [Key4HEP documentation](https://key4hep.github.io/key4hep-doc/) | ||
| - [key4hep-spack repository](https://github.com/key4hep/key4hep-spack) | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.