GPU Assembly of LOR DG Preconditioner #4834

Toni-ko · 2025-04-28T22:20:56Z

This pr adds a batched gpu assembly of the lor dg preconditioner. New preconditioner is used in the lor solvers miniapp. A unit test is added to compare with legacy implementation.

PR	Author	Editor	Reviewers	Assignment	Approval	Merge
#4834	@Toni-ko	@tzanio	@pazner + @tzanio + @YohannDudouit	4/30/25	4/30/25	⌛due 5/21/25

PR Checklist

Code builds.
Code passes make style.
Update CHANGELOG:
- Is this a new feature users need to be aware of? New or updated example or miniapp?
- Does it make sense to create a new section in the CHANGELOG to group with other related features?
Update INSTALL:
- Had a new optional library been added? If so, what range of versions of this library are required? (Make sure the external library is compatible with our BSD license, e.g. it is not licensed under GPL!)
- Have the version ranges for any required or optional libraries changed?
- Does make or cmake have a new target?
- Did the requirements or the installation process change? (rare)
Update continuous integration server configurations if necessary (e.g. with new version requirements for each of MFEM's dependencies)
- .github
- .appveyor.yml
Update .gitignore:
- Check if make distclean; git status shows any files that were generated from the source by the project (not an IDE) but we don't want to track in the repository.
- Add new patterns (just for the new files above) and re-run the above test.
New examples:
- All sample runs at the top of the example source file work.
- Update examples/makefile:
  - Add the example code to the appropriate SEQ_EXAMPLES and PAR_EXAMPLES variables.
  - Add any files generated by it to the clean target.
  - Add the example binary and any files generated by it to the top-level .gitignore file.
- Update examples/CMakeLists.txt:
  - Add the example code to the ALL_EXE_SRCS variable.
  - Make sure THIS_TEST_OPTIONS is set correctly for the new example.
- List the new example in doc/CodeDocumentation.dox.
- If new examples directory (e.g.examples/pumi), list it in doc/CodeDocumentation.conf.in
- Companion pull request for documentation in mfem/web repo:
  - Update or add example-specific documentation, see e.g. the src/examples.md.
  - Add the description, labels and screenshots in src/examples.md and src/img.
  - In examples.md, list the example under the appropriate categories, add new categories if necessary.
  - Add a short description of the example in the "Extensive Examples" section of features.md.
New miniapps:
- All sample runs at the top of the miniapp source file work.
- Update top-level makefile and makefile in corresponding miniapp directory.
- Add the miniapp binary and any files generated by it to the top-level .gitignore file.
- Update CMake build system:
  - Update the CMakeLists.txt file in the miniapps directory, if the new miniapp is in a new directory.
  - Add/update the CMakeLists.txt file in the new miniapp directory.
  - Consider adding a new test for the new miniapp.
- List the new miniapp in doc/CodeDocumentation.dox
- If new miniapps directory (e.g.miniapps/nurbs), add it to MINIAPP_SUBDIRS in the makefile.
- If new miniapps directory (e.g.miniapps/nurbs), list it in doc/CodeDocumentation.conf.in
- Companion pull request for documentation in mfem/web repo:
  - Update or add miniapp-specific documentation, see e.g. the src/meshing.md and src/electromagnetics.md files.
  - Add the description, labels and screenshots in src/examples.md and src/img.
  - The miniapps go at the end of the page, and are usually listed only under a specific "Application (PDE)" category.
  - Add a short description of the miniapp in the "Extensive Examples" section of features.md.
New capability:
- All new public, protected, and private classes, methods, data members, and functions have full Doxygen-style documentation in source comments. Documentation should include descriptions of member data, function arguments and return values, template parameters, and prerequisites for calling new functions.
- Pointer arguments and return values must specify whether ownership is being transferred or lent with the call.
- Any new functions should include descriptions of their intended use e.g. for internal use only, user-facing, etc., along with references to example code whenever possible/appropriate.
- Consider adding new sample runs in existing examples to highlight the new capability.
- Consider saving cool simulation pictures with the new capability in the Confluence gallery (LLNL only) or submitting them, via pull request, to the gallery section of the mfem/web repo.
- If this is a major new feature, consider mentioning it in the short summary inside README (rare).
- List major new classes in doc/CodeDocumentation.dox (rare).
Update this checklist, if the new pull request affects it.
Run make unittest to make sure all unit tests pass.
Run the tests in tests/scripts.
(LLNL only) After merging:
- Update internal tests to include the new features.

…r sx, sy, and sz are changed).

…here nx, ny, nz, sx, sy, and/or sz are changed.

Add commented-out code for testing 3D face permutations

(Test still fails)

Resolve error: lambda capture 'this' is not used

tzanio · 2025-04-30T15:51:33Z

This PR is now under review (see the table in the PR description). To help with the review process, please do not force push to the branch.

Copilot

Pull Request Overview

This PR introduces GPU‐accelerated assembly for the LOR DG preconditioner used in the LOR solvers miniapp. Key changes include the addition of batched GPU assembly kernels and associated tests, updated mesh permutation utilities and integration rule handling, and modifications to penalty parameter scaling in the DG discretizations.

Reviewed Changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/linalg/test_same_matrices.hpp	New utility to compare matrices for unit tests.
tests/unit/fem/test_lor_dg.cpp	Unit tests updated for LOR DG preconditioner using batched assembly.
tests/unit/fem/test_lor_batched.cpp	Refactored tests to remove redundant implementations for DG.
tests/unit/fem/test_face_permutation.cpp	Mesh permutation tests now use new MeshOrientation helpers.
tests/unit/fem/make_permuted_mesh.{hpp,cpp}	Refactored mesh permutation functionality for clarity and consistency.
miniapps/solvers/plor_solvers.cpp & lor_solvers.cpp	Updated assembly-level conditions and changed penalty scaling (kappa) to 10×(order+1)².
fem/pfespace.hpp	Added accessor for the face neighbor global dof map.
fem/lor/lor_dg_impl.hpp & lor_dg.hpp	Introduced GPU assembly implementation for DG preconditioning with new methods for face info and boundary penalty factor computation.
fem/lor/lor_batched.{hpp,cpp}	Extended batched LOR assembly support for DG spaces with customized CSR conversion routines.
fem/lininteg.hpp	Minor documentation update for DGDiffusionIntegrator error terms.
fem/fe/face_map_utils.hpp	Added new inline function to compute face-to-volume index mapping.
fem/bilininteg.hpp	Exposed DG penalty parameter via a new getter in DGDiffusionIntegrator.

Files not reviewed (2)

fem/CMakeLists.txt: Language not supported
tests/unit/CMakeLists.txt: Language not supported

miniapps/solvers/plor_solvers.cpp

tests/unit/fem/test_lor_dg.cpp

fem/lor/lor_batched.cpp

Co-authored-by: Will Pazner <11493037+pazner@users.noreply.github.com>

pazner

Thanks @Toni-ko for this contribution!

Looks good to me, but I was also heavily involved in the PR, so we should get two more independent approvals.

AMR + DG diffusion + PA is not yet implemented

Was previously potentially dereferencing null pointer

pazner · 2025-06-03T18:49:28Z

BTW @YohannDudouit, the preconditioner is defined in section 4.5 of this paper.

pazner and others added 30 commits April 28, 2025 15:14

Factor out test same matrix unit test utils

3c2e847

Update fem/CMakeLists.txt with missing headers

53c561c

Framework for LOR DG preconditioner assembly

1368cfb

Correct sparsity Pattern for LOR DG, wrong values

7634a4b

make style

986cee6

Extract kappa in BatchedLOR_DG

aa56746

Small fixes in BatchedLOR_DG::Assemble2D

556b43f

Use eta instead of kappa in LOR Batched DG unit test

ea0abf4

Get (p+2) Lobatto vertex coordinates for DG LOR

b8fe4cb

Template for unifying LOR DG matrix assembly

cfe25c4

Debugging

a5fccc7

Small LOR DG fixes

6dcc19a

testing

1729961

Debugging

12bafa6

debugging

c315298

Bugfix

e50ebb5

gpu debugging

208f9b8

quick edit

e194835

debugging for gpu

606a597

some clean-up

8a1a264

Factor out test code for making permuted meshes

847183e

Change mesh permutation function names

07dc0f6

Passes tests on ref-cube, and inline-hex (including when nx, ny, nz o…

0bb71aa

…r sx, sy, and sz are changed).

Passes tests on ref cube, inline-hex, and any version of inline-hex w…

e49854f

…here nx, ny, nz, sx, sy, and/or sz are changed.

DG LOR test case

2e9f545

Add commented-out code for testing 3D face permutations

Fix permuted test case

dcec51b

(Test still fails)

passes orientation tests

e78c1b8

Got rid of shoelace formula for element area and volume computations

e7e00e6

gpu compatiblility

277199c

Simplified implementation of BatchedLOR_DG::Assemble2D

dbd55a8

Toni-ko requested a review from pazner April 28, 2025 22:20

Toni-ko self-assigned this Apr 28, 2025

pazner added 4 commits April 28, 2025 17:57

In LOR solvers, use fast assembly for RHS only for H1

808f5c9

Fix lambda capture

c271283

Resolve error: lambda capture 'this' is not used

Fix shadow warnings

38ee46c

Add make_permuted_mesh.cpp to unit tests CMakeLists.txt

63ea83c

tzanio added in-review and removed ready-for-review labels Apr 30, 2025

tzanio assigned pazner and tzanio Apr 30, 2025

tzanio self-requested a review April 30, 2025 15:51

tzanio added the in-next label Apr 30, 2025

tzanio assigned YohannDudouit Apr 30, 2025

tzanio requested a review from YohannDudouit April 30, 2025 15:51

tzanio removed the in-next label Apr 30, 2025

tzanio added this to Pull Requests Apr 30, 2025

github-project-automation bot moved this to Review Now in Pull Requests Apr 30, 2025

tzanio added this to the mfem-4.9 milestone Apr 30, 2025

tzanio requested a review from Copilot April 30, 2025 15:53

Copilot AI reviewed Apr 30, 2025

View reviewed changes

miniapps/solvers/plor_solvers.cpp Show resolved Hide resolved

tests/unit/fem/test_lor_dg.cpp Outdated Show resolved Hide resolved

fem/lor/lor_batched.cpp Show resolved Hide resolved

tzanio and others added 2 commits April 30, 2025 09:00

Merge branch 'master' into lor_dg_preconditioner

7cfd3f5

real_t instead of int in DG LOR test

f17b1c4

pazner reviewed May 6, 2025

View reviewed changes

fem/lor/lor_batched.cpp Show resolved Hide resolved

Add comment

4ce11b4

Co-authored-by: Will Pazner <11493037+pazner@users.noreply.github.com>

pazner approved these changes May 6, 2025

View reviewed changes

pazner added 3 commits May 16, 2025 20:25

Remove AMR + DG LOR sample runs

35040b0

AMR + DG diffusion + PA is not yet implemented

Merge remote-tracking branch 'origin/master' into lor_dg_preconditioner

c9115e7

Fix bug in LORBase::AddIntegratorsAndMarkers

ea449e1

Was previously potentially dereferencing null pointer

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GPU Assembly of LOR DG Preconditioner #4834

GPU Assembly of LOR DG Preconditioner #4834

Uh oh!

Toni-ko commented Apr 28, 2025 •

edited by tzanio

Loading

Uh oh!

tzanio commented Apr 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pazner left a comment

Uh oh!

pazner commented Jun 3, 2025

Uh oh!

Uh oh!

GPU Assembly of LOR DG Preconditioner #4834

Are you sure you want to change the base?

GPU Assembly of LOR DG Preconditioner #4834

Uh oh!

Conversation

Toni-ko commented Apr 28, 2025 • edited by tzanio Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tzanio commented Apr 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pazner left a comment

Choose a reason for hiding this comment

Uh oh!

pazner commented Jun 3, 2025

Uh oh!

Uh oh!

Toni-ko commented Apr 28, 2025 •

edited by tzanio

Loading