Skip to content

Conversation

@anujj
Copy link
Contributor

@anujj anujj commented Nov 24, 2025

This PR basically extend the support for pre-registered plug-in for NvTenosrRtRTX which was earlier done for Cuda in the PR: #1850

What We Are Doing

  • Enable NvTensorRtRtx pre-registered plugin provider support - Allow users to register custom NvTensorRT execution provider libraries (similar to existing CUDA plugin support)
  • Add example code demonstrating plugin provider usage in C++ and Python

Implementation Flow
Pre-Registered Plugin Path (PATH A)
When a custom EP library is registered

    app Code:
      └─> register_execution_provider_library("NvTensorRTRTXExecutionProvider", "/path/to/*dll")
           └─> Creates model with provider="NvTensorRtRtx"
    
    Model Creation (model.cpp):
      └─> Check GetOrtEnv().GetEpDevices() for pre-registered "NvTensorRTRTXExecutionProvider"
           └─> IF FOUND (PATH A):
                └─> Use AppendExecutionProvider_V2() with registered provider

Built-in/Default Path (PATH B)
When no plugin is registered (standard behavior):

User Code:
  └─> Creates model with provider="NvTensorRtRtx" (no pre-registration)

  Model Creation (model.cpp):
    └─> Check GetOrtEnv().GetEpDevices() for pre-registered "NvTensorRTRTXExecutionProvider"
         └─> IF NOT FOUND (PATH B):
              ├─> FALL THROUGH to generic provider registration block
              └─> Generic code uses AppendExecutionProvider() with built-in EP
                   └─> On WinML: includes device filtering (multi-GPU support)
                   └─> On non-WinML: simple provider append

@anujj anujj marked this pull request as draft November 24, 2025 08:11
@anujj
Copy link
Contributor Author

anujj commented Nov 24, 2025

@gaugarg-nv @ishwar-raut1 for review

@anujj anujj marked this pull request as ready for review November 25, 2025 11:15
@anujj
Copy link
Contributor Author

anujj commented Nov 25, 2025

Adding @kunal-vaishnavi and @baijumeswani for review

if (provider_options.name != "cuda" && provider_options.name != "rocm" && provider_options.name != "DML") {
// Check if this is NvTensorRtRtx with a pre-registered provider (already handled above)
bool nvtrt_already_handled = false;
if (provider_options.name == "NvTensorRtRtx") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we generalize to something like this?

bool already_handled = std::any_of(GetOrtEnv().GetEpDevices().begin(),
                                   GetOrtEnv().GetEpDevices().end(),
                                   [](const auto& device_ptr) { return device_ptr->Name() == "NvTensorRTRTXExecutionProvider"; });
if (already_handled) return;

// For providers that go through the extensible AppendExecutionProvider API

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@anujj
Copy link
Contributor Author

anujj commented Dec 9, 2025

@kunal-vaishnavi : I have addressed the review comments in the latest commit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants