Testing Plugins

Last updated: 2026-08-14

Test the narrowest public boundary that proves your plugin works. Start with AgentPluginLoader for manifest and contribution behavior, then add host integration coverage only for activation, catalog merging, or a real agent run.

Note

A loader test proves structure and registration. It does not prove that a plugin is safe to trust, enabled in the host, allowed for an agent, or able to complete an external network call.

Prepare a compiled fixture

Loader tests need a real compiled plugin assembly. The repository's framework test project keeps the Echo build in the project graph without referencing its types:

<ProjectReference Include="..\..\src\Oscova.Agents\Oscova.Agents.csproj" />
<ProjectReference Include="..\Oscova.Agents.Plugins.Echo\Oscova.Agents.Plugins.Echo.csproj"
                  ReferenceOutputAssembly="false"
                  Private="false" />

Adapt those paths for your solution. Before each test:

1. Create a unique temporary plugin root.
2. Create one child directory for the plugin.
3. Copy the plugin's bin/<Configuration>/net10.0/ output into that child directory.
4. Copy its manifest as oscova-plugin.json.
5. Delete the temporary root during test cleanup.

Copy the complete output rather than one DLL so plugin-private dependencies and dependency metadata stay available. Do not load directly from a developer installation under app_data/plugins, because shared state makes tests order-dependent.

Test manifest inspection and loading

Call the public AgentPluginLoader.InspectDirectory or LoadDirectory methods. Assert both successful plugins and AgentPluginLoadDiagnostic values.

This minimal xUnit test needs no compiled assembly because the manifest fails required-field validation first:

using Oscova.Agents.Plugins;

public sealed class PluginManifestTests
{
    [Fact]
    public void LoadDirectory_MissingId_ReturnsDiagnostic()
    {
        var root = Path.Combine(
            Path.GetTempPath(),
            "example-plugin-tests",
            Guid.NewGuid().ToString("N"));
        var pluginDirectory = Path.Combine(root, "broken");
        Directory.CreateDirectory(pluginDirectory);

        try
        {
            File.WriteAllText(
                Path.Combine(pluginDirectory, AgentPluginLoader.ManifestFileName),
                "{}");

            var catalog = AgentPluginLoader.LoadDirectory(root);

            Assert.Empty(catalog.Plugins);
            var diagnostic = Assert.Single(catalog.Diagnostics);
            Assert.Equal(AgentPluginLoadDiagnosticLevel.Error, diagnostic.Level);
            Assert.Equal("Plugin manifest must set Id.", diagnostic.Message);
        }
        finally
        {
            Directory.Delete(root, recursive: true);
        }
    }
}

This proves that an incomplete, well-formed JSON manifest returns a loader diagnostic instead of a loaded plugin. It does not prove invalid JSON parsing, assembly loading, or entry-type validation.

For a valid compiled fixture, assert:

• the exact Id, Name, Version, Assembly, and EntryType values.
• the expected toolset and provider contribution types.
• the absence of error diagnostics.
• the behavior of the overload that accepts IReadOnlySet<string> enabledPluginIds.

The production loader also rejects missing or incompatible entry types and entry types that cannot be created with a public parameterless constructor. Add focused fixture assemblies for those cases when you change that boundary.

Test reflected tools

After loading the plugin, invoke its public toolset factory with an AgentPluginContext, add the result to CrewToolCollection using the manifest ID as the origin, and look up the expected qualified name:

using Oscova.Agents;
using Oscova.Agents.Collections;
using Oscova.Agents.Plugins;

var plugin = Assert.Single(catalog.Plugins);
var contribution = Assert.Single(plugin.Toolsets);
var modelStorageDirectory = Path.Combine(
    Path.GetTempPath(),
    "oscova-plugin-tests",
    "models");
var context = new AgentPluginContext(
    new NullCredentialResolver(),
    plugin.PluginDirectory,
    modelStorageDirectory);
var toolset = Assert.IsAssignableFrom<ICrewToolset>(
    contribution.Factory(context));

var tools = new CrewToolCollection();
tools.Add(toolset, plugin.Manifest.Id);

var qualifiedName = ToolNaming.BuildToolsetQualifiedName(
    plugin.Manifest.Id,
    "echo",
    "test_echo");
var tool = tools.Find(qualifiedName);

Assert.NotNull(tool);
Assert.Equal("test_echo", tools.GetOriginalName(tool!));

Add a nested resolver to the test class that returns no test secret:

private sealed class NullCredentialResolver : IToolCredentialResolver
{
    public Task<string?> ResolveAsync(
        string credentialKey,
        string fieldName,
        CancellationToken cancellationToken = default)
        => Task.FromResult<string?>(null);

    public Task<string?> ResolveAsync(
        string credentialKey,
        CancellationToken cancellationToken = default)
        => Task.FromResult<string?>(null);
}

This test proves that the registered factory creates the declared toolset and that normal reflection publishes the Echo method as oscova_agents_plugins_echo__echo__test_echo. It does not invoke the tool or prove host activation.

Test [ToolNetworkScope] and [ToolPolicy] on a toolset that actually declares those attributes. The Echo tool declares neither, so it cannot prove non-default governance metadata.

Match the current focused evidence

The repository's focused tests currently prove these boundaries:

Test	What it proves
LoadDirectory_ReturnsEmptyCatalogForMissingRoot	A missing directory produces an empty catalog and no diagnostics.
LoadDirectory_LoadsTestPluginAndReflectsToolset	The Echo manifest, toolset, model provider, speech provider, and reflected name load through the public loader.
InspectDirectory_ReturnsInstalledPluginWithoutLoadingIt	Manifest inspection returns the installed Echo manifest without calling LoadDirectory.
LoadDirectory_WithExplicitAllowlist_LoadsOnlyEnabledPlugin	The allowlist overload skips an unselected plugin and loads a selected one.
LoadDirectory_LoadsDefaultWebPluginToolsets	The live Web plugin creates WebSearchToolset and NewsToolset from a host-supplied context.
LoadDirectory_RejectsDuplicatePluginIds	Only one case-insensitive duplicate ID loads and an error diagnostic is returned.
LoadDirectory_ReportsMalformedManifestWithoutThrowing	The checked {} manifest is rejected specifically because id is missing.
LoadDirectory_RejectsExecutionValuesWithSurroundingWhitespace	Leading or trailing whitespace in id, assembly, or entryType is rejected.
LoadDirectory_ReportsMissingAssemblyWithoutThrowing	A missing declared assembly returns an error diagnostic.
LoadDirectory_RejectsAssemblyPathOutsidePluginDirectory	A relative path escape is rejected.
LoadDirectory_RejectsRootedAssemblyPath	An absolute assembly path is rejected.

Do not describe the current loader suite as direct proof of invalid JSON, missing entry types, incompatible entry types, or constructor failures. Those are implemented loader outcomes but need their own focused regression tests.

Add host tests only for host behavior

Use the host integration harness when the assertion depends on authentication, persisted activation settings, startup, merged provider catalogs, or a model-backed run:

• PluginSettingsApiIntegrationTests proves administrator-only access, explicit allowlist persistence, unknown-ID rejection, and startup activation.
• PluginProviderContributionIntegrationTests proves enabled provider contributions reach model, credential, and speech catalogs, while an installed but disabled plugin contributes nothing.
• PluginTool_EchoToolset_VisibleAndCalledInRun proves the Echo tool reaches a live agent and is called. It is skip-gated on the configured Ollama runtime.

Keep loader failures in framework tests. A full host boot adds no useful evidence for a malformed manifest or path-containment assertion.

Treat NuGet installation as a separate boundary

The online installer adds package-specific checks that AgentPluginLoader does not cover:

• package type OscovaPlugin.
• oscova-plugin.json at the package root.
• the single declared plugin DLL under lib/net10.0/.
• a total package size no greater than 25 MB.
• a folder-safe manifest ID that does not conflict with an installed plugin.

The current skip-gated PluginRegistryApiIntegrationTests use NuGet.org to prove marketplace filtering, Echo metadata, installation of the Echo DLL and manifest, duplicate-install conflict, and administrator authorization. They do not directly exercise malformed package shapes, the size limit, unsupported extra DLL payloads, or failed-install staging cleanup. Add focused service-level package fixtures before changing those validations.

Run focused commands

Run the loader and Web-plugin framework tests:

dotnet test tests/Oscova.Agents.Tests/Oscova.Agents.Tests.csproj --filter "FullyQualifiedName~Framework.Plugins"

Run reflected-tool metadata tests:

dotnet test tests/Oscova.Agents.Tests/Oscova.Agents.Tests.csproj --filter "FullyQualifiedName~Framework.Collections.CrewToolCollectionTests"

Run the host activation and provider-contribution tests:

dotnet test tests/Oscova.Agents.Host.IntegrationTests/Oscova.Agents.Host.IntegrationTests.csproj --filter "FullyQualifiedName~Plugins.PluginSettingsApiIntegrationTests|FullyQualifiedName~Plugins.PluginProviderContributionIntegrationTests"

The plugin registry suite contacts NuGet.org, and the live Echo tool test needs the configured Ollama runtime. Run those skip-gated seams only when you need their distinct evidence.

Troubleshooting

The fixture output cannot be found

Keep the plugin project in the test build graph and derive Debug or Release consistently. Do not silently fall back to stale output from another configuration.

The valid plugin test reports a missing assembly

Copy the compiled output into the same temporary child directory as oscova-plugin.json. The manifest's assembly value is resolved relative to that file.

The plugin loads but the expected tool is missing

Instantiate the contribution factory, pass the manifest ID to CrewToolCollection.Add, and calculate the name with ToolNaming.BuildToolsetQualifiedName.

A host test passes without proving startup

Install and enable the fixture before the throwaway host starts. Changing the activation allowlist after startup proves only persistence for the next restart.

A registry test is skipped

Check its prerequisite before treating the result as package-install evidence. A skipped NuGet.org or Ollama test does not prove that external boundary.

Related articles

• Echo Plugin
• Plugin Manifest
• Toolsets