proposal: add model sync adapter framework

[chlins]

This pull request introduces a comprehensive proposal for a new Model Sync Adapter Framework in Harbor, aimed at enabling seamless synchronization of AI models from upstream model hubs (starting with Hugging Face) into Harbor as OCI artifacts. The framework is designed to be extensible, policy-driven, and deterministic, while reusing Harbor's proven architectural patterns without modifying the existing OCI replication system.

The most important changes are:

Framework Design and Architecture

• Proposes a dedicated Model Sync Adapter Framework that runs parallel to Harbor's replication system, introducing a new adapter contract tailored for model hubs with a fetch → convert → push pipeline.
• Details a modular architecture: adapters per hub, system-level sync policies, deterministic packaging, and reuse of Harbor's registry management and execution/task management.

Adapter Contract and Extensibility

• Defines a narrow, hub-agnostic Model Adapter contract (resolve + stream) to make adding new model sources straightforward and maintainable, with Hugging Face as the reference implementation.

Deterministic Packaging and Provenance

• Describes a deterministic packaging process that produces bit-identical OCI artifacts for the same upstream revision and file filters, ensuring idempotence, deduplication, and reliable provenance tracking via tags and annotations.

API, Data Model, and UI Integration

[elliott-davis]

Something I don't see called out in this proposal is secrets management. Some models require the acceptance of a EULA and today their access is gated behind a users token that has accepted the EULA. Is there any effort to support these models in this proposal?

[elliott-davis]

Will this proposal make any attempt to differentiate model artifacts from container artifacts in a repository? A confused developer may try and pull one of these converted models expecting an engine to be packaged with it like a NVIDIA NIM and will experience a failure.

[elliott-davis]

Overall this proposal aligns with what I would like to accomplish for model syncing in Harbor.

[chlins]

Something I don't see called out in this proposal is secrets management. Some models require the acceptance of a EULA and today their access is gated behind a users token that has accepted the EULA. Is there any effort to support these models in this proposal?

@elliott-davis Good question. Gated models are covered by the existing token mechanism rather than a new secrets subsystem: the user completes the EULA or license acceptance on the hub side with their own account beforehand, and the access token configured in Harbor's registry management (encrypted at rest) carries that entitlement. From Harbor's perspective this is plain token authentication when fetching; it implements no hub-specific consent flow. I've updated the Security Considerations section to make this explicit.

[chlins]

Will this proposal make any attempt to differentiate model artifacts from container artifacts in a repository? A confused developer may try and pull one of these converted models expecting an engine to be packaged with it like a NVIDIA NIM and will experience a failure.

@elliott-davis Yes. Converted models are model-spec OCI artifacts with their own artifactType and media types, so Harbor's UI can distinguish them from container images and will render them with a distinct icon, the same way charts, SBOMs, and signatures are differentiated today. On the pull path, however, both go through the standard Distribution v2 API, and the server should not intercept or reject pulls based on artifact type. So the safeguard is discoverability (distinct type and icon in the UI, plus the model-spec media types for tooling to inspect), not server-side gating; whether an artifact is runnable is up to the consuming tooling.

[Vad1mo]

I don't think the claim that registry adapter won't work holds true, given the determinism the proposal already guarantees (Goal 4).

If the packer turns a given revision plus file filters into a byte-identical manifest, the adapter isn't faking registry semantics. It derives them. The same packer could sit behind PullManifest, and PullBlob(repository, digest) would map the digest back to a file path:

[Vad1mo]

I think we still should strive for making it work with adapters, I think it will not only be easier to implement and maintain, but also for users to adapt and understand. The whole ecosystem in build around that too. cli, pulumi, tf proviers and all the scriptiin people has been made over the years. IMO it is worth reusing that.

The adapter has policies, cron, executions, bandwidth limits, chunked resume and quota from replication, and pull-through from proxy cache via PERMITTED_REGISTRY_TYPES_FOR_PROXY_CACHE (your Phase 2). No new table, API or job kind is needed. The core changes left are a file filter type, a source-only flag in RegistryInfo, and a Redis digest → (repo, commit, path) map because proxy cache creates a new adapter per request.

Now it is only Hugging Face what if other model source become availible and needed replication too and the same capabilities?
we would reimplement the same thing over again.

It is not worth it, there no additional benifits in doing so.

[Vad1mo]

One good example where we failed to do it right in the first place, are the P2P preheat functionality. Harbor has 100lk deployments, theon only reported problem and users of this feature is the company who have made the PR. We never reecieve feature reqeuests, issues regarding that feature excapt from the original authors. and if it breaks it goes unnotices, until the origial team is migrating harbot to new version and notice that error. I don't want to repeat that again.

[Vad1mo]

To compare the two designs on running code, I opened a draft that takes the replication/proxy route: goharbor/harbor#24008. It adds a huggingface registry adapter. Existing pull replication and proxy cache then bring Hub models into Harbor as model-spec artifacts, with no new table, API, job kind or Administration section.

The adapter builds the manifest from the Hub API for a pinned commit, and the output is deterministic. One raw layer per file; LFS sha256 is the layer digest; non-LFS files are hashed and checked against their git blob id. The same commit always gives the same digest, so replication's digest comparison handles idempotence and no revision cursor is needed.

Results on a dev Harbor with HuggingFaceTB/SmolLM2-135M:

• Pull replication worked, and Harbor typed the artifact CNAI. A second run logged "already exists, skip".
• A proxy-cache project served the same model, and modctl pull + modctl extract worked.
• Both paths produced the same digest.
• All extracted files are byte-identical to the Hub at that commit.

Some points in this proposal hold up, and the PR says where:

• preview before import;
• per-sync file filters;
• a source-only capability;
• immutable revision tags;
• an ArtifactRegistry split into read and write parts.

Each can be added as a generic replication/registry extension that other adapters also benefit from, so none of them needs a parallel framework. The PR description lists the compromises I made and the Hub quirks that affect both designs, for example the anonymous 401 for missing models, case-sensitive ?author=, and the tree API's 1000-entry pages.

Happy to fold anything from this proposal into that PR, or to work on it together with @chlins and @elliott-davis.

[Vad1mo]

The main reasons the ArtifactHub adapter was removed were its low-quality, buggy implementation and the upcoming OCI support.

[chlins]

Fair point, I checked and it went out together with the chartmuseum backend in goharbor/harbor#18191, so it doesn't really say much about non-OCI sources. I'll drop that bullet.

[Vad1mo]

https://github.com/llmmanorg/llmman is a good reference. llmman transfer hf.co/... builds a model-spec manifest straight from HF and streams it into the push without touching local disk.

[chlins]

Thanks, didn't know about llmman. Streaming straight from HF is what we had in mind too, so this makes the PoC argument kind of outdated. I'll update that part.

[chlins]

Thanks for putting goharbor/harbor#24008 together, having running code to compare against is really helpful. Let me sync with @wy65701436 on the direction first, will come back here by next week.

[wy65701436]

Thanks @Vad1mo and @chlins for putting together — having running code on both sides makes this a much better discussion. I’ve gone through both，

Where I agree with:

• The claim that an adapter “must fabricate registry semantics” doesn’t hold once the packer is deterministic.
• Re-implementing transfer infrastructure is not worth it. Chunked copy, bandwidth limits, retry, quota accounting, cancellation, execution/task tracking — that is ten years of hardening in replication, and I don’t want a second copy of it.
• Proxy-cache for models falling out of the same adapter is a real advantage.
• The P2P preheat point is fair.

Where I still think “separate” is right — but narrower than the proposal says

• The part of the proposal I want to keep is the product surface, not the execution engine.
• Replication is a copy of the same artifact; the user model is “mirror this registry”. Model import is a conversion; the user model is “bring this model in, show me what’s in it first, let me pick files, pin the revision, tell me the license”. Those are different enough that I don’t want them to share a policy form, a filter model and an API. Concretely, the “generic extensions” in your table — browse API, file filter, revision tags, source-only capability — are in practice all for one adapter. Putting them into the replication policy model makes replication heavier for every other adapter and still gives model users a worse UX than a dedicated page would. That’s the “don’t touch replication” concern on my side: not the transfer code, the policy model.

Proposed middle ground:

• Hugging Face or other AI model registry adapter with deterministic packer
• Keep a separate Model Sync policy, API and UI: preview, file filters, revision tags, license. This is where we iterate independently and where nothing can regress replication.
• Don’t add a second execution engine. A model sync run becomes a execution with a different vendor type; same execution/task tables, same job, same logs.

Does this split work for you? If there’s a reason the policy layer also has to be replication, I’d like to hear it, because that’s the one place I’m still choosing “separate”.

[wy65701436]

One more structural point.

Replication is symmetric by design: a policy has a src_registry and a dest_registry, either side can be the local Harbor, and the same adapter serves as source or target — Harbor-to-Harbor DR and multi-site is the whole reason it exists.

Model Sync is strictly one-way: the source is always an external hub, the destination is always this Harbor, and there is no push direction — you can’t sync an OCI artifact back into a Hugging Face repo. Modeling that as a replication rule means a source-only capability flag, a UI that hides the type in the destination picker, a form full of greyed-out push/pull and namespace options, and an adapter where half the interface is MethodNotAllowed stubs. A dedicated Model Sync policy simply doesn’t have those fields: source → model → revision → destination repo. And the two features chain rather than overlap: Model Sync gets the model in, replication moves it between Harbors afterwards — exactly as it does for any other artifact.