fix(server): preserve graphs during task shutdown

[imbajin]

Purpose

Deleting a graph could remove its configuration or registration before the local task scheduler finished closing. Pending tasks or an incomplete close then left deletion partially applied and difficult to retry. Deletion now requires local scheduler shutdown before destructive cleanup and retains registrations when graph closure fails, allowing an explicit retry.

Main changes

• Reject destructive graph deletion until the local scheduler acknowledges shutdown; single-node deletion waits up to the configured task timeout before refusing. Retain configuration, data and registrations while tasks are still draining.
• Fence new local task admission during scheduler closure and count queued dispatches even when another graph occupies the shared worker pool. Closing releases unstarted persistent dispatches with no pending cancellation or deletion coordination, keeping their durable status available to another scheduler.
• Keep task reservations until execution and owner cleanup finish, including active cancellation callbacks. Ephemeral dependency retries reuse their reservation after owner cleanup. Distributed cancellation runs callbacks outside the request thread so they cannot close its transaction; TTL reads retain the existing capacity-rejection behavior.
• Coordinate persistent cancellation through the existing task lock and CANCELLING state. A local queue entry does not imply execution ownership; completed or removed remote tasks release local reservations without rewriting their results.
• Reload preserves backend data and local configuration, including without PD. It reopens successfully detached graphs while retaining graphs that could not close, then reports the incomplete closure. Preserve registrations and original errors when deletion or creation rollback cannot finish. Retry skips completed backend cleanup; it does not roll back data or metadata already removed during a later failure.
• Match complete graph-space names when removing graphs and services, and reject backend cleanup on a fully closed graph.
• Document explicit retry and the limits of remote notifications. No automatic notification replay or cluster-wide admission protocol is added.

Verification

The latest Linux unit, core and API suites, Commons/RPC tests, and compatibility CI passed. Complete coverage was uploaded and both Codecov checks passed. Focused regressions, formatting, a clean reactor compile and independent source review also passed.

Runtime tradeoffs

Task teardown now destroys thread-local transaction owners, requiring later tasks to reopen them. Distributed cancellation uses the shared cron executor, so slow callbacks can delay scheduling across graphs; queued reservations are bounded per graph. Task-heavy throughput has not been benchmarked.

Documentation

The retry contract is documented in docs/server-runtime.md.

[codecov]

Codecov Report

❌ Patch coverage is 79.48718% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.90%. Comparing base (7644566) to head (dc994ab).

Files with missing lines	Patch %	Lines
...pache/hugegraph/task/DistributedTaskScheduler.java	81.33%	24 Missing and 29 partials ⚠️
...n/java/org/apache/hugegraph/core/GraphManager.java	67.53%	12 Missing and 13 partials ⚠️
...n/java/org/apache/hugegraph/StandardHugeGraph.java	69.23%	4 Missing and 4 partials ⚠️
...g/apache/hugegraph/task/StandardTaskScheduler.java	90.16%	1 Missing and 5 partials ⚠️
...in/java/org/apache/hugegraph/task/TaskManager.java	80.00%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3308      +/-   ##
============================================
+ Coverage     52.29%   52.90%   +0.61%     
+ Complexity     6052     2335    -3717     
============================================
  Files           762      770       +8     
  Lines         67399    68030     +631     
  Branches       9138     9256     +118     
============================================
+ Hits          35244    35991     +747     
+ Misses        28601    28469     -132     
- Partials       3554     3570      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

Blocking: yes. Summary: The deletion ordering and the retained registrations work as described, but three paths behave differently from master for callers that are not retrying a failed deletion: a drop with the local scheduler no longer waits task.wait_timeout for short tasks, a plain close of the distributed scheduler cancels dispatched tasks that never started, and reload() stops halfway when one graph cannot close. Evidence: read the full diff 87a06a9..9918ff2 (14 files, +3653/-169) and the surrounding code in GraphManager, StandardHugeGraph, TaskManager, HugeTask and both schedulers; built both trees with JDK 17 (mvn -pl hugegraph-server/hugegraph-test -am install -DskipTests) and ran one JUnit probe on each: dropping a memory graph while a 1.5 s task runs returns after 1720 ms on 87a06a9 and throws after 0 ms on 9918ff2; the scheduler-close and reload() points are from reading the code and from the assertions in DistributedTaskAdmissionTest, not from a run. All check runs at 9918ff2 are green or skipped.