offers not sorted by price in dstack apply plan — cheap offers hidden behind expensive ones

[megheaiulian]

Description

When running dstack apply -f service.yaml (with fleets: [my-fleet]), the offers displayed in the run plan are not sorted by price. Cheap offers (e.g., Vast.ai RTX 4080 at $0.37/hr) appear after expensive offers (e.g., RunPod A6000 at $0.98/hr), making them invisible with the default --max-offers=3. By contrast, dstack offer --fleet my-fleet correctly shows offers sorted by price ascending.

Steps to Reproduce

1. Configure a fleet named my-fleet with backends: [vastai, runpod] and GPU requirements (e.g., total_memory: 60GB..)
2. Configure a service that references fleets: [my-fleet] with matching resource/backends config
3. Run dstack apply -f service.yaml — observe first offers are expensive (e.g., $0.98+)
4. Run dstack offer --fleet my-fleet — observe cheapest offers first (e.g., $0.37+)

Expected Behavior

Both commands should show offers sorted by price ascending (cheapest first), since all offers from Vast.ai and RunPod are marked AVAILABLE.

Actual Behavior

dstack apply shows offers in an approximately-sorted order where cheap Vast.ai offers are buried after expensive RunPod offers. With the default --max-offers=3, users only see:

#  BACKEND          RESOURCES           INSTANCE TYPE   PRICE
1  runpod (US...)   1xA6000, 48GB...   ...             $0.98
2  vastai (...)     1xRTXPRO6000WS...  ...             $1.16
3  runpod (US...)   1xRTXPRO4500...    ...             $1.28
 Shown 3 of 97 offers

While dstack offer correctly shows:

#  BACKEND          RESOURCES           INSTANCE TYPE   PRICE
1  vastai (...)     1xRTX4080S...      ...             $0.3744
2  vastai (...)     1xV100...          ...             $0.43
3  vastai (...)     1xQRTX8000...      ...             $0.45
...

Root Cause Analysis

Two related issues:

1. _get_job_plan() sorts by availability only, not price

In src/dstack/_internal/server/services/runs/plan.py:835:

job_offers.sort(key=lambda offer: not offer.availability.is_available())

This is a stable sort by availability. Since all Vast.ai/RunPod offers are AVAILABLE, it’s a no-op — preserving the input order, which is not price-sorted.

2. Backend offers are not guaranteed to be price-sorted

get_backend_offers() (src/dstack/_internal/server/services/backends/__init__.py:495) uses heapq.merge to merge per-backend offer lists:

offers = heapq.merge(*offers_by_backend, key=lambda i: i[1].price)

heapq.merge requires each input iterable to be pre-sorted. However:

• Vast.ai (core/backends/vastai/compute.py:58-76): get_offers_by_requirements() returns the raw get_catalog_offers() result without sorting
• RunPod (core/backends/runpod/compute.py:79-90): get_all_offers_with_availability() similarly returns unsorted results from get_catalog_offers()
• gpuhunt Catalog.query() explicitly documents that its results are "not strictly sorted by the price" — it uses heapq.merge on per-provider lists that may not be individually sorted

So offers flowing into _get_job_plan() via the dstack apply path are in an approximately-sorted but not strictly price-sorted order.

Why dstack offer works correctly

get_backend_offers_in_run_candidate_fleets() explicitly re-sorts by price at plan.py:732:

backend_offers.sort(key=lambda offer: offer[1].price)

The dstack apply path (find_optimal_fleet_with_offers() → _get_backend_offers_in_fleet() → get_offers_by_requirements()) does not sort by price after the availability-only sort in offers.py:116.

Code Flow

dstack apply path (not sorted by price)

1. get_job_plans() → find_optimal_fleet_with_offers() → _get_backend_offers_in_fleet() → get_offers_by_requirements()
2. get_offers_by_requirements() sorts by availability only (offers.py:116)
3. _get_job_plan() sorts by availability only (plan.py:835)
4. Result: offers in approximately-sorted order, not by price

dstack offer path (correctly sorted by price)

1. get_job_plans() → _get_offers_in_run_candidate_fleets() → get_backend_offers_in_run_candidate_fleets()
2. get_backend_offers_in_run_candidate_fleets() sorts by price (plan.py:732)
3. _get_job_plan() sorts by availability only — but since all are AVAILABLE, the price order is preserved
4. Result: offers sorted by price ascending

Proposed Fix

Add price as a secondary sort key in _get_job_plan():

# plan.py:835, change from:
job_offers.sort(key=lambda offer: not offer.availability.is_available())
# to:
job_offers.sort(key=lambda offer: (not offer.availability.is_available(), offer.price))

This ensures that within each availability group, offers are sorted by price ascending — matching the behavior of dstack offer.

An alternative (or complementary) fix would be to add the same price sort in get_offers_by_requirements() (offers.py:116), which would fix the root cause for all callers. However, the _get_job_plan() fix is more targeted and consistent with the existing explicit sort in get_backend_offers_in_run_candidate_fleets().

Secondary Issue

dstack apply --max-offers defaults to 3 (src/dstack/_internal/cli/services/configurators/run.py:350), while dstack offer --max-offers defaults to 50 (src/dstack/_internal/cli/commands/offer.py:47). The low default for dstack apply amplifies the impact of incorrect sorting — users only see 3 offers and never discover the cheaper ones further down the list.

This default could be raised to match dstack offer (50), or at least increased enough (e.g., 10-20) so users can see a representative sample of available pricing.

[megheaiulian]

@jvstme Does this make sense ?

[jvstme]

Hi @megheaiulian. Vast.ai offers are actually intentionally not sorted by price, because cheaper Vast.ai offers tend to be unreliable, especially in terms of network.

For example, I've just tested these two offers that came first when sorted by price:

#   BACKEND                 RESOURCES                                               INSTANCE TYPE                   PRICE     
1   vastai (us-california)  cpu=18 mem=8GB disk=100GB gpu=RTX4070Ti:12GB:1          35479070                        $0.1051
2   vastai (vn-vietnam)     cpu=10 mem=16GB disk=100GB gpu=RTX5070Ti:16GB:1         31468234                        $0.1286

I tested them with this configuration:

type: dev-environment

The first one took 6 minutes to pull the Docker image (usually takes less than a minute), then a few more minutes to set up the dev environment (usually takes seconds). The second one started quickly, but took 15 minutes to set up the dev environment.

Since we don't want users to run into such performance issues, we sort Vast.ai offers by score, so that the more reliable offers come first.

But I understand that this ordering can be inconvenient. Some solutions I can propose are:

1. If your goal is only to stay within some budget, you can use the max_price property in run configurations, which will hide the more expensive offers.
2. If you are sure that you want Vast.ai offers to be sorted by price, I think we could make this behavior opt-in via a backend setting. The offers could then be sorted conditionally in the backend implementation (see VastAICompute).
3. In addition to no.2, we could add a backend setting to filter out offers below some baseline score. This could help avoid hitting unreliable offers, even when sorting by price.

Please let me know what you think or if you have any other ideas

[jvstme]

By contrast, dstack offer --fleet my-fleet correctly shows offers sorted by price ascending.

This might actually be unexpected. I would expect dstack offer to respect the backend-specific order too.

This default could be raised to match dstack offer (50), or at least increased enough (e.g., 10-20) so users can see a representative sample of available pricing.

I'd say this depends a lot on the usage pattern. For example, if a user only uses AWS and knows they need an H200 for their workload, they likely only care about the first offer that comes up. In that case, showing more offers just adds extra noise. I personally think 3 is a good default, and it can always be adjusted with an explicit --max-offers.

[megheaiulian]

@jvstme It makes a lot of sense to sort offers by reliability score like that. Its a safe default.
I think users would like that offer and apply would consistently return the same results.
It would also be useful if the sorting was at least partially configurable by the user, and we could filter out offers based on network speed and cuda version.
There are some workloads where the high download time is tolerable ...