Skip to content

[Feature Request] GGUF export and usability improvements for running locally #273

@scosman

Description

@scosman

I think we should make going from cloud to local and back super user friendly. Currently you can download cloud-fine-tunes now, but need to use some libraries/CLI to get them running locally.

This is a WIP spec - please comment below on what you'd want in this space!

  • Download GGUF format from the "Fine-tune" tab (just Lora or flattened model?)
  • Automatic Ollama integration: One button to export model to Ollama?
  • Is Ollama enough, or do people want llama.cpp, vLLM, and other engines?

Priority here is Fireworks since they have 60+ downloadable models compared to Together's 4. Downloading is easy (https://docs.fireworks.ai/api-reference/get-model-download-endpoint) but we'll need to unpack, convert and maybe quantize.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions