Can you fine-tune on localized steering of an LLM? (lemmy.dbzer0.com)

submitted 2 weeks ago* (last edited 2 weeks ago) by hok@lemmy.dbzer0.com to c/localllama@sh.itjust.works

19 comments fedilink hide all child comments

I want to fine tune an LLM to "steer" it in the right direction. I have plenty of training examples in which I stop the generation early and correct the output to go in the right direction, and then resume generation.

Basically, for my dataset doing 100 "steers" on a single task is much cheaper than having to correct 100 full generations completely, and I think each of these "steer" operations has value and could be used for training.

So maybe I'm looking for some kind of localized DPO. Does anyone know if something like this exists?

you are viewing a single comment's thread
view the rest of the comments

[-] iii@mander.xyz 2 points 2 weeks ago

So supervised vs unsupervised, according to you, is a property of the dataset?

[-] hok@lemmy.dbzer0.com 0 points 2 weeks ago

Sorry, I really don't care to continue talking about the difference between supervised and unsupervised learning. It's a pattern used to describe how you are doing ML. It's not a property of a dataset (you wouldn't call Dataset A "unsupervised"). Read the Wikipedia articles for more details.

[-] iii@mander.xyz 1 points 2 weeks ago* (last edited 1 week ago)

It's alright :)

this post was submitted on 20 Dec 2024

12 points (92.9% liked)

LocalLLaMA

2392 readers

2 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 2 years ago

MODERATORS

SkySyrup@sh.itjust.works

pax@sh.itjust.works

noneabove1182@sh.itjust.works