What's happening everyone? In today's video I am talking about how I manage my local LLMs and agents to get frontier level results out of them. No subscriptions here, all done on locally ran hardwa...
Transcript
Well, what's going on, everybody? It's your boy, the logical dude. So I wanted to make a quick video for you today to kind of give you, like, a bit of an overview. This is going to be kind of a higher level overview. I'll break it down into other videos. But of how I get frontier level results out of local models. I'm running everything on my Mac Mini M4 Pro. 64 gigs of unified memory, so that gives me about 52 gigs of V RAM to mess around with for LLMs. And I've got to share it because I've got, you know, Harmony, lyrical GPT from BlockTunes, and then now I've got Bob on OptionsDataAPi .com, that is a chat agent, and they're all having to share my resources. So those are running on a smaller, like, 8B model, and they... you know, they do their thing. So it can run parallel to my development models. And I just have to swap out my development models, my main models, and I'll kind of get into that. But the best thing is to get really any LLM to properly work for you is you have to build a proper workflow. And that includes working in small chunks. Um, yeah, you know, you'd love to be able to just throw a prompt out there and let it one shot and do this whole, you know, a bunch of stuff, but you can't do that. You really can't, and get anything meaningful out of it. You have to be able to guide it. You have to look at it as you were the project manager. and your LLMs are your employees, they're your developers, it's your marketing team, you know, it's whatever it is that you need it to be. A lot of your frontier models can, you know, like Claude, can do like a sub -agent kind of situation. I kind of have something similar that I've set up, but it's built on personas for a couple of different models versus having a whole subset of sub-agents and everything working. I like to work on things kind of one thing at a time, you know, making sure that, you know, everything kind of gets worked in a phase kind of situation. So it allows me to help keep track of what's going on with the project. It allows me to know where we're at and, you know, how to continue with it, that kind of thing. So what I'm using is a combination of primarily two different models. They're both Quinn models, which are from Alibaba, and it's Quinn 3.6, 35B, A3B, which is a mixture of experts model. It has got a larger brain than the dense model, but it only works with, you know, 3 billion of the 35 billion parameters at once. So it's like a big generalized model that... that splits itself up into different experts, you know, on different topics, right? Well, that model is awesome to just be my workhorse, my everyday, kind of just general, you know, hey, I need you to do this, blah, blah, blah. It's great for that kind of stuff. But where it falls short is in kind of tool calling and directional. Now, it's great for tool calling, but unless you are, being directional and telling it what it's supposed to do and what tools it's supposed to call. It just kind of wants to do its own thing sometimes. So where that can get messed up is primarily in planning and in the development process. It really starts to kind of get messed up. when you get into a long session and it's got a whole lot of context it's got to work with. That's why I suggest working in smaller chunks, you know, like, so what I do is I get out my planner model first, which is a dense model, which is Quinn 3 .827B. That thing tests on... Opus levels, right, for reasoning and I think agentic coding abilities and that kind of thing. It was testing as high as Opus 4 .6, which for a local model that fits on my Mac here running at my house, that's pretty awesome. And I don't have to freaking pay any subscriptions. I don't have token limitations for a session or anything like that. So that's why I pull this model out to do the big, heavy reasoning stuff. I can give it a prompt, give it instructions, and just let it go. And sometimes it takes a while for it to do, because a big dense model on these Mac minis, it's going to take a while to run. They don't have the biggest, it's got a lot of ability to hold big, you know, models with all that V-RAM, but the memory bandwidth. So the speed that it reads all that is slower than a GPU. right? So, if I was able to run, if I had, say, a 3090 with 24 gigs that I was able to run that 27B dense model on, it'd be just as fast as my mixture of experts model on my Mac, right? So I'd be able to run 5060 tokens per second, you know, or better. But no, I'm only getting like 10 tokens per second with the dense model. So that's why I give it tasks that I can just let it run and let it go. So when I got a project that I want, it's like, hey. this is what I want to build, blah, blah, blah, I need you to do full market research, figure out what competitors are out there, figure out what's out there, what market niches that we need to fill, what moats that we can build around our product, and that kind of thing, and it will go out and it will plan the project. It gives you all the ideas, it says, this is, you know, and it'll just work out a whole plan. And then I'll tell it, okay, well, I want to build it, you know, we want to build it this way, the most efficient way. possible tell it whether you want to and it'll usually tell you what it wants to build it in whether it's Python rust whatever and whatever it feels like the need is and then I tell it to go out and get all the docks you know, that it's going to need for the project. Go ahead and set up the directories, go ahead and set up the beginning docs, the agent's markdown file, all of my different, you know, agent crawling docs, and read-mys, and roadmaps, and all that kind of stuff that my developing models are going to need to get this job done, right? So, my 27B goes out. does all that stuff. It might take sometimes an hour or two to get the results back that I'm looking for, but it does it and it does it well. And then I will take the plan that that model gives me, you know, we'll tweak it, do whatever, you know, I feel like it needs to happen, you know, and then once I've got the full plan in place, then it will usually save that into like a roadmap document. And then I go and I switch the models to my mixture of experts model. And I switch the persona from the planner and architect to the workhorse developer, the senior workhorse developer. And we start phase one. And it works phase one. Gives me the reviews. commits, backs up, all that kind of stuff. Then we go to phase two. It does phase two, you know, and I might be able to get a phase or two out of a session before it really starts getting goofy, but then I need to switch to a new session. If it starts trying to, if it starts getting caught in loops or something like that, that's when you know that you got to move to a new session at that point. So I will usually say, hey, where are we at? you know, what do we need to continue for this phase, and then you can go into a new session and have it continue and get it all fixed. It does a good job, like, if you can get it back on track, it does a really good job about going back and fixing itself, figuring out where it's at and it fixes itself. But that's why I like the smaller chunk. So I generally like to work in one phase per session. It just makes things a lot smoother that way. And then you can know to go back. So, like, if I need to work on that particular feature, I can go back to that session and say, hey, you know, we were working on this. Now that we've got more stuff built, I need to go back and change this. And it knows what's up, and it'll, from that session's memory, and it'll go through and do what it needs to do. So that's why it's good to kind of stay organized in a workflow when you're working with these LLMs. When you get sloppy with it, that's when you get AI slop, period. So when you're giving it sloppy props and you're being lazy with it, it's going to be lazy and it's going to give you a bunch of slop. But when you work with it on a professional level and you stick to a workflow and you make it do its job and you keep it on track, it... These local models are absolutely giving me Claude -level results. It's absolutely amazing. So, yeah, that's how I'm working with my local LLMs. I give, I have different personas. So, like I said, I got the planner and the architect. I've got the senior developer workhorse. I've got the QA and reviewer, so that's when I go back to my Dents model after everything's been built. And I say, okay, go find all the bugs. do full security functionality audits, all that kind of stuff. I have it go through, and that's when I put the dense model on it and let it, and give it just hours. And that's what's awesome about having the local stuff is I don't have to deal with the sessions. I can just let it chew. I don't have to worry about session tokens getting used up. I can let it really put all of its brain power behind it. not having to worry about paying extra, you know, to continue the sessions, or having it run out, you know, of credits halfway through the review, and, you know, I'm having to sit there and wait to cue it back up, and it's just paying, no, I don't have to deal with all that. I queue it up, half the time, I'll queue up the review and go to bed. And then I come back the next morning, and my reviews there, and it's like, okay, this is what needs to be fixed. So I take the review, usually put it into a review document in the directory, and I'll go back, and I'll switch, go back to my workhorse and say, hey, had a QA done. This is what it found. Go and fix it. And it will go through and fix all the bugs. It's been a beautiful workflow. I'm getting the results. It's all local. No corporations are data mining me, and that kind of thing, which is a huge thing. So, hey, if you can do it, if you've got to compute, it's so worth it to go local. I've shed all those stupid subscriptions. I don't abide with the whole, you know, use will own nothing and be happy. Screw that. I am down to literally one subscription left, and that is my Apple subscription. My whole family uses that, and we're kind of tied up into that whole ecosystem, but I like Apple Music. You know, it works pretty well for me. I like some of those stuff on Apple TV. Ted Lasso is pretty lacking this season, but that's a whole other subject that we can get into another time if you're interested. But, back to the AI. Get Local if you can. Start playing with some small local models. Don't worry about fine-tuning. Learn about regenerative augmented databases. So, rag databases. Retrieval augmented generation. Sorry, I totally borked that up. It's retrieval augmented generation. Rag. I have a rag database. I've been pulling all kind of data from the internet. Turn it into markdown files. Save it into the database. Because if I ever have to go offline. I got all the information I need. So, it's really worth it in that manner. But anyway, if you have any questions, feel free to hit me up in the comments, find me on the Discord, anything like that. You can come find me on blocktunes.net in the bars area. Come chat with us over there, come talk about some music and all that kind of good stuff. Built that one, mainly with a claw, but now I've got the Quinn mixture of experts model that's maintaining it for me. Good stuff. Talk to you guys later. Be cool. Be real. Always abide with yourself. Get you some local AI. And peace out.