arjun-audio
===

[00:00:00] Arjun, welcome to the show. Thanks for joining us today. Thanks for having me, Jeff. Yeah. This is one of the fun times when our guest has the actual walkthrough, and you're gonna show us how you've done the things we're gonna talk about, and we're gonna kind of go through how to ensure great output from AI as you're building kind of complex processes through it, and how you've really done this in real life.

Maybe not in your main job, but in your kind of side hustle experiment that you've been working on for quite a while that I think is pretty rad Before we jump into all that, can you give a quick TLDR on just who are you and, and we'll do all that fun stuff? So my name's Arjun. I started off my career in consulting, but for the last 12 plus years I've been in technology, from technology consulting to product management.

I've also had two startups along the way, and for the past five years I've been a senior PM at Verizon. I started off with Verizon Cloud and now I'm at Verizon Home. As you [00:01:00] mentioned, my side projects, I build a lot with AI and I think it's definitely the future and it's gonna really change the way we operate and execute.

So really excited to talk about this here 'cause like we said, this is one of the shows where we actually get to show stuff, and you're gonna show how you walk through this, not just talk about it. And I think from talking to product leaders and operators, there's a lot of FOMO and anxiety from reading the theater of AI on LinkedIn, and it's very easy to feel like you're behind.

But when you really get into it with the people posting these things, a lot of 'em aren't where they're posting or it's a little exaggerated. And at the same time it feels like there's always a new model coming out and that can get super expensive. What I really liked when you and I first met, and you were telling me about this, is the path to success here doesn't necessarily have to mean that you have to use the strongest, newest, biggest model, which is always the most expensive too, but you kind of develop techniques to get great results by balancing how you kind of put data in and how you set them all up for success and how you kind of break down a [00:02:00] problem, and it means that you don't have to token match.

You don't have to sit there and spend a trillion tokens. You don't have to just throw Opus 5.5 or Fable at something. Maybe we can go through that first, just kind of talk through the theory. For sure. I agree. There's a lot of theater out there and now AI is gonna be such an enabler for the doers , ?

You gotta take your pick, and absolutely token efficiency is going to be the key because it's all about tokenomics and really reducing the points of failure at scale. So as a product manager myself, my main focus is on the P&L or the workflow. For the P&L part of it, right, we have to build something that's financially sustainable at scale without really breaking the bank.

There's no point of spending $200,000 on tokens where I could just hire someone, a great developer, a great full stack engineer who could do that for me. So the way I think about it, think of tokens as like a variable cost, right? This is like your OpEx with a slightly higher processing time depending on where it stands in the workflow, right?

So when the model [00:03:00] actually gives you a layup where, you know, you just throw everything at it to crunch the numbers, crunch the data, you're actually working with an unpredictable OpEx, right? Yeah, you can put a spend cap on how much my model tokens should be, but You're very likely gonna burn through most of it very quickly.

Whereas on the opposite scenario, where you're giving the model a layup, there's certain level of pre-processing that's gonna be done. So I think of that as like CapEx, which is a one-time expense pretty much. Mm-hmm. And what it comes down to is, yeah, would you have a predictable one-time expense or unpredictable OpEx?

This really depends on the risk profile of the company, and a lot of what we're seeing right now, and this goes back to the AI theater, is because there's a lot of funding out there, and there's, like, almost a mandate for all companies to spend tokens to use AI. And that theory I don't think will hold for too long because these are actual dollars.

Yes, it sometimes pops as the stocks, you know, funding comes in better, but these costs have to be reined in. This is an OpEx problem eventually, [00:04:00] right? And you're already seeing this issue with companies like Uber, where they just burn through their entire tokens for the year in a matter of a couple of months.

These are actual problems really when we face them. And I think also it's interesting because not everything you're doing needs the highest, most maxed out smartest model. There are jobs that do, and maybe they're novel ones or ones you're kind of figuring out how to do them. But I've run into a lot of problems where we figure out how to solve it, and we kind of bake in a process, and we really lock it down using one of the really, really high-end frontier models.

But we've been able to break the problem down into smaller subsets, or maybe we know there's elements of it we can solve. You know, maybe we don't need Opus on the whole thing here. We can kind of figure out how to get 98% of the effect using Sonnet. Those are important trade-offs, and sometimes you need that 99.9.

Sometimes you can do well with, like, 97%, and is the cost trade-off worth it? Absolutely, and it really depends on your use case, right? It depends whether you're going B2B or B2C. Do you need that information in real time? I notice, like, B2C [00:05:00] users tend to have lesser threshold for time, so they expect data to come in real time.

Whereas if it's a B2B, that workflow is more about accuracy and getting the right information in the right hands. And there is a little more tenable in terms of lags, in terms of data. But yeah, lower models, they absolutely have the capability. Like, when I spoke to certain people from NVIDIA and OpenRadar at a networking event, and they said 70% of all the models used are open source, and these are not the most powerful ones.

That's a large number, and as you can imagine, OpenRadar, they have the entire view of who's using what and how they're using it. And it is a very real fact that throwing fable will only work at a very complex problem, right? If you throw fable at the smaller issues, it will definitely work, but is it really worth your tokenomics?

That's key to us If you don't need to employ Fable or Opus, there's a lot of decisions you can make with Sonnet or, or even a Haiku here and there. A lot of these models are faster when they're the smaller [00:06:00] models, and that's better for what you need. Absolutely, and this goes back to what we were talking about, the bias towards action, because your use case is gonna be very different.

And the only way to understand which use case fits which model is by experimenting with it. So I remember I spent $25 on Sonnet when I could have just done it with Haiku at, like, a fraction of the cost, right, on certain use case. There's one thing though that you've said that I think really pricked my interest, the idea of giving the model a layup.

I like your thought of you can take complex problems and often break them down into smaller pieces, and that's kind of what you mean by giving the model a layup. It's like, can you clean the data to make it easier? Are there simple deterministic things you can do low cost that are going to make the model have an easier time, or you can employ a, a smaller model or something, or break it up into several sub-questions or something like that?

Maybe explain how you do that, and maybe that's a good way to start talking about the example you're gonna show. So a lot starts with process mapping, right? You gotta break it down into the smallest components, and [00:07:00] essentially the layup comes in the form of pre-processing the data. Just to give a little example, right, what we do at Goodmunch essentially, we have people eat less ultra-processed foods.

Mm-hmm. But there's a lot of data that goes into in calculating the processing score of one individual food item, right? Say like a pesto sauce. What I could do is take a picture of the ingredients and just send that picture to Gemini and get an output. But imagine this across 10,000 users. It's hundreds and thousands of tokens used.

Like, I can't afford that. What I'd rather do is pull in those ingredients, and these ingredients are available in open source, so you can go to Open Food Facts or whatever, and crunch that data and already pre-attribute a processing score to it so that when a user actually looks up the pesto sauce, that score's already ready.

The model is not calculating it on the spot. What it's actually doing is going to the embeddings and the vector database and finding the right ingredient for you as opposed to calculating the score on the spot. So yeah, these [00:08:00] models are very smart and they're only going to get smart over time, but again, do you want to throw a Fable at s- such an issue, right?

And this is where I also feel as a product manager, you need to have the discretion as to what model will be used for what use case, right? And that only comes by iterating and doing the things that are sure for your use case, right? Everybody has a different business model and a different set of stakeholders.

So if I'm understanding right, the process there that you're talking about is one way might be you make the app, you make the capability to process the ingredients on the client side, and then as your users go through, they snap a picture, it goes through, you know, something like a Gemini multimodal.

It pulls out all the ingredients and goes and kind of analyzes each one independently and either, A, adds that to a database, or just every time it does, it scores that product or that ingredient, which is just gonna skyrocket because you're kind of breaking down, okay, do a multimodal call, break it maybe into a bunch of subqueries, maybe even evaluate each one of those, and each one is adding cost, but also adding degrees of freedom [00:09:00] to the answer where you introduce more and more places where it can actually make mistakes.

So you need those higher level models to ensure it stays consistent versus, if I understood right, the one-time cost of you're able to pull down from a database, I assume via, like, an API or, or some kind of formula like that, deterministically a lot of these SKUs and build out the database of what these products are and ingredients are and evaluate and kind of rate them, and it was a one-time thing.

You can kind of go through it once. You don't have to do it all the time. Maybe you update it periodically. Right. And what that means is when the user goes and snaps the picture, yeah, the tool still has to strip out the ingredients in the call, but now it can go back and, like you said, use embeddings or a vector database or something to find those things and get a pretty deterministic answer on each of the questions that the other way would've been a ton of subqueries and things like that.

Is that kind of accurate? Absolutely, yeah. And how I look at it in a very simplistic terms, right? The agent would have, like, a fable, would have to go through five steps to do it, whereas I could just use a sonnet and do the last two steps, and I'm done. Right. And that's where, you know, if you have something that [00:10:00] truly needs the agent to do those five, ten, you know, multiple step chain of steps and chain of logic, that's what they're good at, and they're there for that.

But at the same time, if you have ways to get around it, you probably create a faster answer that is just as accurate, maybe even more accurate and cheaper to you, which in the end is probably better for the customer too, because you can provide it at a better rate to them. Absolutely. And the good part about using such a setup is, like, when the models advance, it'll be easier to replace your current system.

Exactly. So as business leaders, right, we need to constantly keep thinking how to innovate, how to make it better. And making it better means using your, your model generally, maybe a couple of months or years down the line, not immediately. But when you have to swap out that model, the fewer steps you have in the workflow, the fewer steps of evaluations you need to run.

So your evals will have a better time, your change management team will have a better time, your devs will have a better time. Mm-hmm. So again, there's a lot of benefit to keeping it tight, you know? Yeah, exactly. I think you brought something from [00:11:00] Goodmuncher to kind of walk through what this means. Yeah.

So just some context, right? So Goodmuncher, we are helping people eat less processed foods. Right. But we can't necessarily say, "Hey, we're gonna cure XYZ disease," because you cannot do that as per FDA mandates. The FDA might get mad at you for that one. Exactly. A little bit, yeah. So what I built was this content engine, which we call the carousel pipeline, and I'm gonna be showing that how a click of a button, I literally am able to generate a carousel pipeline of up to 10 slides and post on social media, create the awareness, and create the exposure that people need.

And really, it comes in the form of to educate users. Perfect. So what I'm gonna do is just run a command first. So we're in... You're running in the terminal here? Yeah, I'm running in the terminal here. It's called Goodmuncher Content. So what we see over here is just a list of facts, right? So you see you have fact 02, and then I go down 03.

What I've done is collect a list of 50 facts, and you can look at this as the pre-processed version, right? Like I know [00:12:00] this is true. This is a raw fact. This is what the claim is, and these are the citations accordingly. Now, I have scanned and built these 50 facts list, so I know exactly what I'm getting.

Mm-hmm. Right, right. This is the pre-processing part of it. And then what I'm gonna do is run this command. So what it says is run the pipeline of fact number six. This is fact number six. I can see what I'm running, and then it'll go through the operations. In the meantime, while it runs, it's gonna go through a list of required facts, which is in this file.

So it says, "Do not include phrases like lower, it'll lower your health cost. It'll save medical bills, prevent dementia." So this is the deterministic friction that comes in. Yeah. Right. Like we, we have sit and iterated through different phrases, and we can keep adding to this as we go on. Mm-hmm. And it's important to make this, uh, again, deterministic in nature because we don't wanna compromise on this.

These are the red lines that you cannot cross. You will get in trouble potentially if they enter it. And while [00:13:00] models are great at mostly following the rules, uh, I feel like at times it's, it's a lot like parenting one of my kids, where they know it's wrong, they do it anyway, and you're like, "Hey, you did that."

And they're like, "Yeah, I shouldn't have done that. You're right." Exactly. But it'll be like, "I don't know why I did it." And, you know, you just have to be ready to step in, and this process is great because you have it built out. And I'm sure it took a while to do it, but every step you can do to make this better, more complete is less effort you need to put into evaluating the outputs manually, and you can trust it more and more and more Absolutely.

And the more you use it, the better it gets, and then you can feed that data back into this list. And what happens is, it depends on your risk tolerance as well, right? Like right now I'm doing social media, I can check it before what goes on. But imagine doing this at scale. Like, the reason I built this, because I'm like, I cannot see myself building 10 slides at a time each time.

Like, it's exhausting, right? Yeah. It takes forever. Exactly. So the, the slides are being built. So you can see what's happening here, right? Mm. Like, it'll tell you. So it first created the content, then it created the images. Now it's [00:14:00] composing the slides through something called Puppeteer that I use. And then it gives you the status like, okay, this got done.

It's true. The slides are created. These are the lists of slides, and they were created in which folder. We're good to go. I don't know if you've ever played with some of the new, like, decision models coming out that are fast and, and like dirt, dirt cheap. I'd be curious, this kind of process looks like it's born for exactly that, where instead of having to, like...

You can-- I think the deterministic is an important thing, but you could probably even run like a JEFF or something on using this and start to even pull out other permutations or something to ensure that you're not hitting on certain things, you know. Maybe a really pennies kind of level cost to help you speed up eval here might be an interesting thing.

The new decision models are gonna be really interesting i-in this piece that you, I think have been gotten very advanced on already. JEFF is definitely gonna be a game changer specifically for evals, like you mentioned, and the fact that you can iterate it over so quickly, it's gonna be important. So this is the content that was generated.

So this is gen-- this is what was generated just now? [00:15:00] Yeah. It just took a few seconds, right? Like you can see great backgrounds, gives me my branding, gives me the images. This talks about how South Korea ultra process shared, you know, how that developed, right? And again, like it's amazing to me that it can create such great, great imagery in a short span of time and gives content creators like a great layout.

What are you using for, um, the image gen here? This is, uh, Nana Banana. Oh, nice. It's a Nana Banana API. And again, I've given it my branding and what needs to be done. Yeah. Right? And we go through that in the architecture as well. Perfect. I'm always curious 'cause the models on image gen, you know, I feel like leaps and bounds every time there's a new model comes out, it's, it's ridiculously better than what was ever there before.

And every time I think it's amazing, then I see the next one, it's frighteningly more amazing. Absolutely. Uh, so I can actually have the, the architecture real quick. Yeah, please. Yeah. So th-this is architecture diagram. As we saw that we create, we give the CLI command the fact number. And the reason I control for the fact number is because it depends.

If certain topics are trending, I would like to hit [00:16:00] that first, you know. Mm-hmm. And then it marks it as drafted or not, so you know exactly what's been run before. So once I hit give the CLI command, it goes to the pipeline.run_pipeline.ts command. So this, you can essentially think of it as the orchestrator of all the three agents Uh, we touched upon Puppeteer as well.

It can essentially see the slides that are being produced and reads the text on it and checks for what's being written over there. Mm-hmm. That's the high level of it. But what the orchestrator does is it runs its, uh, the workflow in steps, right? So here, step one is it first runs the content gen.md.

So what this file does it, it's, "Okay, these are my rules. These are, this is my formatting. This is my schema." I've also given it some examples, like this is what it should look like. So if you see, like, if it runs from fact one to fact 50, the structure will pretty much remain the same because I've included this content gen.md file, right?

Again, breaking it down to the smallest components, right? Then it takes those examples and passes it to the generatecontent.ts. This is where it starts to really do the [00:17:00] hard work of seeing, getting the banned phrases from the Tripwire file that we saw, and it also goes out to Claude, and this is the file that gives the API call to Claude so that it generates the content using examples from the content gen.md file.

And then step two is image generation that we spoke about in the generateimages.ts. This gives out API call to Nana Banana, takes the content from step one, and layers it onto the images that are generated. And then step three is again the Puppeteer part, where it merges the content, the images, the branding all together into one slide, checks it, creates it in HTML format, and the CSS gives the fonts and then it gives you the final output that you're ready to, uh, post on your social media.

Mm-hmm. It's broken down here into several steps, which I'm sure helps with consistency, and you get exactly what you're looking for every time pretty much. Or is there further eval after this? So I ran the eval prior to creating this architecture, so I've tested it. I've battle tested it before and after, right?

So when I see the slides itself, I read through them. [00:18:00] I'm like... So you can think of this as a human eval. If there's something I don't like, I can just go back and change the content on it. How often do you think that happens at this point? Like, how dialed in is this? I mean, without boasting, I ha- haven't had to do it a single time.

Yeah, I mean, it's a great process. The content was done. It took 30 seconds to run through it. Yeah. You were able to just talk through it and walk through architecture and stuff while it did it. Absolutely, and it really removes the hard work of creating drafts one by one, right? This is just one piece of it, right?

For content generation. Imagine creating these workflows across every piece of your business, and this is gonna be so powerful, but again, it has to be broken down into smaller components. Because the reason I was able to run a very accurate eval is because every component had its own task, right? So, like, I could have easily put in my Tripwire check into the content gen.md, but I did not.

Because I want it to be a different process, and I know this will build out over time, the Tripwires. And I also wanna comment on the Eraser there, even the AI tool to create the diagram of the [00:19:00] architecture here. It's a neat tool. I had only seen that a little bit before, so that, that was... It's a cool way to kind of visualize how these things work.

Absolutely, yeah. This works on both content and pictures, so yeah, it's great. So looking at this, did you have a point where you were doing this before maybe you did all this pre-processing? I mean, both that and kind of the ingredient analysis and stuff. Was there a stage at which you were just trying to go whole process through, and you kinda moved to this?

Or what do you think cost differential and/or token differential would be trying to just, like, one-shot this entire stream versus kind of breaking it up and doing all that kind of stuff? I mean, I would say it's at least a 30X difference at scale. Yeah. Right? Like, if I'm taking approximately even 1,000 users, that bill will run up very, very quickly.

Yeah, I mean, the token costs, they do not go up linearly model to model. It's, like, exponential model to model. Exactly, because the way the tokenomics works is it has to reroute the same tokens back to it in order to understand. The reason it's, like, exponential is, like, the first 10 words that you put in is read, then say the output [00:20:00] is 50 words, so it'll be 50 plus the 10, and so that's 60.

And then it'll keep going and keep going, so that exponential rise across 1,000 users, good luck managing the tokens. I don't wanna get into that, so. No. I think you mentioned when we first met something along the lines of breaking down this and, and being more managed in token cost and how you get better results easier, faster, has allowed you to cut by more than 50% what you would've had to charge for users.

Yeah, that's correct. So for Goodmuncher, right, it's a side project, but it's so pertinent to today's point, right? Like, find a real problem, use AI to solve it through workflows. When we were saying, all right, if we had to charge customers at some point for this, right, what would it look like? So when we do a certain competitor analysis, we see, all right, like, $20 is the max, $25 per year, right?

So how do we bring that cost basis down so we come within the 20 to $25 a year, but also, like, maybe we'll bring it further down, may- maybe bring it to $15, right? Yeah. Or give, like, an enhancements to certain user experiences. So there's so much of [00:21:00] a delta between using a lower model such as Sonnet versus Fable, right?

Pass those dollar cost savings on to your customer, right? Like, this is what AI is there for. Like, collectively it has to be better for everybody to use it, . What I've found interesting about the move to AI is historically we've all had this idea for data management or data analysis of, like, garbage in, garbage out, and that is not untrue still even here.

Like, AI doesn't exempt us from that. Maybe it makes it a little more resilient, right? Like, you can less process, less clean data in, you just need a higher level model, but you would still be better off even if you're using, you know, Astra or Fable. But no matter what, you're gonna be happier giving even those models clean, well set up data versus not.

So I think it's a very interesting thing to look at and a good point that you've utilized well on your side project, but applies, I'm certain, to, to a lot of how you're looking at work in your core job, talking to product folks and engineers and people all over the country. Something that everyone is thinking about is we were told to move fast with AI.

We're told to do [00:22:00] more. At the same time, there are token budgets, and we can't spend $300,000 on tokens. If you're not careful, you can't get the best of both worlds, and I think a lot of this is giving at least, you know, one framework for how you can have your cake and you can eat it, too. Absolutely. And, and I think there's gonna be a large transition towards open source models- Yeah

where you use a lower version even of the open source models to double dip into the optimization I really like the point where you mentioned about garbage in, garbage out, because no matter how strong the model is, if your inputs are not right, you will get garbage out. You have to be very mindful because I go talk to startups as part of, like, uh, the Cardinal Tech ecosystem.

I'm an alumni, and I see a lot of students thesis building with agents. It's so easy just to throw an agent at a- any problem, but what we're not being mindful of is the eval aspect of it. You are building products for paying users eventually, and you're not gonna know when your model hallucinates or goes off track, and this could [00:23:00] have massive repercussions for the customer.

Specifically, for example, in the psychology like Goodmuncher, if somebody has an allergy or somebody cannot eat ultra-processed food and they veer off those trails, that is a large impact. I have to be deterministic of that. No compromise over there. So I do wanna address evals for agents, right? Just because it's gonna be the future about how we are all building, but it's really important to measure what you're building and monitor it, right?

So a good starting point, I feel, you know, for an eval is running an EDA, exploratory data an- analysis on your production data. Can you just define for people an EDA? Yeah. I like to define the TLAs. Yeah, of course. The exploratory data analysis, essentially seeing how many categories you have, say, for example, of diets.

Mm-hmm. And just understanding the entire dataset. So example, how many customers do I have? What kind of food do they eat? What are the different categories? What are the different types of diet preferences they have? Things like that. So just do an overall analysis of your dataset. And there are Python scripts and libraries that can run [00:24:00] EDAs in a few seconds.

But it's important to get a sense of that because it flows into the next step is where you build a golden dataset. So essentially you create a set of 10 to 50 examples, like this is the right one, this is it, and I know this is 100% true. There is a little bit of hard work that goes into building this out, but if you use like a front end, like a frontier model, you'll definitely be able to create that in maybe a half an hour.

And then once you create that golden dataset, you reverse engineer like the measurement metrics of... on that dataset. So for example, if I see 50 diet preferences that I have- I can say that these are the metrics that I need. For example, how many users do I have that are vegetarian? How many that are vegan, right?

So you kind of reverse engineer that and create a rubric to measure that output. So for example, if your output is qualitative, in the case of Goodmuncher, it would be recipes. So I would say my rubric is, does it give a comprehensive list of ingredients? Are those ingredients present in the store that was selected by this user?

Do the ingredients adhere to the user's [00:25:00] dietary preferences? Do they adhere to certain allergies and things like that? The next step, which is really important and very complicated, is creating a staging database, which will be a subset of your production database. So the idea is not to touch the production database, leave your live users alone, but make sure that the staging database is representative of your actual production data.

And we have to be very careful about not blindly copying user data from production. So like keep your PII in place, make the data anonymous, de-identified, things like that. The next step would be to run the eval cycle on your staging database against the rubric that was created. So there are multiple ways to run evals.

There are four primary ways. So it's like human evals, so that was how we created the golden database. There's user evals, there's code-based evals, and there's LLM-as-a-judge. So for Goodmuncher, I use a combination of LLM as a judge and the human evals. So for LLM as a judge, like for deterministic output, you can use [00:26:00] like a lower model, whereas if it's a more advanced, something more qualitative, you want to use a frontier model, right?

So depending on how you did step two, which is the golden dataset, you may need to run between ten to fifty examples through the eval, right? And make sure you run five to ten examples per category. So for example, I would consider a vegan and someone allergic to nuts as one category. So run five to ten examples for each one of them.

And the next step would be to include observability tracers, so something like a tool like LangSmith Highly underrated, would recommend everybody to use that, and it really tells you how the eval breaks, right? So these observability tracers will help you point out at which specific point the workflow is broken.

So example, if there's a certain agent that has hallucinated or where the input mark is not set correctly. Was the output not according to a certain JSON format? So it, it makes it a lot easier to debug the workflow. And then [00:27:00] lastly, I would say create a dashboard if you can based off all these tracers.

Not mandatory, but highly recommended. And just overall, I would say, you know, like deterministic evals are a lot quicker than non-deterministic ones. So again, going back to token economics, feel free to use a lower end model for that. This way, as you take in data, you can ensure the model is running as expected, right?

But also, how do you pivot as newer models come out? How do you kind of run, you know, maybe you formulate the dataset, like you said, on one or two methodologies, but you don't need to continue to use Opus or Fable to run it continuously if you can find that you can get the same results from a lower model.

So this is kind of the framework if you do this, you're gonna be able to run through evals cleanly and efficiently, right? Yes, absolutely. Cleanly, and again, going back to you can switch models out a lot cleaner as well. Yeah. The long run, putting that CapEx effort, that initial effort to set it up, will pay off big time.

Arjun, I appreciate you coming on, and hopefully we can, you know, stay in touch and, and we can have you [00:28:00] back maybe a little while down the line and see how this has evolved and how have you updated and how you think about it in the future. If people want to reach out and come talk to you more about this, is LinkedIn a good spot or is there a better spot to find you?

Yeah, LinkedIn's great, and thank you so much for having me. Yeah. Yeah. Thank you for coming on. Thank you. Take care.