Episode Transcript
[00:00:00] Speaker A: Estimating is silly. It's a waste of time, it's a waste of money, and so on and so forth. Estimation. No, fire the person question.
I want to have as many dev environments as I have people in my company.
I want to have thousands. But I want them to be active and to be running only when they're used. The easiest way to justify anything is to say if we invest this amount of money, then we will get this amount of money back. Right? That's the easiest way to justify it. But it's not really about finops. You don't want to scale because it's cheaper.
Being cheaper is just a side effect. You want to scale because you will have no downtime releases, you will have better performance and so on and so forth. You will get many other technical benefits and also human benefits, many, many benefits. And all those benefits will at the same time result in lower cost. We were very successful. We managed to survive Black Friday because we were scaling up and then a week later kind of. But did we scale down?
[00:01:06] Speaker B: Thanks everyone for being here today. Today we have a very special guest into our FINOP podcast, Viktor Farsic, which is someone that I've been following I think for two or three years. And then I happened to knew that the main meetup several years before. So it was an honor for me to know that he was available for the podcast and to talk today into a topic that I think it's really, really hot topic based on what we saw in Infinox X Conference that is all related to DevOps costs, the Kubernetes, the resource optimizations. And so if you have had issues with kubernetes cost or you hate your DevOps because he's doing a lot of or she is doing a lot of cost in your cloud build, you should stay and know what Victor is going to bring us. So Victor, thanks for being here today.
[00:02:04] Speaker A: Thank you for having me.
[00:02:06] Speaker B: So and of course we always are honored to have our co host, Damian Damien Monojo. How are you doing man?
[00:02:13] Speaker C: I'm doing great. First of all, it's very good to see Victor here with us. I remember back in the days in the first Spain Club Sami where Victor joined us to, to that special event. I think it was great. We know, I know Victor since then and it's good to have you here with us and see you again.
Thank you for joining.
[00:02:39] Speaker A: Yeah, it's amazing being back. Yeah.
[00:02:42] Speaker B: So Victor, even though you are a well known person in the community, especially in the DevOps, but since this podcast is also into more finops that are more specialized. Some people may not know you.
So yeah, go ahead and tell us who you are and why are you here today.
[00:03:02] Speaker A: Oh, who I am? I'm Victor. I just play with random things and I don't even know what they do anymore.
To another, it's impossible for me to describe what they do, to be honest. And even if I would describe what you do, then what I do, then my boss might find out what they do and then start demanding things.
[00:03:27] Speaker B: Yeah, it will be. It will probably. It's not probably beneficial. So we'll. We'll keep it secret then.
No, but.
[00:03:36] Speaker A: And why I'm here?
[00:03:38] Speaker C: I don't know.
[00:03:38] Speaker A: You called me, you tell me why I'm here.
[00:03:41] Speaker B: No, you're here because we have seen that there is a topic, especially in the FinOps community, where two, I don't know, roles or battles lands, which is like the DevOps against the costs in the cloud. So there is a hot topic there because I, for example, I am a cloud DevOps engineer, even though I do a lot of FinOps stuff and that's why, I don't know, I check your things. So we've seen that the DevOps sometimes are more focusing to the reliability and the scalability and the performance of the systems that we are seeing. And now there is a consideration of the costs that we are seeing in phenoms.
[00:04:26] Speaker A: So scalability and cost are the same thing, aren't they?
[00:04:31] Speaker B: Why so?
Because.
[00:04:33] Speaker A: So let me backtrack a second, right? I believe that the big part of finops is really resource management, okay? I think that it's all about how can we make sure that our resources are not using more than they need, but also not less than what they need, that they're just using just the right amount of resources, right?
So if our application uses more or less the exact amount of memory CPU that it needs, then more.
That's how we are reducing costs, right? If our clusters are scaling up and down to provide just enough compute to what we need and not more, that's how we reduce costs, right?
So in my head, and I might be wrong, because pinops is not my main area of interest, it's all about the resource management, right?
And auto scaling is a big part of that, right? Otherwise, if I cannot scale. So let's say a simple scenario, right?
If I cannot scale my application up and down, up and down to meet the changes of the traffic and the demand, what am I going to do? If I don't do that, I'm going to probably set my applications and my servers to use the maximum capacity or the capacity that is equivalent to the maximum traffic I will have.
So let's say that right now in the morning I have hundred users and afternoon I have thousand users. If I don't scale automatically then my capacity will be hard coded to thousand users.
[00:06:28] Speaker B: Yeah, for sure.
[00:06:29] Speaker A: And I will be losing money whenever I have less than that.
[00:06:32] Speaker B: Yeah, but do you think so coming from the DevOps and the practices that I am seeing is that. Do you think that DevOps tend to probably overestimate the provisioning or not care that much about probably the base provisioning. And then even though they set up the auto scaling, which I do, the base provisioning is normally way over provisioning, especially in Kubernetes. What are your thoughts on that.
[00:07:00] Speaker A: Base provisioning?
[00:07:02] Speaker B: What I mean is when you deploy the workload and it's more or less stable, do you think that initial estimate could be over tends to be over provisioning or do you think it tends to be like.
[00:07:13] Speaker A: Well, nobody should ever estimate. Estimating is silly. It's a waste of time, it's a waste of money and so on and so forth. You don't estimate.
At least you may be estimate first day and that's it, right. After that you look at data, you don't estimate. Right. That's just silly. That's how we were doing things 30 years ago, right? I mean nobody knows. So if I now develop a new application, I honestly have no idea how much memory and CPU that application is. I have no bloody idea. And I will challenge anybody who tells me that they know in advance how much, how many resources they need. Whomever says that that person is lying or is delusional, nobody knows. And that's fine because we have now mechanisms to find out.
We can actually even go further. Actually there are certain things that we can do that will actually help us never need to find out, right? Because if you're using Kubernetes, right, you have. And I'm not going even to do fancy solutions and expensive ones. Vertical Pod Autoscaler is part of every Kubernetes distribution. You can use it in observe only mode to find out how much memory CPU your application is using. Right? It's not the best solution, but it's a good start. So estimation, no, fire the person who estimated.
[00:08:44] Speaker B: Okay, that's, that's quite the statement. But do you think that from, from that point when the, the application is deployed and when you, you, you know, you do your, you deploy, you put your requests, I don't know, like By a random value. Do you think that there is the concept or the habit to do the right sizing after that or do you think that that's a practice that is still to be implemented in the lifecycle?
[00:09:10] Speaker A: Most of us have bad habits. Most of us don't know what we're doing. Of course there is a bad habit in everything.
You don't need to talk about finops, anything you can mention. And if you talk about and if you end the sentence with a question, is there a bad habit? Of course there is.
[00:09:26] Speaker B: Yeah.
[00:09:28] Speaker A: We should just try to remove our bad habits.
[00:09:33] Speaker B: Okay.
[00:09:34] Speaker C: That's part of the run. I heard one the other day. I was talking to an architect that he said I start with the smallest resource and then start scaling.
I'm not sure if that is the right approach, but I thought it was interesting. An interesting one. Start small and grow a bit by a bit by. What do you think?
[00:09:59] Speaker A: Yeah, I mean I think that that's how we were doing before when we had.
When we. So you know there were three phases in software industry, right. One phase is that I don't see what's going on. So I'm making pure random guesses and we'll see what happens, right?
[00:10:17] Speaker B: Yeah.
[00:10:18] Speaker A: And then we learned still a long time ago. Okay. Actually I can observe what's going on. I can have something as simple. I'm going to stick with Kubernetes now, but that applies to everything. I can have something as simple as Prometheus when I'm collecting memory and CPU and I can just go there once a week, once a day, once a month, a year, whatever the frequencies, go and see how were resources used over time, what's or not and adjust.
And then now we don't even need to do that. We can instruct literally machines and say, you know what, go there yourself, find out what are the averages, what are the rates and so on and so forth over a period of time. And scale up and scale down. Right. So there is no need for that architect to exist even for that particular task. You just define your applications in a way that they themselves can ask query databases with data and find out what is it that they were doing and how they were doing. As soon as a 14 adjust automatically.
[00:11:34] Speaker C: But kind of like.
I understand the point that you are. Okay, fine, if we have in production, you know, you need to have flexibility and in order it's not only to scale but also make sense. Right. To make sure that it both scale up and down and it's flexible.
[00:11:56] Speaker A: But do you really need hopefully left and right rather than up and down. But yes, right, you're right.
[00:12:02] Speaker C: Yes.
Yeah, that's a best practice.
But tell me something, how will you separate the environment? Because now we are seeing like this approach, you have a multi account strategy and in which in dev you wouldn't need so much scaling only for testing times, right? And then I guess that the question that's been to ask before comes more into play. How do you plan it? Or how do you do this kind of, do you even work in a multi account strategy?
[00:12:40] Speaker A: But in a dev, the logic is still the same, right? So if I have a system that goes to some, to go to some database, something that has metrics and asks, okay, so what was the usage, this or that? And adjust accordingly by scaling up and down or left and right, or both, right? So then if you have in dev, if you have that in dev, your memory CPU usage will most likely be much smaller, right? And it will find much smaller numbers and it will scale down to smaller number of replicas, right? So the logic is still the same, right? The question is only whether you want to bother or not. But if you already set it up for production to be like that, then it's basically, basically no additional effort to have the same scaling capabilities and whatever else you have in other environments.
However, in dev environments, the real cost is not really how much memory CPU you're consuming and through that, how big of your nodes are and so on and so forth.
The biggest bottleneck in dev for many organizations is that they're using permanent environments.
What that means is that you're spinning up some development environment for the whole team, for the company, for individuals, whatever it is.
And that something is running all the time that produces two very potentially big issues. One is that you're wasting money like crazy, right? That doesn't matter how, whether your memory consumption is correct or no, you're still running compute while you're sleeping.
And unless you're very, very rich, that's just silly.
And at the same time, because those things are expensive, you are not only wasting too much money, but never having enough, right? That's why usually organizations have one or two dev environments. And to me that's unacceptable.
I don't want to have two dev environments. I want to have as many dev environments as I have people in my company.
I want to have thousands, but I want them to be active and to be running only when they're used.
So at the same time I'm going to reduce cost but also enable people to be effective because big Part and I feel that this is probably the area that is often forgotten in finops is that the biggest cost are people. Let's start with that. That's what costs the most, right? And whenever somebody needs to wait for something, that's the cost.
Oh, I need to wait because Joe needs to finish something over there before I can start working on that. Something that's cost that is often overlooked because hey, we actually measure the tangible things. We measure how much servers cost and how much networking costs and what's not. We often forget that we are wasting people's time. Right. So I want at the same time to make people more productive, but also by giving them everything that they need, that there is no bottleneck in what we do, but at the same time use the absolute minimum amount of resources. And to accomplish that, the solution is simply to use ephemeral environments or ephemeral servers or ephemeral applications or whatever else. Everything ephemeral, right? Production is permanent. I have permanent clusters that run production and the nodes in those clusters are going up and down, scaling down, scaling up everything else. Hey, I'm going to start working on this. Excellent. I get the environment in a few seconds, maybe a minute, I do whatever I'm doing. The moment the system detects that I haven't been working on it for whatever period of time, let's say five minutes, it goes down.
[00:16:54] Speaker B: Yeah, I totally agree with that. Like we are seeing. So we have. I've seen two approaches. One is like the permanent device and the permanent Q and A, like shared environment with everyone gets there and all the stuff is deployed there. And then we are seeing another approach that is the ephemeral. Like every developer has its own ephemeral cluster. In this case, if it's a deployment or a workload depends on if you do it on namespace basis and then you deploy everything there, you test whatever you want and then it will be deployed for, I don't know, whatever hours or whatever time. And then if there is no activity in there, it will be shut down. And it's also, and I don't know if you test it out, but if you can connect it with the PR stuff. So whenever, when everyone creates a pull request for testing a feature, you can just send it out to create the environment. And then whenever, when that merge of course is closed, but then beforehand it can be also closed for that. What do you think about that approach?
[00:17:58] Speaker A: No, same thing with PRs.
You know, as long as that is. And it doesn't really matter whether it's PR or It's dev environment or whatever it is. If it's not active, shut it down. And we have solutions for that these days, right? You can, let's say if we stick with Kubernetes, you can use Knative that will scale down your application to zero replicas.
You can use V cluster to easily create cluster and destroy it. It's kind of a virtual cluster depending on some actions. There are plenty of solutions, right? And of course it all requires work, but it's not that much of a work either. The biggest bottleneck usually in getting there is that our applications need to change. And this is also the area where I feel that many finops focus people are struggling, right? Oh, I can do this, I can do that. But I cannot ask you to rewrite your application, for example, right? Oh, that's a no go.
Nobody allows that. And very often your application is a bottleneck. Maybe, for example, let's say that your application cannot scale to multiple replicas. It is not fully stateless or not. Right? So if you cannot scale, we cannot optimize the usage of resources because then we need to assume the worst case scenario, whatever is the maximum that we expect ever to happen, right?
[00:19:39] Speaker B: Yeah, and I totally on that point.
If you are not able to first, if you are not able to scale your application, you are bounded to the state. And if you are bounded to the state, that's really, really, really problematic for the cost and especially for reliability as well. Like the moment that the state has an issue.
[00:20:00] Speaker C: By the way, we are hearing a lot lately about what Victor just mentioned. The thing that the application is not being looked at and that requires to do as well where possible, right?
To start doing optimization.
[00:20:18] Speaker A: And you know, most of the time.
Let's say if I stick with application example, right? Let's say that we come to the conclusion we should change parts of the application, rewrite something, whatever. Yeah, right.
FinOps is not the reason to do that, right? FinOps is the best reason to justify it, right? Because the easiest way to justify anything is to say if you invest this amount of money, then we will get this amount of money back, right? That's the easiest way to justify it. But it's not really about finops. You don't want to scale because it's cheaper.
Being cheaper is just a side effect. You want to scale because you will have no downtime releases, you will have better performance and so on and so forth. You will get many other technical benefits and also flow benefits and human benefits, many, many benefits. And all those benefits will at the Same time result in lower cost.
[00:21:21] Speaker B: But I want to highlight something that you mentioned there, and maybe that's a clarification that it needs to be needed, is that the finops is not just to make the application cheaper, to reduce the cost is to. The whole idea is to maximize the value that you get from each of the dollars that you spend. So yes, that's why you can do. And you can do finop. You can justify it with finops. I mean, and you can, it's the best reason in terms of, in this specific term, like you can maximize the ROI that you get for your application. And that's the whole idea. Like if you tell business, you tell your manager, the manager of your manager that this application will make, I don't know, $50 more per user with the same amount of money. That's enough justification to do any change that you need if the cost that you said the human cost and infrastructure cost will be worth. So if you are improving these KPIs and these ratios that we are seeing now, it's a very hot topic as well in finops. Once you are improving that metrics that are related to business, which is the important thing, that's all the justification you need for your manager, right?
[00:22:30] Speaker A: Oh yeah, exactly.
[00:22:32] Speaker B: That's great. That's great. No, no, it's because it's a very hot topic because Sometimes you know, DevOps and certain people will think like, yeah, it's only because to make it cheaper, but it will make it less reliable. And it's not to make it cheaper is to make it much more efficient in terms of business returns.
[00:22:48] Speaker A: Exactly. I mean I think that we are, both of us are actually talking about the same thing, just not maybe using different words. It could be. But yeah, it's not about. I think that we both agree it's not about reducing cost, it's about being better.
[00:23:06] Speaker B: Right.
[00:23:07] Speaker A: And at the same time that reduces costs. Great. Right. But yeah, it's about your users accomplishing the same result come faster that results in more income. Right? Yep. Hey, if Amazon can, can, can make me shop, buy something in 15 seconds instead of five minutes, that's, that's a business value, right?
[00:23:29] Speaker B: Yeah.
[00:23:30] Speaker A: And for them to do that, their site needs to be lightning fast. It's kind of, it needs to happen like this. I'm not waiting for anything. I'm just shopping, shopping, shopping like crazy, right? Yeah, exactly. And you accomplish that by your application simply being better in many different ways, faster, whatever the better means, right?
[00:23:52] Speaker B: Yeah. And regarding this topic, because I think it's very interesting to continue conversation. And now that we are recording this in a very shopping amount of the calendar, in a very shopping date, in terms of especially load testing and scalability and costs. Like how do. Because that's what you're saying, right? There is no good in Amazon that for reducing the cost, they will reduce the amount of orders that they will get on Black Friday because they are selling like crazy or any other, you know, company. So how do you think this load testing and this curve of costs and users should be considered on the like load testing DevOps site? Especially in the Q and A and testing. When someone is testing like Black Friday warlocks, you should consider like business metrics, KPI metrics. Also in this Q and A, this QA phase.
[00:24:52] Speaker A: Yes and no. At the same time, I think that testing is important. Of any kind of testing, including load testing or performance testing. It's definitely important.
But the only testing that really, really, really matters is testing in production.
[00:25:12] Speaker B: Okay.
[00:25:13] Speaker A: Kind of. Until you reach production, you don't really know because your tests are not. Will never be able to simulate the real scale. Your tests will never be able to fully simulate the randomness of your users and so on and so forth. Right.
I'm definitely in favor of test. I'm a testing freak. So I definitely think that everybody should be testing in any sort of way.
But what is just as important or more important is testing in production. And when I say testing in production, I don't mean, hey, let's run our unit tests in production or load test in production. What I really mean is that, do you.
Is your observability ready for that? Right. And can you really see what's going on? Can you predict what will be happening based on production?
And ultimately can you automate the actions behind those predictions? Right.
What you really want to accomplish? Of course you should be testing what will happen during Black Friday before Black Friday, right? Yeah. But the ultimate goal is for your observability and your automation that uses that same observability to be able to react, right? Yeah. It would be silly to say. Imagine if Amazon would say, hey, we predict that for Black Friday we will need million servers. Right? Let's spin up million servers and have them running just in case.
That would make them bankrupt. Right. What they do have to do is that they tested things that are not necessarily related directly with Black Friday. Kind of. Okay, how many. What is the number? How many nodes can we bring up online if the traffic increases? And how much does it take to do that? Right? Oh, we can handle up to 100 new nodes with everything running new replicas running in those nodes within, I don't know, five minutes time, Right?
[00:27:27] Speaker B: Yep.
[00:27:27] Speaker A: If that's what you tested, you're ready, right? I mean, that's not the only thing, just to be clear. But.
[00:27:33] Speaker B: No, no, no.
[00:27:34] Speaker A: Basically you want to. You want to observe what's going on. Not you. Sorry, not you. You want machines to observe what's going on and be able to act accordingly. Yeah. On any situation that imaginable. Right. I mean, it's never 100% like that. You can never predict everything. But that's the goal really.
[00:27:54] Speaker C: But that's a great point. You want to see that your environment can scale up and eventually make it efficient and be. And make sure that the production is ready for at least as much as possible.
It's ready for their. For the real thing. Right. Because as you mentioned, you cannot match it 100%. And even if you could, it would make you bankrupt even if you're Amazon.
This is an interesting view.
[00:28:27] Speaker A: It's also a calculation, right.
It's easy to make something that is the most performant. Right. And what I said before, let me spin up million notes and stuff like that. It's also about the calculation. Okay. I want my users to have as good experience as possible, but not at any cost.
Right. So I would make a calculation. Okay. So yeah, I'm going to start spinning up notes not in advance, but maybe after I see that the traffic increases and that will maybe make some percentage of users have slightly worse experience. But that reduction of potential experience in the cost is maybe lower than the cost I would need to pay for, and so on and so forth. Right. It's the easy thing is to make it perfect. Not perfect, but kind of to make it, hey, everybody has great experience, but I need to balance that cost versus return of investment. Right?
[00:29:34] Speaker B: Yeah. You cannot either wait to have 100% CPU consumption, but you cannot do it at 20%.
[00:29:40] Speaker A: Of course. Exactly. Exactly. So at least in Kubernetes, I feel that it of course depends on the size, but I would probably aim for maybe 80, 90% resource consumption, kind of. That gives me a certain freedom without going crazy and saying, oh, 80% available for whatever happens. No, 10, 20% is the reasonable amount of available resources that you can say, okay, if things start going up, I have just enough resources available to keep me going until new resources are brought to the cluster. Right.
[00:30:21] Speaker B: And also very important that you should also consider the lower boundary for the downscale.
[00:30:26] Speaker A: Oh yeah, that's Usually what people forget kind of. Oh, we were very successful. We managed to survive Black Friday because we were scaling up and then a week later kind of. But did we scale down?
[00:30:40] Speaker B: That totally happened. Like everyone, everyone uses the, the HPA to scale up and nobody put the. The. The cooldown stuff. It's like there are two parts in scaling one it's scaling up. But then you also have to scale down your replicas because otherwise if you are spending the same time, the same amount in Black Friday than I don't know, a week after your. Your cost per user will be definitely a problem.
[00:31:04] Speaker A: Yeah, Black Friday Friday, big surprise. Weeks after that Black Friday we went bankrupt.
[00:31:13] Speaker C: Yeah, that's a great. With Guillermo the other day we were talking about the same thing. Where is the limit? Or what is that line that you say okay, this is enough. Okay. This user experience is enough. We don't need to be perfect as we say, which is giving. It is enough for the user to. To continue to operate in the system.
[00:31:33] Speaker B: To operate.
[00:31:34] Speaker C: Yeah. And the user be. Be okay eventually. This is only a peak times. It's not all the time.
[00:31:42] Speaker B: Yeah.
[00:31:42] Speaker A: You know, that's the same thing. Like when we talk about availability in general, most of the time we talk about how many Nines you have 99 point and then how many nines nobody ever. No. No sane person or experienced person ever talks about hundred.
There is no advanced company that tries to make it hundred because simply there is a. There is a point after which the cost is simply too high. It's not worth it anymore.
[00:32:15] Speaker B: Yeah, yeah, for sure. I mean you only try to put another nine after the. The last nine and that's it. That's what you try to do. So and continue with this like. Because we discuss a lot of topics and a lot of practices for someone that is maybe you know, having a cluster. You know, a standard cluster stuff. It's always running for Q and A dev and production. What do you think could be a practices to improve the resource usage and the resource optimization in general? Something that you have seen in the industry or something that you would do for a standard use case?
[00:32:54] Speaker A: I'll give you the answer. What I've seen and what I haven't seen. And I am surprised that I haven't.
What I've seen is more or less the topics we talked. Right. Kind of mainly around ephemeral usage of everything and about. Right. Sizing stuff. Right. And scaling. Of course. If you have to combine those three things, you are going to be good. Okay. Right.
[00:33:23] Speaker B: Yep.
[00:33:24] Speaker A: But the thing I am not seeing, and I'm honestly surprised is usage of cloud providers that are not the big three.
Okay, so we have aws, Azure, Google Cloud. And I do think that for anybody but small companies, they're the right choices because there are many things that they have that other providers don't like. For example, almost no other provider has three availability zones in the same region. Right. And without three availability zones, I cannot make it, I cannot make it highly available and it's normal that they have it and smaller ones don't. But many of the reasons why we want to use AWS or Google Cloud or Azure are not the same for production as non production. Right.
People can really get amazing results in non production by using Civo or DigitalOcean or Linode or Hetzner or any of those.
Now in the past that was not a good option simply because the effort and the work that we need to put to make same thing running in very different places was too high. Because hey, if I'm, let's say that I'm running on nodes and what's not running in aws and in DigitalOcean is very, very different. Right? The results are going to be different, the processes are going to be different, the automation is going to be different, everything is different.
But now we have a standard and this standard is Kubernetes.
[00:35:07] Speaker B: Yep.
[00:35:08] Speaker A: Right. If I. Now I know that Kubernetes in DigitalOcean and Kubernetes in AWS are not the same. I know that. But they're 99% the same. Right? It's good enough.
And I'm not sure why. At least I don't see combinations of those kind of, hey, let's go over there for production and let's go somewhere else. But it's much cheaper and at the same time also faster in many ways.
If you spin up a node in Civo you can spin up a cluster in literally seconds, while in AWS it can take like 20 minutes. And 20 minutes is not too much. If I'm setting up production because I'm not destroying and creating new production every five minutes, I don't care, right. How much it takes.
But when I'm spinning up ephemeral environments, I, I do care how much it takes because I don't want people to wait so they're cheaper and faster. Not necessarily as reliable, but reliable enough for, you know, development or pull requests.
[00:36:14] Speaker B: Yep, I totally agree with the, with the feminine system. And do you think. So regarding that, do you think that the overhead or when do you start doing that because of the overhead that it would take for switching to a new Kubernetes provider. I know you said that it now is less but there is still a little bit of an overhead especially with the connections. I would say that it could be more problematic to the new providers and stuff between applications.
So the storing the connection credentials for the cluster, all the setup for the CI CD maybe that kind of switches that could change from one environment, let's say one provider to the other.
[00:36:55] Speaker A: I mean there is always work involved but it's much, much smaller than before. Let's say. I'll take one of the examples. Right. You said keys to deploy stuff or something like that, right? Yeah, yeah. We use argocd these days, right?
[00:37:10] Speaker B: Yeah.
[00:37:10] Speaker A: And I just need to have Argo CD pointing to the point. That's it kind of like I have a cluster over there, I have Argo CD in that DigitalOcean whatever it is pointing to the repo. I just need to. I mean yeah, it will take me time to set it up but no, but now kind of like not a big deal.
[00:37:32] Speaker B: No it doesn't. Like as you said, I didn't, I didn't thought about the Argo CD for this, for this scenario. But I think it makes a lot of sense to treat him for, for GitOps let's say to do this stuff.
[00:37:44] Speaker A: You know that's security is very often the thing that people overlook and probably one of the biggest reasons for me for adopting GitOps and is that it is a pool based mechanism. Right.
I don't want my processes to go to a cluster and do something because then I need to deal with keys that you mentioned before and you were probably measuring keys from the perspective of the effort needed to do it. But there is also security part of the story.
Pool based mechanism simply works so much better. Right. I just need, I have my pipelines that are building images and pushing them to registry and doing whatever they're doing and when the time comes they push to git and that's it.
[00:38:32] Speaker B: Yep. Yeah. No, no, I totally agree that for this specific scenario and now that people are doing more multi cloud stuff and testing out new things, the GitOps approach can be so much efficient for, for, for these scenarios. Yeah, I didn't fall into that. I was more focusing like because since we. I've been doing like CI CD all my life. I, I am now adopting the, the githubs. Recently I was always like yeah, these secrets are driving me crazy where I have to put it. I have to set it up to a new provider the new connection is like, man, what's going on?
[00:39:06] Speaker A: Yeah. Secrets are driving everybody crazy. You're not the one, the only one. But even Secrets, right? So you know, there are so many tiny advancements that you know, when you combine them, they make such a huge difference. Like Secrets was a pain before, but now we have for example, external Secrets operator. Yeah. That you just need to deploy it in a cluster and it will be fetching secrets from wherever secrets are stored. And doesn't matter whether that ESO operator is in this cluster or that cluster. You just need to install. It takes few minutes. Cool. You need to give him credentials to go to your secret store, whatever you're using and you're done. Right. And secrets will automatically appear. You don't need to worry about them. They will just appear out of thin air.
[00:39:58] Speaker B: Yeah. And also with SOPs and KSOps and all this stuff that you can also like encrypt all the secrets within your code because it's already encrypted and then you can decrypt it on the fly when you deploy. I think we have found that solution as well to be one of the most effective. And nobody is fighting again, like where this is stored, where this has assets, you just encrypt it, leave it up there and then the pipeline or Argo CD will do the work for you.
[00:40:28] Speaker A: Exactly.
[00:40:29] Speaker B: Yeah. There is autonomous.
[00:40:31] Speaker A: Just not to be misunderstood. Might sound like I'm saying that it's no work involved. There is always work involved. I'm just saying that we got so many improvements, especially on standardization level that yeah, there is work, but it's infinitely smaller than 20 years ago. Like I'm an old dude that kind of like, you know, I lived in an era. I don't know whether maybe Damian lived in that era when we would go over Friday night and sleep in the office for two days because it took whole weekend just to deploy a new release. Yeah, we passed that.
[00:41:10] Speaker B: Yeah. Now the times to deploy are much, much less. And also the watch times, the availability is way easier to achieve. Definitely.
[00:41:22] Speaker C: One more thing that I wanted to ask you, Victor, is from what do you look into every day? I mean, do you work with the finops? Do you work on Dashboard? Any visibility, any things that you are looking into, or do you have your own way of looking into usability and cost?
[00:41:45] Speaker A: So I'm not much involved in whatever has the cost as a bird, that's not my focus.
But I am very much involved in things that are about optimizations. Right.
Which I feel end up having a lot of overlap. Right. I am very much involved into scaling, observability and so on and so forth. And I might be completely wrong because as I said, phenops is not my area, but I feel that those are very, very overlapping.
[00:42:26] Speaker B: No, for sure we can tell you that those are phenomenal and that's hugely impacting the phenotype practices. Because as I told you, Kubernetes is one of the hottest topics. And scaling down, scaling up, optimizing workloads, it's a core thing to do.
[00:42:45] Speaker A: Whatever we were scaling until recently is nothing to the scaling that is coming now with companies training models. Now we're jumping from hundreds to thousands or tens of thousands of nodes and processes. And what's not going to go crazy.
[00:43:05] Speaker B: Because now we are going to be in a whole new metric, of course, which is the GPU consumption that often is forgotten, especially on kubernetes. Like you always focus on CPU memory stuff. But now it's like distributed training to a huge scale and the amount of time and resources. Because I've been, you know, I was working on AI on 2008, which nowhere close to what we are reaching right now and there was still a lot of consumption. I cannot imagine the amount of money that could be invested now in. Well, I said millions in a train, like in a train round for a model is like millions of dollars. So what we are going to be now is going to be crazy. Especially if Kubernetes starts with the gpo, which is already there.
[00:43:54] Speaker A: No, it's, you know all those people that were annoyed that were not working in AI and were annoyed by you saying, hey, you should optimize this, you should do this. Now they're fine. Now nobody's going to ask them anything because now their expenses are going to be nothing compared to the expenses of the data scientists. Now. Now you're fine, you don't have to do anything, Nobody cares about you anymore. Now we need to go after those guys.
[00:44:19] Speaker B: Yeah, and now we are seeing it. That is a trendy topic as well. Like there is a duality in phenoms now that is AI for FinOps or FinOps for AI. Like what do you do? Do you start using AI to your FinOps tools or to your FinOps practices or you, you start doing your. Or the other part, that is you do phenoms for your AI. You're trying to improve your prompts, you're trying to improve your training, you're trying to improve your RAG systems, how much each user costs in terms of the usage of the AI and that's another whole level that is going to be there because AI is going to be a huge part of the IT budget in the next few years.
[00:45:02] Speaker C: There's both of them.
[00:45:05] Speaker A: We will see both of them.
[00:45:08] Speaker B: For phenoms and let's see. Because now people are starting to do agents which it seems to be rocking the US and everything that works in the US that's going to be like all in our YouTube feed every now and then. So it's going to happen.
But yeah, that's going to be. That's going to be a whole new level to the optimization and you know, also availability because you know to keep that training running is not easy to have that amount of information running and available and training and not having any incident while training.
[00:45:41] Speaker A: Oh yeah, it's going to be a nightmare. I mean it already is.
[00:45:45] Speaker B: Yeah, indeed. Especially which is good.
[00:45:47] Speaker A: We all get to keep our jobs. Great.
[00:45:49] Speaker B: Yeah, that's great, man. No, I think it's a. It's a great point to.
To close the stuff. So thanks a lot for. For having today's call. It's been a pleasure. And for those that don't know you, where you can say where. Where they can find you and you know, what stuff do you do online? Because it's someone to check on if you haven't.
[00:46:12] Speaker C: Definitely. If you're not following Victor, start doing it now.
[00:46:16] Speaker A: Yeah, I do many things. I have a YouTube channel like I.
There is a podcast that a colleague. Ex colleague of mine is doing where I'm always. I speak in conference. Just Google my name.
[00:46:29] Speaker B: Yeah, Google my name.
[00:46:30] Speaker A: You'll find me, I promise.
[00:46:32] Speaker B: Yeah, I can tell you you could find him. So yeah. Victor, thanks. Thanks a lot for. For being here today. It was a pleasure talking to you from. From the DevOps perspective and also from. From PhenOps to DevOps with Amiana. From DevOps to DevOps. I think it was Syntog man. So thanks for coming by.
[00:46:49] Speaker A: Thank you for having me.
[00:46:50] Speaker C: Thank you very much, Victor.
[00:46:52] Speaker B: Okay. And see you next time.
