[Music] all right um hey everyone. I hope you can uh hang around for a little longer. Uh and uh thank you for the awesome participation with the questions in the previous panel. Um so uh today what I'll uh dive into here with you is essentially sharing some of the learnings that we've had here at Catalon over the past six months as we've been working on our flagship AI-powered capability and true test that I briefly mentioned before. So hopefully this will be very valuable because I know many of you are diving into the AI world and testing AI-powered applications yourselves. Uh you might be experimenting with some AI-powered testing tools so hopefully some of these learnings that I'll be sharing here are things that you are also seeing from your perspective um and if not at least you can take some of those lessons learn and and and you know not repeat the same or not not go go

down the same path that we went, and then we learn, and then we pivoted. Hopefully, it saves you some time. So, um, with that, uh, let me just do a quick, um, uh, you know, a quick rundown on, uh, the journey as we see the journey in AI of AI in software testing. So, in the past, uh, we were focusing on mostly analyzing, uh, a bunch of data that has been generated, uh, by, you know, by different sources. So, for example, to be more practical, uh, we had a lot of test case results, right, uh, that are generated after test execution, either through a CI/CD pipeline or other, uh, other means, uh, but then we would, uh, take, uh, AI, different AI algorithms, and, uh, apply that to the to those data sets, right, in this case, the test results in the example, uh, and then based on those results, we would, uh, be able to predict, uh, what, uh, you know, predict different events. So, predict if the defa concentration would be more, uh, uh, based on those results, would be more, uh, revolving around a particular

functionality or a particular module in the application. So, it's essentially taking data that has been generated in the past, analyzing it, and trying to find patterns that would help us gain insights, basically. And that was good, it was great, but it was uh, it was the state of the art right at the time. Today, with the latest advancements in AI, specifically in generative AI, we're able to shift from that more reactive approach where we analyze data that has been generated in the past in order to try and predict, you know, something that might happen or will likely happen based on past, you know, historical results. We shift from that into more of a proactive approach where we are now effectively taking what we are learning from data and generating new output. And that's opening up uh, Limitless possibilities. And we covered some of those in the previous panel where things are going, it seems. And again, no, there's nobody that can

uh accurately predict anything. When it comes to generative AI, things are changing every day. Uh, but what we're seeing is that the most effective uh way of uh uh leveraging AI and software testing moving forward will be uh by focusing on uh helping testers uh achieve a state of hyper productivity. Um, and similar to what developers are experiencing right now, testers are lagging based on the reasons we discussed earlier. So, not only did a hyper productivity will obviously accelerate the testing life cycle, uh, hopefully, uh, or with the goal of, um, stop, you know, testing from being the bottleneck, uh, which is what usually happens. U and then, um, with testing not being the bottleneck anymore, then we're able to use, uh, our human power, right, our human brain power to solve more complex challenges and and think about. So, translating what, what complex challenges could mean is we could think about additional scenarios, more complex test scenarios that, uh, we should be validating in the application, uh, uh, that we're not able to.

think through those uh today uh so that's what we're kind of um what we've seen what we're seeing today uh and what we believe will be the next Evolution here uh in terms of AI and software testing and uh we based on our um beta uh experience here with true test uh we're seeing a lot of uh uh let me move my panel here to the other side um we're seeing a lot of uh not frustration well I think we could say frustration in the sense that uh we've been talking about uh Ai and testing and the possibilities of AI and testing for a long time right uh and uh people are reaching a point we see that in our customer base they're reaching a point of okay enough talking show me something right well are we is it going to be only talk about possibilities forever or are we actually going to reach a point where hey now we have something to show results to show so uh we're we're seeing the the the move right from those theoretical conversations to more of U early result

of applying AI or you know generative AI to different uh testing use cases and uh and one thing is and I'll cover some of those a little bit but one thing that has become clear uh as as part of that experience in running uh our beta program is that uh AI is clearly not a replacement for any um any uh activity that happens in in in the DAT day for for the tester. It's really uh a tool. It should be looked at at tool just like you have, you know, a hammer, you have a screwdriver. It's a tool that you're going to use as a tester to help you do your job better, faster, more efficiently, and so on. It is not going to, you know, make anybody useless, right? It's, uh, it's basically make you much more productive, much better at what you already do, and, uh, and it fosters that collaboration. So a friend of mine likes to say, Hey, we need to start thinking about AI as your, you know, your buddy, as your.

personal assistant, and that you can just as if you were in an office talking with your colleague that sits next to you. And you're brainstorming about ideas for creating test GES, for example, based on a certain requirement. You're spitballing, or you know, just collaborating. Think about AI as a helpful friend, right, that's going to help you achieve different or fulfill different tasks that you have in your day-to-day as of today. So, that's kind of what we're seeing. And again, as part of our beta program, we've seen proof of that as well. So, that's a good indication. And more specifically, about some of those Lessons Learned over the past six months is that, in this, we kind of covered during the panel the how developers look at those AI-powered use cases for themselves. They're much more open to embracing AI into their day-to-day workflows, right, into their developer workflows. And AI has proven

to be great at offering uh uh additional ways of solving a certain problem that uh maybe the developer doesn't have time to think through. Doesn't have the brain power right or the or the or even the skill in some cases to to to think through those um. From a tester's perspective, we are really uh going to a a place where uh uh you know if we're thinking about testing from a functional testing perspective or a non-functional perspective, uh where the the differences in skills from from a a testing perspective where you have you know the the functional tester, the uh performance test engineer, the security test engineer, and so on, uh those roles that today are very distinct because the skills required are very different, uh you know AI seems to be bridging the gap to help those different skill sets collaborate or those different roles collaborate much closer and more seamlessly, uh you know with the with the goal of shifting uh all of testing to go as much as possible, and we're seeing AI as enabling that happening, we're

seeing early results for that already. Um, and uh, one thing that still remains as an issue, and when we cover some of that as well in the panel, is that that variability of the output of a large language model is still a challenge for the testing community. Uh, there are different workarounds that we have implemented, such as caching that responses. And then, uh, if we send the same prompt, right, if we send the same input that we sent before, instead of going to the large language model, uh, asking for something to be generated based on that, uh, we provide our, you know, what we cached before. So, that's not a solution, it's a workaround, um, and uh, but in certain situations, it's a good enough of a workaround, right? So, kind of sharing some of the inside info for some of the lessons we have learned over the last six months, um, and uh, and that workaround has helped some of the teams overcome that skepticism in terms of, you know, whether they can trust or not or how.

much they can trust or not. Uh, what AI is generating for them in terms of, you know, a testing artifact. So, it's, it's really, um, uh, we're in the infant stage here of the possibilities for the testing community. Uh, and there's a lot. Well, let me rephrase that. We're in the N, uh, you know, the infant stages. Uh, however, there, there, there has been. You can apply AI to all, you know, use cases you can think of from a testing perspective. Uh, that's not a problem. The challenge is whether the testing professionals will embrace those use cases that are powered by AI because of that lack of repeatability, that lack of, uh, auditability, explainability like Sergio mentioned. Also, correctness is an important critical factor. Uh, so what we're trying to focus here at Katalon is making sure that the use cases that are powered by AI are use cases that testers will embrace, right? They will trust what those output are. Uh, the outputs are and they will, uh, uh, U take that and augment, right, and then

become or reach that hyper productivity that we keep talking about. Um looking forward here. Um, the other aspect is really, um, if we think about manual testing specifically, how use case we're also exploring and are going to go in beta soon is that, uh, how can we, um, help the manual tester, uh, or anybody doing manual testing, excuse me, um, uh, do be able to do that much more efficiently, right, because the whole point of test automation is to really, uh, enable the testing life cycle to happen much faster, right, so you can release a release faster to production, um, and therefore, uh, be more nimble, be more agile, uh, for your business. Um, manual testing is a challenge in that goal of speed, so test automation has been the de facto solution, but what if we could use AI to help manual testing go much faster, help manual testers become hyperproductive, uh, become much more efficient, and not slow down the testing life cycle. So that possibility is something that we're seeing already.

early results that in some in certain cases some you know some teams might uh might be okay with just having super fast manual testers right not necessarily having to think about automation uh in the traditional sense so those are some of the possibilities we've been exploring and and again all about augmenting or or enhancing the current capabilities of our platform um seeing some good results so far um and then in terms of future where where things are going is uh is really about uh enabling the team the the the testing team or the quality engineering team to build confidence in uh in uh in the outputs of AI through collaboration right obviously if you ask the user you know the tester to do to provide everything to to the uh AI system in order to generate good output then it kind of defeats the purpose right there has to be a a balance between how much you ask for that collaboration and what you get so uh we've been fine tuning that in in new use cases in addition to true test that I mentioned earlier uh to to to find that balance uh in in in some

in a way that's effective and that's uh meaningful to our customers. All right, so the data privacy is a huge issue that we have been dealing ourselves as we venture into the AI space. Obviously, if you leverage the open uh or the publicly available LLMs out there, there's only so much you can do uh with them because obviously for you to get an output, you have to share data, right? And when you have customers uh like we do, uh we are not able to send uh data, even you know test case related data, test results related data, send that to a public uh large language model without the our customers' consent, right? And many of our large uh customers are uh basically blocking their uh their entire team, their entire staff, the whole company. They're blocking them from using um uh public, publicly available large language models like OpenAI Bard, right, and Google's Bard, and so on. So how do we in that environment make sure that we can still add value when our customers are you know

their own policies prevent any their own employees, and this, in, you know, uh, and also their vendors, of sharing their own data. So that's a, that's a challenge, totally understandable challenge, but what we've been finding, um, is that, uh, understanding, or, uh, getting to that balance, is something that, that is, uh, is extremely hard for, for our customers, in this case for you guys that are, that are here with us in the, in the session, uh, and it's understandable, right? So in many large organizations that we work with, and, and as we have those conversations about what are we sharing, you know, with, uh, OpenAI, for example, so that we can get, you know, automated, uh, tests back, uh, in those conversations, we, we say, hey, all right, so what are your, you know, data privacy policies, uh, around large language models? And the interesting thing is that the response we get back from the security architecture team, uh, compliance teams, is that they are literally writing those policies right now, uh, so there's no, uh, there's no, uh, there, there's not enough experience.

in anybody's uh Team uh you know, whether it's legal or security, you know, compliance, there's not enough experience in anywhere to uh, to basically say, here's how you deal with with data privacy, uh, and data sharing, um, and and securing that because um, for the company to achieve the the, you know, the outcomes they're hoping for, they're hoping to achieve with AI, they themselves, our customers, they have to share their data. So what's happening is we're um, we're hearing a lot of feedback. Again, talking specifically about feedback, we've heard during our beta program is that um, those companies are creating their or they're creating, you know, pilot projects where they are essentially deploying open source large language models into their own premise, right behind their firewall, secured, so that they can now fine-tune that large language model, uh, with their own data, right, with their own IP, their own intellectual property. So if it's a bank, for example, um, they are deploying a large language model behind their firewall, highly secured, and now they're pumping

data right they're training they're F fine-tuning that larg language model with the with the bank information uh which include their IP right intellectual property Trade Secrets and so on and so forth so obviously that information is never going to go out you know to a public uh largest language model and then what they're asking of us is that whatever whatever we build on our side uh to um you know for for their for our catalog uh users uh make sure that it is portable in the sense that hey today we're pointing to open AI right true test for example uh we're pointing that to interact with open AI but when we have our own uh large language model model operational and working I want you to switch and point uh Your solution you know your catalon solution to our own large language model within our premises highly secure and so on so uh that's something that is has been tremendous uh tremendously beneficial for us because obviously that changes the way we ar contct our solution right uh so that we

we can whenever we go ga. Whenever we release um, uh, you know, uh, our solution, uh, to, you know, general public, uh, uh, it is already ready to support, uh, our, you know, the customers that already have their own large language model. It's portable, right? Um, so that's something that, uh, that was a valuable lesson and, and also humbling to, to learn that most of those security teams, compliance, legal, like, they are literally writing the policies, uh, right now, and, and it's not something that they will do or they are doing one and done, right? It's something that they are continually, continuously iterating as they learn more, just like all of us. We're all learning together. All right, cool. So, what, uh, what also we've learned, uh, so far is that, um, uh, we, we are, we have seen enough evidence now. It's no longer theoretical as we, as I was mentioning in the beginning. We've seen enough evidence that, uh, if we apply AI to the right use cases, uh, testers can really become hyperproductive and, and, you know, keep up.

with uh with the pace of delivery we've seen that during our beta program and we're really excited to uh to get that validation because it was something that we were uh questioning. Can we actually pull this off? And we are seeing uh results, not even early results. We have seen results, and we're really excited about that. Um, we've seen that, um, in general, our customers expect, uh, you know, AI, uh, tools to, uh, provide accelerators to the organization, provide insights, and, and ways and different ways to optimize their existing testing processes. So, um, so that's something that we've also been able to accomplish in terms of results, uh, so far, um, and again, we, we mentioned this in the panel earlier, but it, it, it again, very clear that AI will not replace testers. It will make them super productive, supercharge them, super testers, superhuman powers. Um, call it whatever you want, but it's all about making, uh, making them better, right? So, um, and for that, it requires, uh, the tester to continue to be, uh, in the

driver seat, uh, like in the early versions of what we were doing with True Test, we were, uh, basically, uh, having it operate in a very autonomous fashion. Well, and what we've learned is that, U, there has to be a human in the loop, right, uh, not only for the output to be better but also for, uh, the, uh, the solution itself to be embraced, to be adopted, right, um, and when the human is in the loop, the output is so much better, uh, it's, uh, it can be refined, and that collaboration, uh, is something that just enriches the output. So looking ahead, um, what, where are, where are we going here in 2024, uh, so not, uh, basically in the next 12 months, uh, what we're doing is, uh, we are making sure that, uh, you know, QA teams or quality engineering teams, testing teams, they become more comfortable in, in interacting with, uh, with an AI solution, uh, and, uh, and hopefully get that, that apprehension, uh, you know, slowly fading away. Uh, I think there's a lot of, in I saw in the chat in

in the question box as well there's a lot of um interest and excitement and curiosity. No pun intended then but a lot of curiosity uh around oh what are the use cases we can apply this where where can I you know see things working etc etc. But the reality is those are the early adopters uh most organizations large organizations uh that we work with um that there is still a lot of apprehension like banking Insurance health care right highly regulated environments how do I even think or entertain using a large language model right um and uh. So it's really about making sure from from our perspective it's really about us making sure that our customers uh are no longer apprehensive right that they are uh they are becoming more comfortable they are willing to to try new things and and our goal is really to make that process much smoother um and uh to make them comfortable. We also see that by partnering up um um uh we we with with service providers that are uh creating uh their own large language

models or deploying their own large language models and training. You know, fine-tuning those large language models, they specialized in specific domains. So, think of a large language model for banking, Financial Services, an LLM for travel and Hospitality, an LLM for healthcare, for example. We're seeing several Global Systems Integrators and other service providers building that themselves to offer their services that harness all the power and all the data, all the intelligence from those specialized LLMs and making sure that our solution is, as I mentioned earlier, architected to work with those is something that we've already done. So, think of it, hey, if you ask Chat GPT, for example, to generate, you know, give it a requirement from a banking system, a mortgage approval system, for example, give it to OpenAI to generate test cases. Or now you give the same requirement to a specialized LLM that has been fine-tuned with banking, Financial Services data, you can see the

the power, uh, the power in that, in the sense that the test case is generated by this specialized LLM, uh, is going to be of not infinite, but you know, exponentially better quality, uh, in correctness, and more meaningful than a generic, uh, LLM like OpenAI, uh, so that's kind of coming in 2024 and being ready for that is going to be paramount for our customers, um, and then that's how we're going to move away from that skepticism in adopting any, you know, uh, public, uh, LLM is that, uh, really ensure that, uh, uh, organizations, of, you know, in this case, our customers, are accepting, uh, that, um, they can trust, uh, the usage of large language models in testing, and, uh, and, and we'll accomplish that by sharing those early success stories, uh, and use cases, uh, that, that will, uh, you know, start helping people become more comfortable, uh, and we are seeing right now, uh, the role of AI testing kind of shifting from just that experimental, uh, level to becoming

foundational for us at Catalon AI is becoming foundational in the platform because we see it as the way to take our customers from their current state to their desired state, which will augment the capabilities they already leverage from Catalon. They have been leveraging from Catalon. So, AI being foundational on the platform, or in other words, being inherent, being part of the platform cutting across several use cases, is going to accelerate the journey to the future state that our customers desire. And luckily, we've already been able to see results in the past six months that are very encouraging, and we've gotten great feedback from different customers.

[Quality Horizon 2024] Revolutionizing Testing with Generative AI

Scripts