The technology that guides us: Gen AI in practice and the future

Pierluigi Costanzo
I think that the future is something which we can call a proactive AI, or probably in the case of AI would be more like hyper proactive AI in the sense that we will have all these agents and system that they will be constantly analyzing data, gathering information, running queries from the database autonomously. And they will be actually prompting us, I say like this, by just telling us, hi, Szymon or hi, Pierre, based on this data, this is and these other things are happening. And then they will be asking us, which is basically prompting us again, what do I do now with those information?

Szymon Głowania
Słuchasz trzeciego sezonu podcastu TechChatter, audycji Capgemini Polska, w której zanurzymy się w świecie technologii. Posłuchaj rozmów naszych ekspertek i ekspertów, odkryj projekty realizowane w Polsce i zobacz, jakie innowacje, które współtworzymy, kształtują naszą przyszłość. Przekonajmy się, że praca w sektorze IT może być naprawdę pasjonująca. Gotowi? Zaczynamy!

Hey, good morning. Today we will be speaking with Pierluigi Costanzo, one of the experts from the Capgemini. And we will talk about the GenAI, preparing some prompts and all funny topic connected with this tool, with this algorithm.
Usually at work, you will try to prepare a best solution we’ve used the AI for our company. What can be very difficult in current time, but we try to do it. So maybe today we will speak a little bit about the preparing some AI tool, maybe on the start. What is the GenAI for you?

Pierluigi Costanzo
Hello, Simon. So GenAI for me, that’s actually a really good question. I’ll take it as a tool. Yeah, I won’t even call it a GenAI model. The model behind it’s just AI model. It comes as a tool, right? We use GenAI for building applications that serve end users, improve productivity, make our work faster, hopefully better. And we can discuss the hopefully later.

Szymon Głowania
Okay, so we use this tool, yeah. Everybody needs to use some AI tool. All company or part of company needs these sentences and this part of tool in some solution, because it can be smart, very useful for the user and for the company. So, we try to improve something. And usually, we try to do this with some, for example, GenAI tool. When we’re preparing this conversation with our assistants, because usually it’s something like assistants, we try to prepare the prompt. How does it look like when you start the prompting?

Pierluigi Costanzo
Usually when I was going to some conference and presenting something about GenAI, I always say something like, GenAI know what to do, but it does not know when to do. And this is the reason why we prompt, right? So it’s not like if we build an application that is like, you know, sentient and it can, you know, kind of like take over or just do whatever it want. But usually when we need something, we go in a conversational mode, like a chatbot powered by an LLM, things like this, and we ask a question or we prompt.
So in the moment in which when we do that, there’s the moment when we tell to an LLM, now you have to do this. So that’s why it does not know when to do it. In most of the cases, it knows what to do because we just ask it for it. I don’t know, like summarize me this document or search on the internet for this and that, generate a chart, generate a piece of code, or I’m looking for this information and or help me out brainstorming. And I think an important part is the way how we prompt or actually the way how we ask.
There is a topic around the anthropomorphization of AI. That is something connected on how we interact with other people, like our colleague on teams, and our family on some messages, or WhatsApp messenger, other things, other platform for chatting, or interacting on forums we have with proper comment and reply. And when we have a chatbot, although we know exactly that it’s just powered by an LLM, so it’s just behind just a machine that predicts our token or something like that, we still use hi, right? We say, how are you? Or we just say, or we just say, please.

Szymon Głowania
Thank you! It was a great code.

Pierluigi Costanzo
Exactly. And actually that one, it’s a very important thing because even when we are going to build and we are building an application for our client, for example, this is an important thing to keep in mind. If you think about the way how this large language model has been trained, like they pull out a lot of data from public sources, from different conversation, which basically are human conversation. So if I would ask you something and I will be very, very rude, how would you react to it? Would you be cold or would you be, oh yes, now I will help you, of course, but…

Szymon Głowania
People usually Don’t like do it, yeah?

Pierluigi Costanzo
Exactly. So it will just probably not answer to me or just like yes, no or whatever But if I will ask you kindly and I will say, Hey, Szymon, how are you today? And then I will ask you, please, can you help me with that? Or, you know, can you look at this file that we were talking about later?

Szymon Głowania
Yes, of course. Can we meet?

Pierluigi Costanzo
Right. We can. And the same is with the prompting. If model has been trained with this kind of data, right? If you ask nicely, if you ask hi, if you ask please, you actually get more information, which in other words, more verbosity sometimes. Although you might still end up being maybe not kind of rude, but kind of like straightforward and say give me this or give me this information or write this code. It’s still gonna do the work. Maybe not as good if you will ask politely.

Szymon Głowania
So the prompting is a way how we can get something from some GenAI model. We learned this model with all data, which we have, but we still should remember it’s a language model. It’s prepared for the documents and sometimes for another part, but it’s still connected with these documents. It’s not a thinking model. In something like that, people know about the thinking. It’s try to predict something, help us, but it’s another way to prepare the resolution like people prepare. When we try to prepare a good prompting, so we try to get the way for the bot to the best resolution for us, we try to prepare this prompt in some steps, sometimes all in one message.
When you’re thinking about how to ask the GenAI about something, about the language, how it works with this model, do you have some tips on how is the best way to start this prompting?

Pierluigi Costanzo
Yeah, I’m more a big fan like based on experience of the project that we built of more like divide and conquer mode, even with the prompting. Of course, there are usually some that they prefer write everything in one prompt, and hopefully you will get an answer. Hopefully you ask for a very big prompt, I’m looking for this and this and this, with lots of instruction, break it down by this and these other things. Although for the last 2 years that we have been building conversational AI, of course the models of course get better and the beginning was obviously very good at the beginning of 2023, not as good as the latest model of this year.
But when you are more in a conversational mode, you don’t really have to think about writing like a huge prompt. You actually conversate. You will ask as more things. You will get an answer. And then you might give a feedback, right? Okay, this is not the answer I was looking for. And then maybe adjust it in this way. And then you ask something else which is connected to your previous one. And the reason why this one, I think, is giving much better result is because when we do that, we are adding context, lots of context in our conversation.
Which part comes from you, part is the output from the model. By adding more context, it becomes more clear what we are looking for. If we will start with a very large prompt at the beginning, or we are 100% sure what do we want, how do we want it, which is usually a big problem for people to know exactly 100% with clarity what do we want, how do we want it, and even put it in writing.

Szymon Głowania
Yeah, when I write some prompts, usually I try to remember it’s a young child, 5, maybe 6 years old, so you have to get all information, how do that, what you need, how results should look like. So all these things should be added, because AI try to predict some of our needs, but don’t do it for everything. If something, it can be easy to apply by them, do this. And return something what sometimes will be something different, like our needs. So we try to help them to prepare some resolution. When we prepare the prompts, we should remember about the context.
And usually and right now, the context and how long context GanAI remember is one of the KPIs of this tool, because we can prepare a lot of messages, give some knowledge about this part of knowledge. It’s something that the GenAI has to learn to help us. And sometimes when we have the shorter, it will be better, but forgot about some first steps. But when we have the longer context, we can use more message to this GenAI tool. So it’s the part of preparing the conversation and some part of the elements of working the GenAI.

Pierluigi Costanzo
This is an important part. The context that we add around, of course, it helps with the prompt. And there is something with something that’s just called context window. So this is kind of like a limitation of any large language model, if you think about. So they can only take in one time certain amount of information. In other words, more to use a GenAI jargon, or they can only take X amount of token. Some model has more, some model has less. This relevant is still the context window that is used. What does it mean is that when you add your question, your initial prompt, your output, and then another prompt, and then the output, that’s all fill an entire context window.
And usually when that context window finish, That’s all the information that an LLM can take. And that’s where techniques like, for example, RUG, retrieval, augmented generation, or using agents, or using sliding context window come into place. And that is why, for example. I remember when we started to build from last year, some of the from some of the tool for our clients, at some point, we realized that I think that apart building technically and having a nice UI having a good model behind, and then employee RUG and employing agents and other things. One important part was doing workshop for end users, actually how to use this GenAI tool.
And which it kind of also end up doing, of course, workshop on prompting. But lots of things was also like to try to make them understand, not only how to prompt, how you need to be clear in your prompting or split it up and so on and so on. But also try to give them as many information so that people can logically understand how to interact with a chatbot. And one of the things was exactly this context window. Try to explain that you can have a very long conversation, but at some point, due to the fact that there is this limitation of number of tokens and information that you can fill in the LLM, the initial information, your initial question, they started to be forgotten because they are not part anymore of that conversation.
So to be very careful on how long the conversation and for how long you can employ you to remember that if you then refer to something that was written at the beginning of this conversation, the model does not know anymore. Of course, now we have, there are techniques to trying to remember some part of the conversation. You summarize all the initial conversations to keep them in the memory so you can always refer even if it’s very very long. But these are all techniques that kind of adapt to this context window.

Szymon Głowania
OK. I think the next element which we have to give some knowledge about this, it will be a Rug agent. What is something new and what can be improved in this one?

Pierluigi Costanzo
So maybe let’s take them kind of like separate. Yeah, we take a Rug and we take an agent, and then we’ll discuss them first separate, and then we put them together. Usually, I try to get some example or some metaphor on how to explain what might be a Rug and what is an agent. The one that I use for Rug is like imaging an open book exam. So you got your question and you open your book, you scroll it, you find the paragraph that you need to answer your question, you read it, and then you give the answer.
And that’s in a very, very high level what RUG does. So when we ask, when you prompt a question, the model employ a RUG where there is lots of information, they are in a vector database, and they are indexed and everything. We use different technique on how to retrieve, so how to extract this information for a database till we find the relevant one. This relevant one is that there’s a context in our conversation, and the model, knowing this context, it giving us an answer. Now, the importance of RUG is that usually when we have an LLM has been trained on certain amount of data, it does not know everything.
So it can be something that out of the knowledge of the LLM and thanks to this information that has been retrieved, we can get an answer which is way less hallucinating. Yeah, the problem of hallucination of the LLM because it’s specific to a context that has been given to the LLM. This is really powerful. And in fact, the interesting fact that when Meta developed the Rug system, it was not for LLM. It was for training a model, an AI model, deep learning model, by basically employing some retrieval throughout the training. And they just discovered that this was improving the performance of this model.
And then the same technique has been employed after to the LLM in this way. And the second one are like the agents. And I think everyone holds heard even from the Capgemini last 2 years report about the agentic system, multi agentic system and things like this. And the example that I always use for describing an agent is always like agents are like entities, small entities, which are part of the conversational application that we build. They are connected to something which we call orchestrator, which is basically the main large language model where the user conversates. So if you think about how we are biologically designed, yeah, we have our head, we have our brain, we have our nose and ear and hands and feet.
And, you know, if you need to pick an object, you will probably use your hands, right? If you need to walk from one place to another, you will stand up and then with your feet and your leg, you will walk. If you are hearing something, you will use your ear. You will use your eyes to see, yes, your mouth and your tongue to output my voice like I’m doing right now. And although this one will be just recorded, I usually also just gesticulate. So as I’m Italian, I can also have a conversation with my hand if you think about, but that is just a bug in my DNA code.
But the point is jokes apart is that the same happened with the agents. So all these small entities, they do very specific things. They can do very specific task. For example, searching internet, running a piece of code, choose which is the proper database to then do Rug.

Szymon Głowania
Some mathematical calculation. 

Pierluigi Costanzo
Mathematical calculation, exactly. Calling APIs, for example, knowing what is the schema and everything, calling an API, many other interactions that can be presented. And so what happens is that we prompt a question, the main LLM orchestrator decides if it makes sense to employ one of these agents, and then run the agents, get the output and use that again in the context and give us the answer.

Szymon Głowania
Usually when we think about the GenAI we think about the people, human, which we can speak with, see, we can give some tips, give some question and would like to get some answer. So we would like to communicate, usually like people so we can speak, sometimes writing. And our GenAI tool should have some small agent to get this chance. Because when you would like to speak, we need another model to get this conversation to the text. When you would like to search something, when we would like to add some numbers, when you ask usually some GenAI tool and 2 plus 2, you get different answers.
So when you have the smaller agent to the more specific elements, it will be better for the resolution.

Pierluigi Costanzo
Exactly. And usually, especially the last year we used a lot for insights. So like how an LLM and conversation thanks to agents and in this case doesn’t have to be one, can be many of them, can speed up, for example insight, right? If we think about when we need to do insight in an organization and we need to go from different source of data to, for example, put it in a dashboard or create visuals, create a report, especially when the data is very, very big and the information like lots of informations that require a lot of data process, kind of like a hard job for data engineering, especially when there is lots of data to process lots of information, you need to write a lot of queries, you need to build a data pipeline and then, you know, if you need to even add some AI model to do maybe prediction, classification, and this adds up even in your data pipeline and then all this data already at the end, clean it up and everything goes to data visualization expert to build a dashboard or you build a report, you use a business analyst to understand this data.
And right now each of them can be basically replaced by an agent. You can have an agent that interacts with a specific database, write your SQL query based on what you ask just by prompting in English, in Polish, in Italian, whatever language the model can understand. And it transforms that knowing that you are looking for certain information, that they are in a certain database, transform this information into SQL query, retrieve this information from the database, like run the SQL query for you, and then give you the answer. And it can do even more, right? It can then pass maybe the data into an agent that can generate a code for visualizing this data and it even returns you an image, right?
Of course, this is what my sounds like that it’s replacing or something, but I will say that it’s more like even shaping a lot of roles, right? For example, data engineering, especially on a lot of these mundane tasks and data pipeline, they will become obsolete and done by agents. But actually, if you remember before, we talked I say, RUG, and I also mentioned Vector Database. And a lot of the work of future data engineering, especially in AI, it become around vector database, right? Vector database is not only you just can pull any amount of data that create your embedding and something, but it requires specific techniques on how you index the data, how many vector database you create if you need different one, not everything in one, there is not one size fits all.
And the point is that although we can replace a lots of things to make it faster, yeah? Lots of jobs will get faster and better. Some other role will be shaped, probably in something even more advanced, like in the case of engineering, I would say. And to conclude, as I said before, yeah, we had, we talked about the Rug, we talked about the agents, then actually we also put it together. I’m actually, I’m not a big fan, and this is probably the last 6-8 months on all the application that we build. We push it also together with the team to change all of the RUG-based system into first agent system and then RUG.
So what does it mean is that it’s better for an LLM to first just have an agent that can understand where to find information. Let’s say this vector database has access to, I don’t know, your internal data. This other has access to other specific internal data. So it first choose the database, the agents, and then employ RUG to extract the relevant information. I think this has like a lot of pros compared to just purely Rug system. So the usage of the agents that decide how to use a Rug in a sense that on a Rug system, wherever you would be prompting the main LLM orchestrator, it always has to retrieve something.
But sometimes we might ask just, hi, how are you? And there is nothing to retrieve in the context, right? So when you do like that, the agent will understand that you don’t need to retrieve anything. And only when you ask a very specific question, it knows, it understands, or let’s use the word understanding, it understands what to retrieve, right? So what to do and when to do it.

Szymon Głowania
Yeah, so I think it will be helpful in this approach. Usually when we prepare some joke about the GenAI, the Rug will be helped because one of the parts of this joke connected with the return answer always will be a better answer for this question or we don’t get an answer for another question, not ours. And the part of some agent we can specify in grades, we do some specific rules and after that combine together and get a complex model to get all these specific things connected with our needs. When you’re talking about it, I have some questions about, and you use these sentences probably 4 or 5 times.
So what do you think about the prompt engineer and this position? It will be something that will be developed or everybody should know everything about the GenAI and try to get this knowledge?

Pierluigi Costanzo
Maybe let me ask you a counter question. Do you think that in an organization, do we have today as 2024, almost going 2025, do we have specialized persons that can work on Microsoft Word? Or like everyone more or less knows the basic of that?

Szymon Głowania
Yeah, so we have the same opinion on this topic. So I think the basic knowledge about how this works, what we can get and how we can use will be necessary for all because all companies need to use this AI tool because the markets give this information we have to get the answer faster, better and etc. Usually we try to improve it and we can do it easier with some AI too.

Pierluigi Costanzo
Exactly. So exactly. That does not mean that prompt engineering is not an important role or something. But because usually when we may be on a very, very low level by low level, I mean like on the technical things that training a model and designing it, it requires someone that is really good on prompt engineering. Prompt engineering can be also coaches, right? We have lots of people that they need to still adapt. We need coaches, basically, engineering, prompt coaching, basically, that can help in the organization to understand how to use this tool, especially if we are going to employ much, much, much more for productivity, right?
Like in the past, we always had people that can help us on some other productivity tool, how to use a word, how to use properly at PowerPoint, how to use a properly in Excel, right? So you always have people that they’re coaching on this kind of stuff, learning that will make them prompt engineering or prompt coaching. Of course, when we come to a very specific system that require maybe let’s call it, it might sound negative, but it’s not negative at all, like hidden prompt that can help the entire system or GenAI system to work for the user.
Sometimes we have prompt that we don’t see and those are very, very advancedly crafted prompt that would require prompt engineering to work with it, right?

Szymon Głowania
So when I think about the prompt engineer, usually I see some guy with knowledge, maybe not big, but the basic knowledge about how this model works and try to give some more knowledge because we can use some pre-learned model and try to implement this small learning in some project. But something would you tell, it can be more useful. Because when we think about the shared knowledge about the preparing prompt, usually these people should know about this GenAI. So this part, from my opinion, will be done. But the shared knowledge in the company, because in our company we have a lot of training, so we can think about preparing the prompt.
It will be the next part. Because we have a lot of trainings connected with AI and some GenAI tool. When we talk about how we talk to the GenAI, so some specific thank you, etc., sometimes we give feedback – It was the great code. So I think about something, what we can give feedback to the model to learn from our answer. Because we have some system to give the feedback for the model. But usually, it’s some button to click. And after that, somebody try to use it to learn model to the next level. But I think about something when the model can learn in online, when we speak about something and after that use this language.
So get this feedback from the context and from our message.

Pierluigi Costanzo
Yeah, that’s usually very much a part of…

Szymon Głowania
It could be the next agent.

Pierluigi Costanzo
It could be the next agent. Like, you know, it understand that you’re giving a feedback and then maybe employ something else. When we add all together, like the Rug, the different agents and the prompt engineering and the feedback loop that you just mentioned with the human, not only like a thumbs up, thumbs down button, but actually like written feedback. All of this one, they actually, they help us to steer the main, the foundation, also called foundation, LLM of the application. If you think about it, there was always a risk a bit of like homologation. So If you think that there is, imagine that everyone will always interact with the same LLM.
Of course, there is always some randomness. Your LLM can always return slightly different answer. But Imagine that every billions of people on earth will start to use exactly the same LLM. No different prompting, no agent that can do different things, no Rug to add the specific information that does not know. And actually I will add even more, even with prompt engineering and RUG and agents, I think it’s important for the organization to also think about using different language model, not stick only to one for different reason. It can be from, of course, from bias reason.
So you need to have for homologation reason to not have everyone around the same model that return the same single things for efficiency. When I say each agent is an entity, but each agent doesn’t have to only return a code, doesn’t have to do an API, doesn’t have to do only, an agent can also employ another small LLM to do some other task, right? I don’t know, understanding your feedback, for example, maybe give you an answer if I like or not, your feedback, something like that. Or this is just a fun example, but it can be anything else.
And especially when we have agents with other LLM, we probably would like to have like maybe small LLM, not that costly. Imagine that every time I need to pick an object with my hands, I’m consuming, I don’t know, a thousand calories. Well, probably we wouldn’t be very, very, very efficient, right? If every time I move my finger, I need to consume a lot. So that’s, and having all these different elements also help a lot on the homologation.

Szymon Głowania
Yeah, so we think about they have some generative but big model to conversation and communicate between some more specific agents and prepare some model with the more specific knowledge and to give some feedback, get some elements, return something, calculate something connected with the specific topic.

Pierluigi Costanzo
Yeah, exactly. We had the very big LLM that we have now. They can do many things, but let’s say that I need small agents that only has to return SQL or piece of Python code. Do you care if it can answer the meaning of life? Probably no. You only care that it can answer a very good Python code, a very good SQL query, right? So very, very, very, very specialized and probably very also very small because it doesn’t have to be able to do thousands of things. It only needs to be able to do one single thing and really, really good.
To add more of this one, how I come out with this thoughts of a location, I also did a post on LinkedIn. I basically created a conversational AI with 2 same model, so model A and model B, which they were exactly the same foundation model. So what happened, I give them an initial topic to trigger both of them. And then both of them were basically talking with each other, interacting with each other. And the point is that it didn’t even matter what topic I was giving at the beginning, it didn’t even matter if I even change a bit the prompt behind the 2 models, at the end, they will always agree on something.
Always agree on something. After some iterations, speaking with each other, they were just like, yes, I’m on LLM, I can answer this question, this is what I think and the other not, this is what I think. But I was like, okay, maybe you’re right and something. And then at the end, they agree on the same things. And I could give them any different topic to talk about. And they would just always agree.

Szymon Głowania
So when we think about some specific movie from the past, I’m robot or Terminator, usually AI agree.

Pierluigi Costanzo
Usually AI agree. I don’t think we’re going to get to that level, at least not with the LLM, right?

Szymon Głowania
But learning from the simulation, but between 2 or more agents, it’s, in my opinion, something what we have to do like a new feature right now. It’s something what was the game changer, the great steps for the next level. Because right now we read a lot of documents from the internet and from some communication, etc. So this knowledge was discovered a lot of. But to get the next steps, something like AlphaGo, when we try to learn some game to the agent, it will be learning by the conversation between some agents. We return something new, for example, proposal to the models for the programming or some specific steps to do or divide some elements for different classes, etc.
Because we can learn from our knowledge, but this knowledge is something more like people produce because it was something prepared by the computer. Few moments ago, you talked about something to prepare and communicate the agent and learn from our conversation. So we have to remember about something like that. We have a lot of topics which we don’t like to talk about it because it’s something connected with some political or the history elements. And it’s something that, for example, would like to learn this LLM. The user from the internet, the first steps, learn the Nazi for the agent.
It was something that was prepared a lot of years ago from the first bot, probably for the Microsoft or something like that. So some topic should be in another pool, another part of this knowledge and we try to prepare something that will give this wall for this AI. So something we try to do always works.

Pierluigi Costanzo
You mean about this content filtering, right? When you ask something very specific, then the the LLM always returns a very polite, correct answer, like I’m not able to give you an answer about around this topic or we shouldn’t. A very good question.

Szymon Głowania
It’s something from the conferences.

Pierluigi Costanzo
I think that the topic of bias is probably touched at least a bit around this topic. I think bias has always been seen as a negative term, the worry that the model is biased, the worry that we are biasing our result. Although if you think about bias, we’re all biased due to the way how we grow up, due to the information that we see in TV since we are little, due to the information that we see on the internet, the news that we read, what news we read. We are biased in a way that we have more knowledge maybe about football because someone likes more football, more knowledge about the basketball because someone likes basketball, so you are biased towards that sport because you follow more.
Usually bias, now at least in LLM, is being used and it’s a very negative term in a way that the model should not be biased toward certain things, like maybe certain topic or something. So as we are ingesting certain information into an LLM, we are ingesting them when we train it. Let’s always remember that there is a statistical model behind, right? So the more information around certain topic in a specific way, they are the one that they are being biasing basically the model. So if the more information that I gain something or someone, then the model will be always prompt to give you an answer again, something or someone if you ask for it.
And that is why there is something which is called content filtering to avoid certain topic. Although in some cases, bias might be something positive. And sometimes being biased also means respecting of different culture. Imagine that we want to use maybe not a text LLM, or it can be also a text LLM. We want to make a campaign, a marketing campaign, we employ a model with image generation and even like a text generation, we want to make a campaign for a certain group, for a certain target group, for a specific community or a local community, a specific country, then maybe they are a minority.
If a model wouldn’t be biased toward that specific group, there is a very risk that the image that is being generated and the text that is being generated might not respect the value and the culture of this minority. So to answer in all, it’s like a content filter is more used to avoid problems. But sometimes we might need certain bias, we might need certain topic, and we should probably try to explore. Otherwise by just hiding and hiding, I would just keep afloat whatever LLM is gonna give us and we are not able to dig deep enough.

Szymon Głowania
Yeah, I think in my opinion, one of the best way to prepare what we should get to the specific part of this knowledge, so maybe not necessary to share. It gives some access to our tool without any elements of the limits. And the test group should be big, probably 100 person. Usually we get some specific, try to give some tips for our tool and get a lot of questions how we can prepare the gun from 3D printer. Probably in these people we found somebody who asked maybe not the best question for our tool. And after that, to prepare this solution, it’s a joke.
But we have to think about this because the trust to the GenAI, prepare the knowledge about this tool, it’s very specific. Because when we talk in the general, usually some concept is clear, but when you go deeper and try how it will be work when we change some parameters, it can be very, very difficult because this model is very large and some part of them is non-excible. So, we have to prepare some tool with these elements about some wall and some limits for the model. We have to think about it and prepare some agent, for example, or some part with build, which will be used to filter some content.
And when we are preparing the elements for some of our agents, we have to think about some trust to this GenAI. Usually, in some specific implementation, we have to think about the explainable, about some elements, because it’s necessary, for example, for the bank or another part, because it’s not easy to do. I talked about the explainable for the artificial intelligence. It’s right now one of the top topics because we would like to tell a little bit about how we can prepare this solution to get a result in a specific context. Because when we go to the shop, it will be different when we go to the bank.
Because the level of the security and the knowledge of the steps, elements, attributes necessary to get after we use some tool in the productive environment.

Pierluigi Costanzo
I think that while there is a lot of, especially in Europe, actually, many companies focus on explainable AI. And I think we are moving very forward even with deep learning and explaining deep learning, although deep learning is still considered as a black box. We went very much forward and things are pretty good even to explain some of the deep learning architecture. Of course, machine learning is still very explainable, some model more than others. When it comes to LLM, I remember a couple of years ago, and this has happened especially with the closed source model, where we don’t have the control of the randomness and something.
I remember that when we built one of the first conversational AI, then we landed to a user and the user were working on it. So they started to prompt and then they texted the API. I started to use this app and but it’s actually it’s not working. So then basically take the same prompt. I went in the same application and prompted and it gave me the right answer. So then I come back like, well, actually I try here. I tried the same problem, maybe there is something is not working.

Szymon Głowania
And now it’s working.

Pierluigi Costanzo
Right? It works now in production. And then after a couple of times, this colleague tried again and again did not work. And it was really difficult to have this traceability because every time you would prompt, even when you prompt the same things, especially with the closed model in the past. Now things are much better. The answer was different. So most of the time was good or most of the time was bad. Few times were good or few times were bad. Now it’s a bit better because there is a concept of tracing, right? The result of your LLM.
And there is also a way to control the certain randomness. So if you start from the same random seed, you will get the same results. Although I remember now maybe not only for the banks, but I remember that was part of a conference, but it was more like a small round table. A person from another company, she just mentioned like, okay, let’s imagine that we are writing a report and we get some information. We wrote this report together with maybe with the co-pilot or ChatGPT or something. How do we reference that? If I go and ask the same question and they got the different result.
Referencing is really important when you write a report. So in that case, it’s probably not the right place, at least not probably the moment. In the future, probably yes, thanks to this traceability and other things, although it will require also lots of privacy and copywriting because the model can still…

Szymon Głowania
But it can be a very big problem in the company because when you have some model, GPT or some copilot, 2 or more people in the same team prepare the same dashboard, report, etc. And all these people get different answers. Which one will be the better? Which choose?

Pierluigi Costanzo
I have Copilot on the Outlook and yeah probably lazily sometimes I got like some emails and if I don’t see it for a couple of hours, there is like a plenty of conversation in the same email chain. So what they basically do, I just click summarize with copilot and I go to the meet that sometimes then I always feel like, okay, maybe let’s just go back into this email chain, email that I might miss. Maybe although Copilot might summarize very good or something, maybe there was some points that it missed. And maybe I think it’s important that points, right?
Obviously I wouldn’t read my mind and know what is important for me or not in that case, in that context. So yes, you probably sometimes need to come back anyway on the source of the data or source of the email in my case and double check it anyway.

Szymon Głowania
When I think about preparing some conversation with some agent, usually I think like a programmer. Some conversation is a specific object of this class. And all these objects can give different answers. Sometimes it can be communicated, but always it’s one specific element. So, when 2 different people ask about the same, can get different, maybe a little, but a different answer, It’s the same situation when I ask about something more than ones. So sometimes it can be problematic, sometimes we have 2 different answers, so we can paste this text to 2 different messages. So it can be useful, but we have to remember about it.
One short thing about something that you tell about the use of some models in the work. Right now we have a lot of agents, a lot of plugins, a lot of elements which can be used in our, for example, IDE. And we can program with this AI tool. So we can debug this code, get some tips, what can be improved. This solution right now is implemented in all available elements by the plugin for the web browser, for some programming elements, so we can communicate in different ways. Sometimes it says some code, debugging, plugging, web browser page, so we can get some text, some movie, some picture.
So the specific application of these elements, AI and in the specific way, GenAI, it’s all. When we would like to go to prepare some solution with GenAI and talk a little bit about how it looks like in our daily work.

Pierluigi Costanzo
Maybe it might sound banal, but it’s always like starting small. And obviously starting also with what is the use case that we need to build. Yes, usually what they’ve been done in the last year is mostly like building those solutions. And I think that that’s really important sometimes to understand that usually while many technical people, they’re building a solution, lots of times they are not the one that they’re actually using this solution. So when we started to build this generic application, the first thing to always keep in mind is like how we do workshop with our client, with our potential users that need to use this application and then kind of like a feedback loop and improve this application.
You can employ, of course, starting with the basic foundation model. You slowly, you can add your agents one at a time and some agents will also have access to your Rug system. So you start to work on the different vector database. I’m not being fun to have everything in one vector database, but I prefer the agent understand which vector store to use and employ different one. And then always come back to the user, right? We discussed it, how it’s important to having like not prompt engineering, but prompt coaching, right? In the future. And so even if we’ll be building with the team and, or I make an example before on someone that just said, yeah, I wrote the same prompt and it didn’t give me the right answer.
Then I put it and it gave me the right answer. And it’s like, how do you explain that? How do you explain that? So I think that having a workshop and explain a few basic stuff and show how you can use it for the use cases and then basis on the use cases, you adjust your prompt, you adjust your technique, you adjust your hidden prompt, you adjust your agents. You’ll probably have to adjust also a lot the rug. That definitely helps on building a really good generic application that is not only technically, maybe not perfect, but almost perfect and really good technically, but it’s actually also usable.
It’s easy to use, it gives value to your end user or to your client.

Szymon Głowania
So maybe right now we go to the more specific projects. So can you tell us about your best project, which you are most proud of?

Pierluigi Costanzo
So I think one of the projects that started last year is a project specifically for innovation on our clients and rebuilding with the team. And of course, it’s a huge thank you to the IT team that we have in… Can I say that we have in Warsaw? They’re my team, okay. So with the team that they’re part of the Warsaw Hub. So congrats to you guys. It’s a project for innovation in which our client as a user, as a brand manager can actually create a real new innovative product that can follow the brand guidelines. And so it’s fit for purpose.
And then it can be put into test. So this was like a long journey. It started 4 years ago. There was GPT-2. So it was like a painful process to work with. Definitely 

Szymon Głowania
Very old model.

Pierluigi Costanzo
not as powerful as today. There’s a different parameter to adjust and the results were quite very scarce, but with some hard work and others. And the example that I shared before about the funny stuff of changing the prompt and doing the workshop, they all come from this project. And then they’ve been employed also in many other projects. And this use obviously the rug. It used the agents to extract specific brand guidelines, information, for example. And then even how to, it use agents also to understand what are the latest trends from different data sources. And of course, the user step by step, right?
In few steps, then is able to generate. And of course, there were a couple of cases where, for example, someone was asking, okay, what if I just add a huge big prompt at the beginning with all my information and the result of course was better when you do it step by step.

Szymon Głowania
From your opinion, what is the big challenge for the use of GenAI in projects like that, so when we have some part of the company or some market.

Pierluigi Costanzo
So on the technical side, I will say on the technical side and on the end user and business side. I think for the technical side, though they are getting better and better, it’s processing different type of data that come from different sources, like in a sense that come from different type. So imagine like tons of PowerPoints, old PowerPoints, Word file, Excel files, which are probably the very difficult Excel file and PDF are much more easy. Text file, markdown, they’re usually fairly more easy. So when it comes to this other type, like especially the Excel and especially when we have like numerical data, purely dumped them into a vector store and retrieve them.
That is not ideal. That is definitely not ideal. From the user and business perspective, it’s to really just really work and having like proper coaching and proper workshop to make sure that the user are actually using those 2 correctly. Even that they don’t lose faith and trust in what we are building. That’s always difficult. We need to make sure that at the end of the day, once you spend one hour or maybe 2 hours of workshop together, they can go out of this workshop. They will go home and they will go on their desk in the office, they will open up the laptop by themselves, then they will start to prompt.
So you need to make sure that then they will do it properly. Otherwise, they try a couple of prompts, they don’t get what they want, they close it and whatever you build, maybe get the best feedback even if you did a great, great, great job even if it’s actually working. So the point is that the challenge is not only to focus on the technical things, which in some cases there is also their own challenge by itself, but working together, technical side and business side of these use cases. I think another challenge for the, maybe for more like senior consultant at the Capgemini that they also get those kinds of question is about costing it out.
You know, when we do a POC, usually in scalability. So when you do a POC, everything works fine. It’s just few user that is gonna use it. Everything was fine. It does not cost much to run it. You had to run costs once you build it. But then, especially with some models that you basically pay per token, for example, you can try to predict how many users, how much they are going to prompt and for how long they’re going to prompt per day. But you’re never 100% sure. So you can always like ballpark a potential monthly cost, but it can easily go out of hand.

Szymon Głowania
Yeah, the solution provides when we use, we pay. So if all people from our company use all time, the billing will be high. So it’s something natural what we have to remember. But the most tool provides the limits. Because when you apply this solution, add some cards, you can add sometimes one, sometimes more than one limit. And usually, we have to be enough with the billing. So it’s something that can be very difficult when you start your first project. When you have the next one, you have to some knowledge how we can predict it. But the first application, usually from my opinion, it can be difficult.
But sometimes I think it’s a good way to start something, some small project with implementation and check how it works for us. So when we go to the end of our conversation, we have to think a little bit about the future of GenAI. What will be the game changer in GenAI from your part?

Pierluigi Costanzo
Okay. So if you remember at the beginning of the conversation, I say something like the GenAI knows what to do, but don’t know when to do it. And that is the reason why we prompt. So without us triggering, naturally with the prompt, with the question, any GenAI application is not sentient, it’s not taking initiative, let’s say like this. So it’s not proactive at all. So I think that the future is something which we can call a proactive AI, or probably in the case of AI, it would be more like hyper proactive AI in the sense that we will have all these agents and systems that they will be constantly analyzing data, gathering information, running queries from the database autonomously.
And they will be sending us, they will be actually prompting us, I say like this, by just telling us, hi, Szymonor hi, Pierre, based on this data, this is and these other things are happening. And then they will be asking us, which is basically prompting us again, what do I do now with those information? So it will be up to us to take a decision, obviously. I don’t believe that AI is taking over. Rather generative AI is more a way that is going to be embedded into our work, so it’s going to be reshaping it.
And in the case of the future of this proactive AI, they’re still gonna be tools that they will take decision. They will be mostly doing lots of mundane work and analyzing autonomously, but they will always come to us. And we don’t have to prompt all the time. We may even get bored at some point to prompt all the time. They will be prompting us and say, OK, what do I do now? What action I need to take next? What decision do you have for me now that I just told you what’s happening? These are the information from the data and from the database.
So the question also to you then would be, like, are you ready to be prompted by an AI?

Szymon Głowania
So the next day you get the message, hi Pierre, I do your task from your wall. You can take a break and drink some coffee. It will be the game-chamber for us. In my opinion, it’s two parts of the future. The first one will be something really connect with this, what you talk about it. So when we try to do more automated action, do like in the process, where we have some chart of the process. So the AI tool can do this all tool like right now we ask Google about some navigation to the point or something like that.
So we have a small part of automation prepared manually in this way. But I think about the more advanced AI model to automate this process. So in these cases, it’s something what you tell. You get the answer without the question. So it’s something automated. But in all process, when you click some reports, you would like to get some email, etc, and you can do it probably speaking. It’s something the next. And the second point is connected with the learning. It’s something what I told earlier. It’s connected with the learning, some new elements with the simulation.
When some agent speaks with another one and gives something new, some new type of the best practice in the programming, etc. It’s something that we can improve because we have a lot of knowledge and we can add this knowledge to the agent and try to do something new from it. It’s, from my perspective, two great things to do. Okay, so today we speak a little bit about the GenAI prompting and preparing the best solution with some approach. So we have some future and probably it will be something new. So go to the next few steps, how we can go with the project with AI from the start to the end and get something for us.

Pierluigi Costanzo
Building an engineer solution, it’s challenging under many aspects and we covered some today and putting together business users and technical user, well, it becomes more like a human challenge, but It actually helps a lot. So we can build something that is very good technically and data scientists, data engineering together, software developer, and more AI prediction. There will be experimenting more with agents, adding more agents and getting a feedback loop, agent to agents, using a rag and using different models, small model into the agents and all of these things. From the business perspective, bias can be a thing.
Anthropomorphize with the, say hi, say hello, say please. Thank you, when you interact with the model to get information. Give clarity, right? Don’t just write, nobody likes to read 3 pages long emails. So step by step when you ask something, right? That really helps. And that’s been employed successfully on things like Chain of Thoughts or other things. And of course, the importance of having maybe not prompt engineering, more from the technical side for the intent prompt and something like coach prompting that can help in your organization, Everyone on how to use not not not only the level of how to problem how to use.
Properly and efficiently and effectively generate the solution for the different use case. For the future I think that when when we will be ready, everyone in actually prompting, probably thanks to proactive or hyper-proactive AI, we will also then be ready to be prompt by GenAI application.

Szymon Głowania
Nice. Thank you so much, Pierre, for your time, opinion and knowledge. So we have to know how we can start with the GenAI.

Pierluigi Costanzo
Thank you, Szymon.

Szymon Głowania
And probably we prepare some agent.

Aby nie przegapić kolejnych odcinków, zasubskrybuj podcast TechChatter w swojej ulubionej aplikacji. A jeśli spodobał ci się ten odcinek, daj nam znać, wystawiając ocenę na Spotify lub Apple Podcasts. Wszystkie linki do zagadnień poruszonych w odcinku znajdziesz w jego opisie.