Sign in Subscribe

A middle manager's AI odyssey

A middle manager's AI odyssey

Foreword

The following is a short fiction setting the scene for this project and how it could fit in a corporate/enterprise context. If you are in such an environment, I'd love to hear from you and better understand your needs and concerns in implementing your company AI strategy.

Monday 10AM

"By end-of-year we'll be AI-first and a whole new company." Good news is my camera was off and no one saw me spitting coffee all over my pants.

It's been a while I don't turn on my camera at our monthly Engineering All Hands. They are usually quite boring, more of the same news from various teams, not much excitement these days. I was browsing at the same time, looking for some nice beaches for our upcoming summer vacations in a week. Yet today was different, I should have paid more attention when I saw that John Cleever, our CTO, was going to be the one presenting 🤦‍♂️

Doesn't take long for my slack to start buzzing was various DMs:

  • What the heck was that? How do they think we could get AI first while we can't even get any workload in the cloud 😂 - This is Daniel, our lead IT ops, and the most cynical of the bunches.
  • Were you aware of this? How come such a decision can be taken without consulting LEGAL! If we start using AI, are we not feeding all our data and internal communication to foreign companies and governments? 😱 - Suzane, she works in legal, I know her quite well given she is an old friend of Jenny, my wife.
  • Please Steve tell me you were not aware and didn't sign up for this. We are swamped with the current SSO migration and product is killing us with the Christmas release roadmap. No way we have any kind of resource to pour in a new gigantic-doomed-from-start project 😤 - Meet Peter, team lead of the Platform team. Can't say he is wrong.

Paul knows me well enough and was ready with THE ANSWER: "I won't argue with you, we both know why we're in this position, but reality is THEY have decided, it WILL happen, so either you pull a rabbit out of your hat like you sometimes do, or we're both toasted." Then he adds the nail: "John would like a plan presented to him on Wednesday. We have two days. If we can't come up with something internally, he wants to hire the AI startup of a buddy from out of town."

Time to pick up the phone and tell Jenny that the next few days will be difficult. She reminds me that I need to pick up Chris at 5:30 to drive him to practice and that I'm the one on meal duty tonight as she won't cancel Yoga, no matter how urgent my 'AI plan' is. So, not much support on the home front. Sometimes I hate remote work; if I was in an office, I could pretend they're keeping me hostage.

Monday 10PM

House is finally quiet and asleep. I grab the laptop. Open Notion. Start writing some notes on what would be a scope for our magical AI project. Let's jot a quick MoSCoW before hitting bed, otherwise I know I won't sleep.

  1. Must be self-hosted - no way we'd clear legal/compliance with anything SaaS based on time.
  2. Must integrate with our new SSO, we can't introduce a new tool in-house if it comes with a custom user engine. Not after so much work in streamlining the rest.
  3. Must leverage our database of wine notes. We have close to 5000 wines in catalog. Gazillions of tasting notes from our scouting team and partners. Nothing structured in there. Still probably a gold-mine that can be leveraged to enrich the user experience.
  4. Should integrate with our internal API, in particular with our Enterprise Resource Planning (ERP) system. What's the point of an AI if it can't answer basic questions on stock, availability, and pricing?
  5. Should be open source, so we can extend it with our own requirements, use this as a platform for innovation and not a one-off project that gets quickly forgotten.

I'm happy with this top five. At least it's the start of a scope that could make sense. Jenny is back from Yoga. I try to explain that this might be a fun project after all. She shrugs. She knows I was down five hours ago, excited tonight, and will be depressed again before lunch tomorrow. It's midnight; I can get six hours of sleep and hopefully not destroy my sleep quality series on my tracker.

Tuesday 4AM

I wake up with a flash. There's no way we deliver this by end of year if we need to recruit new skills. Given the amount of integration, we'll need a Java stack. Ideally Spring Boot. Easy to package in microservices and deploy on our Kubernetes. And now I'm worried. Everything I've seen about AI lately was a mix of Python and using React/JS in the frontend. Peter will quit if I don't have a good story for him.

Tuesday 8AM

I woke up way too early. Yet still full of energy and enthusiasm. My agenda is cleared, everyone understands this is an emergency, and that this might be the project on which I say goodbye to the company. Four hours to start some research and find the beginning of a solution.

I don't have much time, so, let's ask an AI on what it thinks about deploying an internal AI 😅 : "I'd like to setup an internal chatgpt in my company, working entirely offline, and that can be extended with features to hook into our internal data and APIs. Ideally in Java so our existing teams can easily integrate. Any suggestions ?" - Yeah, I know, for some reason I keep my style very formal when writing to an AI, I keep anthropomorphizing it 🤷‍♂️

The output is long and super interesting. I'm really thrilled by the first elements as it shows it won't be as difficult as I had anticipated:

  1. There are many "open source" Large Language Models that we can run off-line and on-prem. These models can even be fine-tuned with our own data and there are tools and tutorials to achieve this!
  2. A fully offline solution will require massive compute. To have GPT-level quality, we need to run large models that have 70B parameters. This means we'll need a server with multiple GPUs with substantial memory. Acquiring them might be challenging and there is serious cost. But that isn't really my problem at this stage.
  3. Feeding our internal data into an AI is not crazy, a whole space of solutions is dedicated to that and called RAG for retrieval-augmented generation. It requires a step of pre-processing and storing the data in a special vector format. Good news: Postgres has an extension to perform embedding and similarity search, so our Ops/DB team should be able to deal with it.
  4. Calling our internal APIs can be done through a concept of Tools where an LLM outputs some specially formatted JSON messages to 'trigger' a request to an external tool whose answer is then fed back directly to the LLM for processing.

Finally, complex workflows and tasks can be achieved with a super exciting concept of Agentic, where an LLM is used as a reasoning engine, receiving a complex task, splitting it into multiple steps, executing the steps and reacting in case of errors to auto-correct and plan the next step! TLDR: we won't have to do complex process orchestration, this things is smart enough to do it itself 🤯

Tuesday 10:15AM

Time for coffee. The day is bright. I feel like a hero with a plan. I get out to enjoy my ten minutes of morning sunlight with a of cup Arabica.

"Morning Steve! How's it going?" - Peter pinging me on Slack. I could ignore. I'm marked as busy. I can't resist and suggest a quick Zoom so he can act as a sounding board for my plans.

"Yup, this is excellent and super good news. Exciting to learn that this whole AI madness is in fact not so complex and just a bunch of legos that need to be integrated!" - seems I managed to convince him that AI is nothing more than a buzzword around a set of well-rounded tech that aren't different from what we usually play with.

"But..." he continues ... "do you really think we can assemble a bunch of random open source projects into a cohesive enterprise application, supporting our strict guidelines and processes, extending them with our requirements without breaking everything, doing that with our current team, and all of this by end year?" 😬

Can't say he is wrong. Everything I've read links back to research paper, open source projects in various frameworks and languages, and nothing yet very mature that looks like ready for us 🙃

Tuesday 11AM

Back to search. But this time I know what I'm looking for and can be much more direct. Let's google this one: "open-source extensible self-hosted enterprise AI agentic platform springboot java".

After three sponsored links trying to sell me a cloud-hosted solution, one project gets my attention. It seems to be everything I've been looking for...

OpenGPA has it all:

  • free, open-source, self-hosted solution
  • not a 'framework' but instead an integrated opinionated solution with all the key components I was looking for
  • extensible so our team can build in-house custom actions tapping into our internal APIs
  • written in Java, using spring-boot and spring-ai, giving me full flexibility in terms of SSO integrations, choice of databases, choice of LLMs etc.

No time for lunch today. I won't have a slide deck by tomorrow. I'll have a demo !

git clone git@github.com:eschnou/OpenGPA.git

Afterword

OpenGPA is not ready for Steve yet. We are just getting started. What is released today is a 0.1 version with basic core features such as multi-step reasoning and tool use, including web search, web browsing, file access and text-to-speech.

I hope that a small community sharing the vision can grow around this project, contributing to the catalog of third-party actions and integrations; improving the core engine; inventing the future of Enterprise Agentic Applications.

Interested? Then please take action:

Let's build the future of Enterprise AI together!

Thanks

  • To the AI open models community and their fantastic work in contributing and sharing high-quality LLMs with the rest of us.
  • To the AI research community that continues to invent in the open, sharing new ideas on Agentic in the open. Some of this research has been integrated into this project and more will be in the future.
  • To the AI open source developers community for the many open projects building all forms of frameworks and agents, they were an inspiration and a great place to learn.