Explaining AI Agents in Simple Terms: A Conversation with Senior AI Engineer Hanyang Li

Mother of Success sat down with Hanyang Li, a senior AI engineer with deep experience in NLP and large-scale AI systems. In this conversation with Xiao He, founder of Mother of Success, Hanyang shares his journey from traditional machine learning to the new era of AI agents, and offers practical insights for founders and engineers navigating this fast-changing field.

Xiao He:

Hanyang, thank you for joining us today. Could you start by introducing yourself to our readers?

Hanyang:

Yeah, sure. So my name is Hanyang. I’ve been in the machine learning, especially NLP field, for about seven years now. I went to Carnegie Mellon University to study machine learning for graduate school.

My first job out of school was at Bloomberg’s AI group in New York. That was before the LLM age, so my work was more like a traditional machine learning engineer. That means I owned the end-to-end machine learning model lifecycle: collecting data for a problem, training a model, deploying it, making sure it worked in the real world, and evaluating it. At Bloomberg I worked on things like translation, named entity recognition, and sentence embeddings for semantic search.

In the most recent three and a half years, I worked at a Series C startup in Silicon Valley. When I first joined, my role was traditional: training models, doing evaluations. But then ChatGPT was released, and that was a big paradigm shift in NLP. After that, we fully embraced an agentic AI architecture, and my work became about how to build that architecture and make it work from scratch.

Xiao He:

“AI agent” has become a buzzword. In your view, what is an AI agent? And if you were to describe it in practical terms, how would you do that?

Hanyang:

Yeah, I think AI agent is kind of a buzzword. A thousand people probably have ten thousand different definitions. But the core idea, I’d put it this way:

In the old world, especially for conversational AI, you needed a step-by-step NLP pipeline. When someone asked a chatbot a question, you had a language detection layer, then maybe a translation layer, then named entity extraction, intent classification, slot filling—basically a software pipeline where each model did one narrow task.

In the new AI agent world, because of the capabilities of LLMs, people rewired how the system should work. For conversational AI agents—like ChatGPT or what we built—you really put the LLM at the center. On its own, the LLM can only do text-in, text-out, based on its pre-training. But to solve many real world problems, the LLM needs to interact with some environments in a digital world. The environment can be the public internet, corporate databases, personal calendars, CRMs like Salesforce, or HR systems like Workday. To enable the LLM to interact with these environments, you need to equip it with tools. A tool is a bridge between the LLM and that environment, letting it read or write. In practice, a tool can be an API or an abstraction component that combines APIs with some business logics. And then you also need memory, because in a long or complex conversation, the agent needs help with context management to understand what it should focus on for the current task.

So simply put: an AI agent is an LLM-centered software system that can access tools, manage context, and execute tasks with some level of planning.

Xiao He:

Could you give a concrete example for readers who may not be technical?

Hanyang:

Yeah. “Talk to a file” is a very common use case. Say you have a really long PDF, maybe 100 pages. You don’t want to read through it yourself. Instead, you just ask questions and let the agent find the relevant information and synthesize it for you.

If you’re building this, you’d use something like the OpenAI API plus a framework like LangChain. One key system consideration is preprocessing. If the document is short—say one page—you can feed the whole thing to the LLM. But if it’s long, you need to chunk it into smaller pieces, maybe 200–300 tokens each. Then you index those chunks, so when a user asks a question, the system retrieves the relevant pieces and passes them to the LLM. That way it can answer accurately without being overwhelmed by noise.

Xiao He:

Hallucination is one of the big concerns with LLMs. How do you make sure the outputs are accurate and grounded?

Hanyang:

Yeah, this is really important, especially for knowledge retrieval or Q&A use cases. Making sure the LLM’s response is factual and based on verifiable sources is usually called grounding.

It’s still a challenge—no one can say they’ve solved it 100%. But there are things you can do. First, context engineering: don’t dump 50 pages of noisy text. Retrieve only the most relevant, clean context. That helps reduce hallucination.

Second, set up evaluation pipelines. For example, if you care about citation accuracy, build a golden dataset where you know the ground-truth source, and test whether the model cites correctly. You can also sample responses in production and have annotators check them.

And third, preprocessing. A lot of documents aren’t just text—they have tables or images. You need OCR or conversion tools to normalize them into text or Markdown, which LLMs handle really well.

Xiao He:

What about engineers who want to enter this field? How can a software engineer become an AI engineer today?

Hanyang:

I actually think the barrier is much lower than it used to be. When I transitioned into ML, I spent one and a half years at CMU studying statistics and math, because back then you needed to build models from scratch.

Today, machine learning roles have branched out. There are:

• Research Engineers – who still do training, usually at foundation model labs.

• ML Infra Engineers – who build the infrastructure for training and serving models.

• Applied AI Engineers – who use LLMs and build agentic systems.

If you can code and you’re a software engineer, you will be able to pick up the skillsets of an AI engineer. You just need to get familiar with the common architectures and do hands-on projects. Hackathons are great for that. And honestly, no one has more than three years of experience in agentic AI, because it didn’t exist before that. So the playing field is very new.

Xiao He:

And for companies that want to hire or train AI engineers?

Hanyang:

For hiring, I think it’s important to assess whether a candidate has a solid understanding of the core building blocks of agentic AI systems and how these systems can be applied to real business problems.

For training internally, the best way is learning by doing. This field changes so fast that textbooks are outdated. If a company has a real use case, they should empower a team to build it, iterate, and learn. Keep up with the latest thinking on Twitter, YouTube, or open-source forums—that’s where the state of the art is happening, not in a classroom.

Xiao He:

That’s fantastic advice. Thank you, Hanyang, for sharing your experience and insights with us today.