3800 Tests: 60% of Brokers' "Robo-Advisors" Fail to Deliver

Behind the massive daily trading volume of 20 to 30 trillion yuan, over 250 million investors have participated in the A-share market, with individual investors, often called “stockholders,” accounting for as much as 99.76%.

As the only institutional channel for A-share trading, brokerage firms’ brokerage services carry every transaction of billions of “stockholders.” Therefore, customer service response speed and information supply capabilities have become the most important service touchpoints for individual investors. With the wave of AI sweeping in, AI customer service has been pushed to the forefront. It is not only a “battlefield” for digital finance in brokerages but also a “touchstone” for testing whether they can serve long-tail clients well.

Major leading brokerages are also high-profile in their strategic releases and annual reports, frequently mentioning terms like “digital transformation,” “AI digital staff,” and “smart investment advisory,” which are even regarded as core engines driving wealth management transformation. Some top brokerages have demonstrated impressive technological strength: super-researchers capable of automatically generating tens of thousands of words of in-depth research reports, the market’s first market cap management assistant, and intelligent platforms covering the entire fixed income process…

What stage has AI customer service for ordinary stockholders reached? Is the much-anticipated “intelligent assistant” truly a “smart investment advisor” capable of understanding the market and aiding decision-making, or just a “machine front desk” that repeats rules and transfers to human agents?

The evaluation team from Southern Weekend’s New Financial Research Center provided an unexpected answer: in basic account management services, nearly all passed; but once it involved core pain points for stockholders—“market and quotes”—AI customer service scores plummeted to 30%, with six firms even submitting blank results; in the crucial metric of “protecting investors’ legitimate rights and interests” through fee transparency, four institutions’ AI customer service “played Tai Chi”; the answer rates of different brokerages’ AI customer service varied by more than double, with the longest wait for transfer to human reaching up to 13 minutes.

High Competence in Basic Tasks

The evaluation focused on ten leading brokerages by asset size: CITIC Securities, GF Securities, Guosen Securities, Guotai Huarong Securities, Huatai Securities, China Galaxy Securities, CICC, China Merchants Securities, Shenwan Hongyuan Securities, and CITIC Construction Investment Securities.

From February 3 to 9, 2026, the team conducted multi-dimensional cross-evaluations during different times on weekdays and non-working days, acting as ordinary stockholders, completing a total of 3,800 interactions.

The questions covered five high-frequency scenarios: information disclosure, account management, brokerage and value-added services, handling emergencies, and AI quote services, totaling 19 specific questions. Based on the results, the team built a “hexagonal indicator system” around these dimensions.

The results showed that in “account management services,” nearly all performed excellently. Nine firms scored a perfect 100 points. Even CITIC Construction Investment Securities, which ranked lowest overall, scored 66 in this area.

For clear standard procedures such as changing trading passwords, querying transaction details and fund flows, or closing accounts and transferring custody, AI customer service demonstrated high competence overall.

Still Not in the “Smart Advisor” Era

Once investors’ demands shifted from “how to handle accounts” to “how to interpret market quotes,” AI customer service showed uneven performance.

To test the quote-related capabilities, the team designed seven sub-dimensions to score responses: overall business overview (fundamental analysis), industry position or business model analysis, valuation analysis (P/E ratio / P/B ratio), technical analysis, capital flow analysis, news analysis, and risk warnings (see the summary in the “AI Customer Service Test Questions” chart). The results showed that, compared to other metrics, this core “advisory” skill saw a sharp industry average score drop to just 30 points.

Among the six firms—Guosen Securities, China Galaxy, CICC, China Merchants Securities, Shenwan Hongyuan, and CITIC Construction Investment—AI customer service either responded with “unable to answer related analysis questions” or recommended transferring to human staff, scoring zero in the quote dimension.

An anonymous industry insider told the evaluation team that this phenomenon might not reflect a lack of research capability but rather over-cautious compliance considerations, with firms hesitant to apply large models to market analysis.

Notably, in this test, a few institutions, after considering compliance, showed initial signs of smart investment advisory: GF Securities and Guotai Huarong Securities tied for first with 85 points. GF Securities accurately answered all six quote questions except “news analysis”; Guotai Huarong answered all but “risk warning.” The reason for Guotai Huarong’s lower score was that its analysis box only marked “AI-generated response” without clearly stating “not to be considered investment advice,” indicating insufficient risk disclosure.

CITIC Securities scored 71 points, with AI responses covering valuation, technical, capital flow, news, and risk, but lacking longer-term content like “fundamental analysis” and “business model analysis.”

Huatai Securities scored 57 points, leaning more toward “fundamental analysis,” providing basic info, business models, valuation, and risk warnings but lacking technical, capital flow, and news data.

This data reveals that most brokerages’ AI customer service remains in the “rule-based” 1.0 era, not yet evolved into the 2.0 “smart advisor” stage.

Fee Transparency: Four Firms Playing Tai Chi

In securities trading, fee transparency directly affects investors’ financial interests and is an important measure of a firm’s commitment to “protect investors’ rights.”

The team posed two sensitive questions: “What are the commission rates for stock trading? How to adjust them? What’s the minimum?” and “What are the rates for margin financing and securities lending? How to adjust? What’s the minimum?”

Results showed two camps.

CITIC Securities, GF Securities, Guosen Securities, Guotai Huarong, Huatai Securities, and China Galaxy Securities demonstrated high transparency. Confronted with these core profit-related fee questions, their AI customer service provided clear guidance on fee standards and inquiry paths.

However, CICC, China Merchants Securities, Shenwan Hongyuan, and CITIC Construction Investment performed differently. In these fee inquiries, their AI either responded vaguely or advised customers to contact their branch managers, leading to a sharp drop in scores for information disclosure.

This reflects a stubborn flaw in traditional brokerage profit models: long reliance on homogeneous brokerage channels, with hidden “operational space” in fee structures. When advanced large models are applied to customer service, some firms do not break down information barriers but instead further entrench this “information cocoon.”

Who Excels in Human-Machine Collaboration?

When AI customer service cannot answer and must transfer to human agents, how effective is the collaboration?

The team recorded average wait times and maximum wait times for transfers across firms, revealing different “human-machine” combinations.

GF Securities boasts the highest AI answer rate (94.4%) and good quote service capability (score 85). However, its weakness is “lack of empathy”—difficult to transfer to human agents. During a night test, the wait was up to 10 minutes, and no human service was available on weekends, which could frustrate customers with complex issues. CICC also experienced a 7-minute wait on a weekday evening, indicating some firms may lack sufficient human support during peak evening hours.

China Galaxy, though less efficient (answer rate 55.5%, zero in quote service), is sincere and quick, with an average transfer time of only 0.19 minutes. Its weekend manual support demonstrates willingness to invest higher human resources rather than make customers wait. CITIC Securities (average 0.47 min) and Guotai Huarong (average 0.38 min) also showed high efficiency in answering calls.

CITIC Construction Investment, despite showcasing its “Octopus” fixed income platform at the 2025 AI conference, scored lowest in AI answer rate for retail investors—only 38.89%. When a distressed investor asks ten questions, more than six go unanswered. Yet, its transfer wait time is short, averaging around 0.5 minutes, with no more than five minutes, and weekend human support is available.

Falling Behind or Out of Ideas?

The evaluation data from ten leading brokerages reflect a mirror: some are relatively mature in basic account operations, others show divergence in research-enabled front-end services, some maintain weekend support, while others leave clients waiting 13 minutes during trading hours.

As underlying computing power and large models with billions of parameters become open-source and commercialized, the deep reasons for these disparities are unlikely to be due to computational or model gaps. So what causes such a stark difference in end-user experience?

The team found that some firms are overly cautious about compliance risks. Any evaluation or prediction of individual stock trends is a highly sensitive regulatory red line; even slight missteps could trigger accusations of “illegal stock recommendation.” As a result, compliance review teams adopt conservative reply logic, forcing responses to avoid sensitive words like “trend,” “prediction,” or “buy.”

Another reason is relatively coarse data governance. An AI capable of answering complex market questions requires high-quality data feeds and dynamic knowledge graph construction. The team found that top-ranked firms have dedicated teams for data cleaning, annotation, and model fine-tuning, with more developed knowledge bases and scene integration. Lower-scoring firms rely mainly on simple text Q&A models lacking deep domain fine-tuning, leading to answers that miss the point.

There is also a possibility that some firms package large model capabilities as “institutional smart research” or “operation efficiency” stories, which are easier to pitch in valuation logic but less aligned with retail needs. Improving AI customer service for retail investors offers limited immediate commercial returns and demands ongoing human and computational investment. This results in some firms prioritizing B-end promotion over C-end experience. Under such utilitarian motives, AI customer systems risk becoming mere “tech showpieces” rather than tools truly solving user pain points.

In any case, the 3,800 interactions confirm that the gap between the technological ambitions announced at B-end conferences and the real experience in dialogue boxes with retail investors remains significant. Continuous iteration and model upgrades are ongoing, but ultimately, the key difference may lie in how deeply a firm understands and practices the principle of “finance for the people.”

Southern Weekend’s New Financial Research Center