Community's new AI search capability reviewed and rated...

Read time: 6 minutes

Hey there!

Earlier this summer, Workday added generative AI to Community’s search capabilities, claiming it would supercharge our problem-solving quests 🚀

Ooo-la-la!

We liked the sound of that 😃

Back at the start of our Workday careers—bright-eyed and bushy-tailed, fresh out of training—we were told we’d need to master the art of searching Community.

And with plenty of sweat equity (and a fair share of dead-ends), we did. But lately, even for us, it’s felt… harder.

Some folks over on r/workday (Reddit) say they only find the answer they need about 30% of the time. That’s worse than an F- 🫣

So naturally, this claim perked up our ears.

Does the AI more readily deliver the answers you’re digging for?

We did a bit of testing to see for ourselves, and of course—to share with you 🤗

In today's newsletter, we share a few examples that illustrate what we found 🕵🏻‍♀️

Supercharged AI put to the test 🧪

Any worthy experiment needs measurements! So, we scored examples from our testing according to these 3 scales…

An AI that consistently returns answers with all 5’s on these scales? Now we’d consider THAT a truly supercharged experience 😌

So, is that the reality of the moment? Let’s explore…

Notes from the field 📕

Example 1

🔍 Search: “How do I set up an onboarding plan?”

Alright! This is a decent response that could certainly save you some time if you’re starting from scratch. Here’s what we like about this response…

✅ It points you to the correct location within Workday to begin your configuration—the Onboarding Planner Hub.

✅ The response gets other key components that you need to configure on your radar, like Stages, Audiences, and Content.

The guidance fizzles out as the response goes on, becoming more vague and less helpful.

Also notice, this response has 5 citations—a reflection of Community’s extensive documentation on Onboarding Plans.

Example 1 Scoring:

Accuracy: 🎯🎯🎯🎯🎯 (5/5)

Completeness: 🧩🧩🧩 (3/5)

Readability: 📘📘📘📘 (4/5)

12/15

Not too shabby 👏

An early conclusion we’re drawing? The AI can’t make something out of nothing—its answers are only as deep as what’s already available on Community. If Workday’s own documentation is decent, AI answers reflect that.

P.S. For something closer to a 15/15 response, check out our Onboarding Plans guide 😜

Example 2

🔍 Search: “What data sources include future-dated hires?”

The datasource suggestions here are technically correct 🙌 These are all fair options!

However, we know from hard-earned experience that each of these data sources come with caveats—nuances we feel should be disclaimed before you dive in. The AI’s summary of Community’s summaries (😅) is a great starting point… but will still send you on the same trial-and-error trail blaze if you don’t do your own research first.

For example, the last option, All Active and Terminated Workers, doesn't actually pull future hires. It returns active workers as of the effective-date you run the report with. You can get it to include future hires by setting the effective date to the future! But there’s a catch...

If you don’t include terminated workers in the results, your future terms will fall out of the results since they will no longer be active as of the future effective date.

Example 2 Scoring

Accuracy: 🎯🎯🎯🎯 (4/5)

Completeness: 🧩🧩 (2/5)

Readability: 📘📘📘📘 (4/5)

10/15

The response looks solid on the surface, but once you dive in, the details are patchy! You have to figure out the gaps in your own trial-and-error.

 Example 3

🔍 Search: “How do I configure an accrue-as-you-go time off plan?”

“1. Create the Plan: Use the xxx - Time Off Total to create the plan.”

Oof 🥴 This response is off to a rough start: Vague at best, perplexing at worst.

This first step attempts to call out the most defining characteristic of an accrue-as-you-go time off plan: the accrual calculation.

The problem is, simply being told, “use the xxx - Time Off Total to create the plan” isn’t helpful. The answer doesn’t tell you…

❌ Where the xxx - Time Off Total calc goes (on the accrual, in the calculation input)
❌ What kind of calculation this is (an Absence Component Related Calculation), or
❌ What “xxx” represents (the name of the time off component on the time off plan).

If you’re not familiar with building time off plans in general, this answer leaves you totally high and dry—no task names, no component names, no configuration instructions.

Following these directions would send an unexperienced reader in circles 😵‍💫

Example 3b

Next up…

We explored, does the AI give you the same response for a similar but differently phrased search?

🔍 Search B: “Configuring an accrue-as-you-go time off plan”

In this example, yes and no. The answers are technically different, but not meaningfully so. Given that steps 3 - 7 in the first response are mostly fluff, and the citations listed are the same, these responses elicit the same result.

What is helpful within both responses…

The citations! If you open the first citation, “Concept: Accrue-As-You-Go Plans”, you’ll find that Workday’s documentation does do a pretty good job explaining how you’d configure this…

Albeit, in thick, boring text, but that’s just Workday documentation for ya 🤷🏻‍♀️

Example 3c

And our last exploration on this particular topic…

Imagine you’ve never heard the term “accrue-as-you-go”, and you simply search “What’s the best way to configure an unlimited time off plan?” instead.

Welp! 😬 The AI spits out an answer to the wrong question entirely. However, again, the citations are redeeming…

Click on the “Best Practice to Create Time Off Plan with Unlimited Balance” citation, and you’re taken to a user-generated discussion from 2018 with 1 Accepted Solution (selected and approved by Workday) that throws you a bone with the term “Accrue As You Go”:

If you were starting from scratch, this would be a big step forward in your research! 🥳

But… this citation is also the second result in the non-AI search results…

So was there actually any efficiency gained? Well, in the time it took you to scan through an answer completely unrelated to your question… probably not 🙃

Example 3 Scoring

Accuracy: 🎯🎯🎯 (3/5)

Completeness: 🧩 (1/5)

Readability: 📘📘 (2/5)

6/15

Our takeaways? How you phrase your search doesn’t have too much impact. And, questions specific to niche modules tend to lack depth.

 Example 4

🔍 Search: “What is a boomerang integration?”

Hooray! That’s accurate! 👏

The AI generates sufficient answers for searches that ask for basic definitions.

What it still lacks, though, is “human-speak”.

Community—like most documentation out there—tends to read like a car manual: technical, formal, and full of jargon.

Naturally, its AI adopts that same tone.

Compared to other AI tools out there that lean into a more relational voice, this can be a turn-off.

For contrast, here’s the same prompt submitted to ChatGPT (with “in Workday” appended at the end for context 😉)…

Not bad!

Chat has a fighting chance of being the better choice for high-level definition prompts due to it’s relational, beginner-friendly voice and the option for further dialogue.

Example 4 Scoring

Accuracy: 🎯🎯🎯🎯🎯 (5/5)

Completeness: 🧩🧩🧩🧩🧩 (5/5)

Readability: 📘📘 (2/5)

12/15

For simple searches, the AI nails the definition—just don’t expect it to speak human 😅 

The verdict 

When AI made its debut to Community at the end of April, we were not impressed.

The responses were incomplete, full of jargon, or downright inaccurate (can you tell where our scales came from? 😆).

In the 2 ½-ish months since? We’ve noticed major improvements. Our examples are from just this week—we had to update them when we realized just how much the AI had matured (they grow up so fast 🥲)!

But even with its notable improvements, does it supercharge search, as advertised?

The truth is, most people search Community looking for either:

1. Specific documentation to get an introduction to the nuts and bolts of configuration they haven’t experienced before, or

2. Collaboration and insights posted by other users when they hit a snag and have exhausted their own trial-and-error, e.g. Has anyone seen this before? Is there a workaround? What do other orgs do in this case?

The AI can’t comprehensively fulfill these needs.

However, when put to the task, its summarizations and citations certainly support your investigative efforts, providing a solid starting point—and often a shortcut 🏃‍♀️‍➡️

Now for the real question…

Will your search beget higher success rates with AI than the 30% reported on Reddit?

For the experienced super searcher, probably not.

AI will only ever be as good as the sources it draws from. If Community itself lacks an accurate, complete, and readable answer, so will the AI.

So, while we don’t think the AI is quite at the level of supercharge, we definitely consider it to be a spark ⚡️😊

At this time, we have a word of warning, a recommendation, and a call to action:

⚠️ Fact check every AI response. Start by reviewing its citations at the bottom and call on your support network to help you vet the answers. Don’t follow its directions blindly.

💡 Use citations as your research shortcut. The AI does well surfacing relevant content that could serve as a click-saving shortcut.

🗳️ Throw in your vote by rating its answers. Anytime you catch something incomplete or inaccurate, click the thumbs down. Training the AI is how this thing gets better.

Happy searching, friends! 🥰

As always, thank you for being a reader!

We’re celebrating you and your pursuit of a Well Built Workday 🥳 

Until next time!

Ceci & Mia

Co-Founders of Well Built Solutions

P.S. Loving the newsletter? Leave us a testimonial here 🥰