Does AI-Generated Documentation Have Value?

As many have observed, at best, AI generates either:

  • Mundate, repetitive documentation or code that most experts already know or

  • If an expert doesn’t know it, they can ask an AI to explain it anyway.

Is it the case that if an AI can generate it, it’s not worth adding to your documentation or code? While this is usually the case, there is still value in reading AI-generated documentation as a means of reviewing and validating what you might write differently.

In this post, I show how changing the documentation can affect the AI’s output and how reviewing it can be a useful exercise.

Estimating the value of AI-generated documentation

I tried to gauge the relative value of AI-generated documentation for my knowledge and set expectations for others.

I asked o1 pro to generate user guides in several different ways I considered effective for:

  • a simple class and package

  • a large class and package

  • a complex class and package

Then, I subjectively estimated how many points made were compelling, interestingly incorrect, correct but not interesting, or just plain wrong. I ignored repeated points. This is what I concluded:

Type Average points per query Percentage

Correct but not interesting

35

80%

Just plain wrong

5.4

13%

Compelling to keep

2.1

5%

Wrong But Interestingly

0.8

2%

This assumes I crafted the query to be as specific as possible. Without a specific query, the AI can generate a lot of irrelevant or wrong content.

Understandably, this can lead people to conclude that AI is unusable. However, there is still value in reviewing AI-generated documentation.

Why Bother Reviewing AI-Generated Content?

If we go into this process with the mindset that the content will be ruthlessly edited, we can extract value for relatively low effort. For example, each file I have reviewed had, on average approx 3 points that are compelling to keep or errors worth addressing.

  • Is this point worth emphasising in your official documentation?

  • Is the suggestion so far off that it needs immediate correction to avoid misleading readers?

  • Could it be a prompt for additional tests, performance checks, or clarifications?

Wrong But Interesting

I asked o1 to generate a user guide for a net.openhft.chronicle.core.StackTrace

The best way to get the right answer on the internet is not to ask a question; it’s to post the wrong answer.
— Cunningham's Law

StackTrace for the current thread

There is a method for getting a StackTrace for a given thread forThread(Thread t), but not for the current thread. However, I had assumed developers would know that new StackTrace() would give the current thread’s stack trace.

o1 suggested there was a method that it assumed existed but isn’t needed.

public static StackTrace current() {
   return forThread(Thread.currentThread());
}

Adding to the documentation, I can clarify that a helper method isn’t needed. All you need is new StackTrace().

BTW: Copilot added this last sentence after I wrote the paragraph above.

Logging the StackTrace (Interesting But Wrong)

The AI suggested logging the StackTrace by calling toString() e.g.

// BTW Copilot wrote this
LOGGER.info("Stack trace: " + new StackTrace().toString());

However, this is not the best way to log a stack trace as it will be a single line without the actual stack trace as a StackTrace is a Throwable so you can write:

// BTW Copilot wrote this too, after I wrote the paragraph above
LOGGER.info("Stack trace", new StackTrace());
Note
 Copilot will read your documentation and change its suggestions based on it.

Timestamp for a StackTrace (Interesting But Wrong)

The AI there was a timestamp to the StackTrace, which is a good idea, but it suggested using System.currentTimeMillis(), which is not the best way to get a timestamp as it is not monotonic. A better way would be to use CLOCK.currentTimeNanos() as a nanosecond resolution timestamp.

Likely addition based on this assumed method
class StackTrace extends Throwable {
    // TODO add a timestamp to the stack trace
    final long timestampNS = CLOCK.currentTimeNanos();
    public long timestampNS() {
        return timestampNS;
    }
}

Frequently Asked Questions (Suggested by o1 pro)

I thought these points were worth including somewhere in the documentation.

Q: Why not just use new Exception() or new Throwable() to capture a stack trace?

A: You certainly can. But the StackTrace class provides a simpler, more descriptive pattern and supports capturing another thread’s frames with forThread(…​). This approach avoids confusion with typical exception usage and clarifies the diagnostic intent.

Q: Can I throw StackTrace?

A: The Javadoc strongly discourages it: "not designed as an Error or an Exception and is not intended to be thrown or caught." While it could compile, that’s not the pattern or intention, and it may create confusion or rely on unexpected behaviours.

Q: Is there a performance penalty?

A: Capturing a stack trace is inherently more expensive than just running code. The overhead depends on how often you do it, how large the stack is, and the JVM. Infrequent usage for debugging is typically fine. Constant usage in tight loops is not recommended.

Q: Does it handle partial frames?

A: In certain JVM states, the returned stack trace could be incomplete or lacking line numbers. The method also skips the first frame if isNativeMethod() is true—intended to reduce noise, although behaviour may vary by JDK version.

Q: Are there concurrency concerns if I call forThread(…​) on a thread that’s actively running?

A: Thread.getStackTrace() is safe, but there is no guarantee you get a perfectly consistent snapshot if the thread is in the middle of certain operations. Typically, you get a best-effort stack trace at the moment of invocation. It can’t forcibly pause the thread, so expect snapshots that might reflect a transitional or ephemeral state.

Q: Can I use StackTrace in a production environment?

A: Yes, but be mindful of the performance implications. It’s designed for debugging and diagnostics, not as a general-purpose tool. If you’re capturing stack traces frequently, consider the impact on your application’s responsiveness and resource usage.

BTW: Copilot added this last FAQ.

Conclusion

While AI-generated documentation often appears superfluous at first glance, it can still serve as a launchpad for:

  • Identifying domain-specific insights you may have overlooked.

  • Exposing interesting misconceptions worth clarifying in your official docs.

  • Spurring performance and correctness tests based on the AI’s questionable suggestions.

It can be worth reviewing and editing with a ruthless eye. It can be a good way to get a list of things to consider or to avoid rather than include.

It’s important to remember that even if developers don’t read documentation, AI does read all available text and will change the output produced based on it.

About the Author

As the CEO of Chronicle Software, Peter Lawrey leads the development of cutting-edge, low-latency solutions trusted by 8 out of the top 11 global investment banks. With decades of experience in the financial technology sector, he specialises in delivering ultra-efficient enabling technology that empowers businesses to handle massive volumes of data with unparalleled speed and reliability. Peter’s deep technical expertise and passion for sharing knowledge have established him as a thought leader and mentor in the Java and FinTech communities. Follow Peter on BlueSky or Mastodon.

Key Takeaways

  • Don’t abandon your documentation: AI-generated text can spark valuable ideas.

  • Employ a ruthlessly selective approach: discard the routine, keep the gems, refute the nonsense.

  • Consider performance: capturing stack traces (or any reflective data) has a real cost.

  • Scepticism is your ally: AI’s "interesting but wrong" points can reveal hidden assumptions.

  • AI might shape future developers' understanding: the more precise your published docs, the better the AI will eventually become.

Comments

Post a Comment

Popular posts from this blog

Java is Very Fast, If You Don’t Create Many Objects

Image
  You still have to watch how many objects you create. This article looks at a benchmark passing events over TCP/IP at 4 billion events per minute using the net.openhft.chronicle.wire.channel package in Chronicle Wire and why we still avoid object allocations..  One of the key optimisations is creating almost no garbage. Allocation is a very cheap operation and collection of very short-lived objects is also very cheap. Does this really make a difference? What difference does one small object per event (44 bytes) make to the performance in a throughput test where GC pauses are amortised? While allocation is as efficient as possible, it doesn’t avoid the memory pressure on the L1/L2 caches of your CPUs and when many cores are busy, they are contending for memory in the shared L3 cache.  Results Benchmark on a Ryzen 5950X with Ubuntu 22.10. JVM Vendor, Version No objects Throughput, Average Latency* One object per event Throughput, Average Latency* Azul Zulu 1.8.0_322 60.6 ...
Read more

System wide unique nanosecond timestamps

A Unique Identifier can be very useful for tracing. Those ids are even more useful when they contain a high-resolution timestamp.  Not only do they record the time of an event, but if unique can help trace events as they pass through the system. Such unique timestamps however can be expensive depending on how they are implemented.   This post explores a lightweight means of producing a unique, monotonically increasing system-wide nano-second resolution timestamp available in our open-source library. Uses for Unique Identifiers Unique identifiers can be useful to associate with a piece of information so that information can be referred to later unambiguously. This could be an event, a request, an order id, or a customer id. They can naturally be used as a primary key in a database or key/value store to retrieve that information later. One of the challenges of generating these identifiers is avoiding creating duplicates while not having an increasing cost.  You could ...
Read more

Comparing Approaches to Durability in Low Latency Messaging Queues

Image
  A significant feature of Chronicle Queue Enterprise is support for TCP replication across multiple servers to ensure the high availability of application infrastructure. I generally believe that replicating data to a secondary system is faster than syncing to disk, assuming the round trip network delay wasn’t high due to quality networks and co-located redundant servers. This is the first time I have benchmarked it with a realistic example. Little’s Law and Why Latency Matters In many cases, the assumption is that the latency won't be a problem as long as throughput is high enough. However, latency is often a key factor in why the throughput isn’t high enough. Little’s law states, “ the long-term average number  L  of customers in a  stationary  system is equal to the long-term average effective arrival rate  λ  multiplied by the average time  W  that a customer spends in the system”. In computer terminology, the level of concurrency or par...
Read more