Go Content Extraction: A Density Parser That Distrusts article Tags

A 345-character widget was extracted instead of the body of a Dong-A Ilbo article.

The cause was trusting the <article> tag. The page's rendered DOM contained 23 <article> elements. They had been laid out at the widget level, while the actual 9,400-character body was outside them.

The assumption that a semantic tag pointed to the body was wrong.

I built a web body extractor without using an external API.

There was one endpoint and three caps

The contract was simple. It accepted a URL and returned markdown.

// platform/backend/internal/api/reader.go:13-39
func registerReader(g *echo.Group) {
    g.POST("…", func(c echo.Context) error {
        // body: {"url": "..."}
        // 200: {url, title, site, kind, markdown, hero_image_url, truncated}
    })
}

The presence of kind might stand out. It distinguished whether the page was an article, a feed, or a social post. Social posts had no title and had short bodies, so they needed different handling.

Code that called someone else's server needed caps. I set three.

// platform/backend/internal/reader/reader.go:30-31
const maxBody = 4 << 20      // 4MiB까지만 읽습니다
const minBodyRunes = 80      // 80자보다 짧으면 본문으로 인정하지 않습니다

// reader.go:147-152
client.Timeout = 10 * time.Second   // 10초 안에 못 받으면 포기합니다
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
    if len(via) >= 3 {              // 리다이렉트는 세 번까지
        return statusError(http.StatusGatewayTimeout, "UPSTREAM_TIMEOUT", "too many redirects", nil)
    }
    if err := e.validateURL(req.Context(), req.URL); err != nil {

The last line was worth noting. The address was checked again every time a redirect was followed.

Checking only the initial address did not prevent SSRF. An attacker could provide a normal address and have that server redirect to http://10.0.0.1. This would make the server fetch its own internal network on the attacker's behalf.

I restricted ports to 80 and 443, resolved DNS, and rejected the result if the IP belonged to a reserved range. I did not decide based only on the domain name.

I decided not to keep a table of site-specific rules

Korean news portals used unusual body containers. Naver used #dic_area or #newsct_article.

So I built a table of site-specific selectors. It looked up the host name and extracted the body from the corresponding container.

I removed it. The reason was documented in a comment.

// reader.go:626-635
// There is deliberately no per-site container table here. One was written for
// Naver (#dic_area / #newsct_article) on the assumption that Korean news
// portals need special-casing, but running five byte-for-byte captures of live
// articles through the extractor with and without it produced identical output
// — same title, same body, same length — because the density fallback already
// finds the same node. A lookup table that changes nothing is a maintenance
// cost, so the generic path is the only path.

I captured five live articles byte for byte and ran them through both paths. The output with and without the table was completely identical. The title, body, and length were all the same.

The generic path was already finding the same node.

The output was the same for all five live articles even with the table

Until I confirmed that the results were the same without the table, I thought the table was doing work.

Code like this was difficult to remove. It felt as though removing it would break something. It also seemed that the table would need to be updated whenever a site changed its markup.

In reality, it only incurred maintenance cost. If a site changed, the table had to be updated; without the table, there was nothing to update.

I left the condition for adding a rule in a comment.

Add a rule only when a real capture proves the fallback misses.

I added a rule only when a real capture proved that the generic path failed. I did not add one based on the expectation that there would probably be a special case.

I let candidates compete to select the body

If I could not trust semantic tags, what should I use instead?

The first implementation selected from within <article> whenever at least one existed. It failed on Dong-A Ilbo.

// reader.go:663-670 (주석)
// 후보를 경쟁시킨다. 이전에는 <article>이 하나라도 있으면 그 안에서만
// 골랐다. 동아일보의 렌더된 DOM은 <article>을 위젯 단위로 23개 흘리고 본문
// 9,400자는 그 밖에 있어서, 345자짜리 위젯이 본문으로 뽑혔다(실측).

Now I scored every candidate and let them compete. Four tags could be candidates: article, main, section, and div. I excluded body. It would always win on length and always include the page chrome.

The score multiplied three values.

점수 = 보이는 글자 수 × 문단 가중 × 링크 할인
문단 가중 = 0.3 + 0.7 × (p 안의 글자 비율)
링크 할인 = max(0, 1 − 1.2 × (a 안의 글자 비율))

Each of the two coefficients had a basis.

The link coefficient 1.2 was greater than 1. I set it that way deliberately. When converted to Markdown, a link-heavy node expanded far beyond its visible text because every anchor gained a URL. The page chrome on one Naver page contained 7,213 visible characters but became 16,977 characters in Markdown, while the article on the same page was only about 2,000 characters. Even with the link share discounted by 0.8, the page chrome still ranked first. 1.2 was the value at which the articles began to win across the captured pages.

The paragraph floor of 0.3 corrected in the opposite direction. Prose lived in paragraphs. A container with more text inside <p> elements was more likely to be the body. However, Naver rendered articles with <br>, and React-based sites placed text directly in plain <div> elements. Their measured paragraph ratios were 0.10 and 0.25. If paragraph content were a requirement, both would receive a score of 0.

The measured basis for the two coefficients was as follows.

Coefficient	Value	Measurement used to choose it
Link discount	1.2	Naver page chrome: 7,213 visible characters → 16,977 characters in Markdown; article on the same page: about 2,000 characters. At 0.8, the page chrome ranked first
Paragraph floor	0.3	Paragraph ratio of Naver <br> article: 0.10; React site: 0.25. Making it a requirement gave both a score of 0

The two values worked in opposite directions. The link coefficient had to be strong enough to suppress page chrome, while the paragraph floor could not drop to 0 because it had to preserve bodies without paragraphs. Adjusting only one side caused the other to fail.

Scoring method	What won
Length only	Navigation, headline lists
Semantic tags only	345-character widget (Dong-A Ilbo case)
Length × paragraph weight × link discount	Body

I counted the numerator and denominator with different functions

I made one major mistake in this score.

I counted total characters after passing the text through a cleanup function, but counted link characters from the original. Because the two values used different baselines, the ratio exceeded 1. The link discount then became 0, and the score did as well.

A 3,936-character <main> on Dong-A Ilbo lost to a 297-character comment widget. The actual body had a score of 0.

I changed the implementation to count total characters and link characters using the same criteria in a single traversal. If the numerator and denominator count different strings, the link ratio can exceed 1.

Length remained the dominant term

Yonhap News had a short summary <article> and a long body container. The summary used the semantically more accurate tag, but it lost because it was shorter.

A comment acknowledged this property.

The short story-summary <article> on Yonhap News still loses to the longer articleWrap—the length remains the dominant term in the score.

Because I wanted the body, the longer candidate was correct in this case. I documented that this was a consequence of the design rather than an accident.

The order of noise removal changed the result

There were two kinds of content to discard from HTML: UI controls and hidden elements, and lists of links.

I did not remove both at the same stage.

// reader.go:642-645
// UI-control/hidden pruning runs on the whole document, before scoring —
// see pruneNoise's comment for why that is safe here but not for
// pruneLinkNoise below.
pruneNoise(doc)

root := bestBodyNode(doc)   // 점수를 매겨 승자를 뽑습니다

// reader.go:676-678
// Link-list pruning is coarse enough to shift which container wins, so it
// only runs after the winner is fixed
pruneLinkNoise(root)

I applied the first pass to the entire document before scoring. Buttons and hidden divs were not body content regardless of which container held them.

I applied the second pass only within the winner after it had been selected. Link-list removal was coarse enough to change which container won.

Reversing the order caused link-heavy body content to disappear entirely, allowing another container to win. The order of cleanup and selection determined the result.

Some sites delivered the body as JSON

CMSs built as single-page applications sent the article as JSON and let the browser construct the DOM. The markup received by the server contained only the headline and no body.

In that case, the density parser had nothing to do. The body was not in the HTML.

I handled one case as an exception.

// reader.go:681-692
// Single-page CMSes ship the article as JSON and let the browser build the
// DOM, so the markup we receive has a headline and no story. Arc XP is the
// one worth handling: its payload is a typed array, so there is no guessing
// about what is body text, and one rule covers every publisher on it
// (chosun.com among them). This is a CMS shape, not a per-host selector.
if len([]rune(md)) < minBodyRunes {
    if embedded := arcBody(body); len([]rune(embedded)) >= minBodyRunes {
        md = embedded
    }
}

There were two reasons I handled Arc XP.

First, its payload was a typed array. I did not have to guess which item contained body text.

Second, one rule covered every publisher using that CMS. chosun.com was one of them.

This was different from the site-specific table I had removed earlier. The table used host names as keys and covered one site at a time. This rule used the CMS shape as its key and covered multiple sites.

When making an exception, I had to examine what the exception used as its key.

If that still failed, I used og:description

Even a page whose body was entirely page chrome had a summary.

// reader.go:693-697
// A page whose body is all chrome still has a usable summary in og:description;
// prefer that over returning boilerplate as if it were the article.
if desc := firstMeta(doc, "og:description"); len([]rune(md)) < minBodyRunes && desc != "" {
    md = cleanLines(desc)
}

A short summary was better than returning page chrome as though it were the article.

I gave failures names

Body extraction could fail in several ways. If every case was returned simply as a failure, the caller could not respond appropriately.

Two cases were especially dangerous.

// reader.go:698-707
// A challenge page and a deleted-post screen both have bodies, so the length
// checks pass and either would be returned as if it were the article. Name
// them instead: the caller can tell the user why, and a retry policy can
// skip a wall while still retrying a transient failure.
if isBotWall(md, title) {
    return Result{}, statusError(http.StatusForbidden, "BLOCKED_BY_SITE", ...)
}
if isDeadPage(md, title) {
    return Result{}, statusError(http.StatusNotFound, "PAGE_NOT_FOUND", ...)
}

Bot-block pages and deleted-post pages had bodies. They passed the length check.

Without names for these cases, "Access Denied. Please verify you are human." would be stored as the article body. The user would believe that this was the body of the link they had saved.

Naming the failures made two things possible.

Failure name	What the caller did
BLOCKED_BY_SITE (403)	Told the user that the site had blocked access. Did not retry
PAGE_NOT_FOUND (404)	Told the user that the page was gone. Did not retry
UPSTREAM_TIMEOUT (504)	Treated it as a transient failure. Retried
EXTRACT_FAILED (422)	Reported that the body could not be found. Retrying would produce the same result

The retry policy branched on these names. It skipped walls and retried transient failures.

Detection used both phrases and length caps

Both checks used a list of phrases together with a character-count cap. Checking only the phrases would catch real articles that happened to contain those expressions.

Detection	Phrases	Cap
Bot block	11	3,000 characters
Dead page	5	1,500 characters

The phrases came from measurements. When I opened Google Search in a real browser, 비정상적인 트래픽을 감지했습니다 ("unusual traffic has been detected") arrived as 704 characters; through the static path, 액세스하는 데 문제가 있으면 ("if you are having trouble accessing") arrived as 185 characters. A deleted X post returned 게시물을 찾을 수 없음 ("post not found") as 203 characters, and its title was the same sentence.

The cap prevented an article quoting a bot-block phrase from being classified as blocked. Documents longer than 3,000 characters passed even if they contained one of the phrases.

The limitation was clear as well. The phrases covered only Korean and English. Block pages in Japanese or French passed through. A list-based classifier could not catch anything absent from the list.

I used a library for Markdown conversion

After selecting the body node, I had to convert it to Markdown. I did not build that part myself.

There were two dependencies. I parsed with golang.org/x/net/html and converted with html-to-markdown. I added the base, commonmark, and table plugins to the converter.

Choosing not to use an external extraction API and choosing to use a conversion library were separate decisions.

	Built in-house	Library
Which node was the body	In-house	—
HTML to Markdown	—	Library
Entire body extraction	—	No external API used

Selecting the body had to keep evolving around our data. The markup used by Korean news portals was our problem.

Converting HTML to Markdown was closer to a standard problem. The rule that turned <strong> into ** did not depend on our circumstances.