DevTools Hub

Search tools

Search for a developer tool

Why You Can't Detect a Delimiter by Counting Characters

Part of the Markdown Toolkit

Paste a spreadsheet row like Alice, New York, NY, USA next to a tab character, and a delimiter-detector that just counts characters gets it backwards: there are more commas in that row than tabs anywhere in the file, so it picks comma — and silently turns one address field into three columns. The fix isn't a smarter character count. It's asking a completely different question: not "which character shows up most," but "which character, if I split on it, makes every line agree on how many columns there are."

Raw counts measure the wrong thing

Take three tab-separated lines, one of which happens to contain a comma-heavy address:

Name	Location
Alice	New York, NY, USA
Bob	Boston, MA, USA

Count characters across the whole input and comma wins, 4 to 3 — there are simply more commas in those two addresses than there are tabs in the entire file. But splitting on comma produces [1, 3, 3] columns per line: the header has no comma at all, while the two data rows each get fractured into three pieces. Splitting on tab produces [2, 2, 2] — every single line agrees. Column count, not character frequency, is what a table actually needs.

Scoring agreement instead of occurrences

Markdown Table Generator tries each candidate delimiter — comma, tab, pipe, semicolon — and for each one computes how many columns every line would produce. It takes the most common column count for that delimiter, then counts how many lines actually hit it, and scores the candidate as (lines that agree) × 100 + (columns). The ×100 isn't arbitrary tuning — it just has to outweigh any plausible column-count difference, so agreement always dominates the comparison instead of needing to be balanced against it:

input: "Name\tLocation" / "Alice\tNew York, NY, USA" / "Bob\tBoston, MA, USA"

delimiter ","

columns per line
[1, 3, 3]
raw occurrences
4
lines agreeing on max
2 of 3
score
2×100 + 3 = 203

more raw commas, but one line disagrees on column count

delimiter "\t"

columns per line
[2, 2, 2]
raw occurrences
3
lines agreeing on max
3 of 3
score
3×100 + 2 = 302

fewer raw tabs, but every line agrees → wins

Tab scores 302 against comma's 203, even with one fewer raw occurrence in the entire input. A single noisy line — one address with embedded commas — can't outvote the two lines that were already telling the truth.

The same idea, run on a header row you didn't ask it to touch

This is also why pasting an already-rendered markdown table back in re-aligns it instead of duplicating a row: a GFM delimiter row — cells that are nothing but hyphens, optionally flanked by colons — gets detected and dropped before the table is rebuilt, rather than being split like data and silently adding an extra body row of --- strings. Detecting "this row isn't data" is a smaller version of the same structure-over-content idea as picking the delimiter: look at the shape of the row, not just what characters happen to be in it.

Why the first row still wins when columns disagree

Choosing the right delimiter doesn't make every line agree in practice — a trailing empty field or one truncated paste is common. Rather than take the max column count across every row (which lets one malformed long row force an unlabeled extra column onto an otherwise-clean table), the generator fixes the column count to whatever the first row produces and conforms every other row to it: a short row gets padded with empty cells, a long row gets truncated. The header defines the schema; everything else either fits it or gets trimmed to it.

Try it yourself

Markdown Table Generator turns CSV, tab- separated, or pasted spreadsheet data into a properly aligned GFM table — or re-aligns an existing one that's drifted out of sync after a hand edit — entirely in your browser. For converting that same row-and-column data into JSON instead, see CSV to JSON.

Related tools