Parsing the Infamous Japanese Postal CSV

Late past twelvemonth I released posuto, a package presenting Japanese postal code data successful an easy-to-use format. It's based connected data released by Japan Post, which is infamous for being wide utilized but difficult to parse.

This adorable characteristic by Irasutoya is cute, but the earthy postal CSV information is not.

I first became alert of the postal information erstwhile I entered my postal codification successful an online shape and it auto-completed my reside arsenic "XXX-borough (except the following buildings)". I had nary thought what that parenthetical was referring to, so I looked for a communal root of postal data, recovered the CSV, and recovered the issue. It turns retired the CSV record contains parenthetical notes for anyone reading the CSV record and makes reference to the bid of the rows.

This causes problems. The information is chiefly useful 1 statement astatine a time, wherever the parenthetical is meaningless. Since CSV is simply a field-delimited format, there's also nary request for parentheticals - you could conscionable adhd a statement field.

This is only 1 of galore issues pinch ken_all.csv. You tin find people complaining astir it regularly on Twitter, and location was even concisely a blog conscionable collecting posts from each complete the web about it. A particularly amusing tweet describes group who expect computers to crook to the will of humans being punished successful Hell by having to parse ken_all.csv forever.

The README for the record explains that lines pinch overly agelong fields will be broken up into aggregate lines. Specifically, if the vicinity sanction is over 38 characters, aliases if the half-width katakana (half-width katakana) pronunciation section is complete 76 characters, the statement will beryllium divided into two lines. The overly-long vicinity section will beryllium continued and each other fields will beryllium duplicated. This is an abbreviated sample of what that looks like:

12345,Tokyo,Minato,This spot sanction is really 12345,Tokyo,Minato,very agelong it didn't fresh in 12345,Tokyo,Minato,a azygous statement truthful we had to 12345,Tokyo,Minato,split it

The information for this is not explained. Maybe location was a fixed-width buffer for storing a statement location 30 years ago. I utilized to process CSV and other files from hundreds of different providers astatine an aged occupation and I saw many horrors, but I've ne'er seen this peculiar formatting prime anyplace else. It should besides beryllium noted that while the magnitude limits are arsenic stated, the location wherever statement breaks are inserted successful agelong lines appears random, occurring neither astatine the characteristic limit nor astatine normal connection boundaries.

It's worthy noting not each the issues pinch the CSV are inherently technical; postal codes are ever complicated. The postal codification pinch the astir rows successful the CSV - a stunning 66 - is 〒452-0961, which refers to the Haruhi region of Kiyosu City successful Aichi Prefecture. This has that galore lines because every neighborhood gets a abstracted line. (This peculiar lawsuit whitethorn beryllium related to Haruhi having been the smallest municipality by area successful Japan from 2006 until 2009, erstwhile it was incorporated into Kiyosu City.)

In contrast, the longest continued line, utilizing the statement break rules above, is the introduction for 〒602-8368 aliases 〒602-8374, some pinch 8 lines. These are some in one of a fewer areas successful Kyoto that uses a unique, bizarre strategy of intersection-based addressing. The introduction looks a spot for illustration this:

12345,Kyoto,Kyoto,"North Town (Up Lower Godsroad from" 12345,Kyoto,Kyoto,"the West, Down Turtle Street from the" 12345,Kyoto,Kyoto,"East, Up Old Temple Road from the" 12345,Kyoto,Kyoto,"West)"

I person utilized quoted fields here, but the existent CSV doesn't quote fields and instead uses a different benignant of comma.

There are different issues. There are catch-all postal codes for galore areas, where the vicinity is fixed arsenic "except the following", and the only point to do is look for that nonstop drawstring and exclude it. There's a assortment of similar strings, and it's difficult to beryllium judge I've caught them all.

An illustration of different remark is 一円. Normally this would mean "one yen", but it besides intends "the area surrounding", and is simply a statement successful the CSV that should be removed from vicinity names, except for precisely 1 vicinity successful Shiga where that's really the sanction (〒522-0317).

There's besides a separate romaji file offered by JP Post. It's updated less often than the main files, is often retired of sync, and the provided romaji are highly debased quality. For the infinitesimal I'm still providing the data in posuto successful the sanction of consistency, but honestly you should conscionable use cutlet. To springiness an illustration of bad romaji:

大手町 JAビル OTEMACHI JIEIEIBIRU

What's happening present is that "JA" is being converted to the phonetic reading in Japanese, "ジェイエイ". Then ジェ, which is written "large ji mini e" but pronounced "je", is being converted to "jie" by treating the mini characteristic as though it were large, and the different characters are translated as-is, turning something already successful the italic alphabet into alphabet soup. For contrast, cutlet has nary problem converting "JAビル" into "JA building" (case handling admittedly needs immoderate activity still). Similar issues move "Roppongi Hills" into "Roppongihiruzu", and "Sweden Hills" into "Suedenhiruzu".

Anyway, dealing pinch the record was a humbling instruction successful the magnitude of complexity it's imaginable to battalion into 1 place. I've glossed complete galore details, but you can find them covered successful posuto's README.

You tin usage posuto arsenic a library, aliases if you're not utilizing Python, just download the pre-processed JSON and make usage of that. If you find a bully use for it I'd beryllium delighted to perceive astir it.

Oh, and if you request a Win3.1 aliases DOS programme to transcript the information onto an IBM H floppy disk, conscionable cheque the bottommost of JP Post's page - they've got you covered. Ψ