NYS Tax DatabasesAuthor: Dave Date: 04.01.26 - 12:41am So I ended up having to build a few databases of New York State tax materials — Advisory Opinions, Tax Appeals Tribunal decisions, TSB memoranda, and Division of Tax Appeals determinations — to run some research queries I couldn't run any other way. Download databases & source files (1.2gb unpacked) ( read me / examples ) The project converts a scattered body of New York personal income tax material into a structured, searchable corpus: adjudicated decisions, Advisory Opinions, and Technical Services Bureau memoranda, all indexed with SQLite full-text search. The point is to be able to ask questions of the whole published administrative record at once instead of chasing isolated citations — measure outcome distributions, find every reference to a given regulation, and see how often a particular sourcing or residency concept actually shows up over time. A corpus like this is also a prerequisite for using AI on the material at all. Ask a model about a specific Advisory Opinion or a sourcing holding cold and it will happily invent citation numbers, fabricate dates, and paraphrase rulings that never existed — confidently and in the right format. Grounding every query against the actual indexed text turns the model from a plausible-sounding guesser into something that can only cite what is really in the record. The database is what makes the difference between research and hallucination. Not the most fun thing I've ever built, but it scratched the itch. I'll be filing for an Advisory Opinion soon and will post a copy here once it's submitted.
How long does an Advisory Opinion actually take? Processing time turns out to be measurable — but only up to a point. Prior to ~2019, submission dates were recoverable, often encoded directly in the petition number. In more recent opinions the submission date is no longer disclosed anywhere in the published record — not in the text, the identifiers, or the metadata — so processing time can't be independently measured for them. I filled that gap with a FOIL request, and that's where it gets interesting. A data-hygiene note before the numbers, since three artifacts in the raw set would otherwise distort everything. 18 rows have received_date == issued_date (0 months) — spot-checking the 1997–98 source documents, those opinions simply have no issue date in the header (TSB-A-97(10)I jumps straight from "Income Tax" to "STATE OF NEW YORK"), and the original ingestion fell back to the receipt date when it couldn't parse one. Four (N.1) IDs show 105-month "delays" but are really republications/corrections of much older opinions (all four reissued on the same day in 1999). And five early-1980s rows carry a fabricated December 31 issue date — an artifact of an extractor that defaulted to year-end when it couldn't parse a real publication date. I excluded all three groups — 27 rows total — from the clean statistics below. That leaves a working set of 368 opinions.
====================================================================== PROCESSING TIME BY DECADE (issued year) ====================================================================== Decade Count Avg Min Max Median ---------------------------------------------------------------------- 1980s 108 5.8 mo 1 34 4.0 1990s 91 3.7 mo 1 12 3.0 2000s 79 7.4 mo 1 28 6.0 2010s 62 10.4 mo 1 36 8.0 2020s 28 41.1 mo 8 85 42.5The 2020s line is the headline. In the previously available data there was exactly one opinion in that decade (42 months). With the FOIL data there are 28, and the average barely moves — confirming that lone outlier was representative, not a fluke. ====================================================================== SLOWEST 20 OPINIONS ====================================================================== 85 mo [2024] TSB-A-24(1)I,(7)S recv 2017-06-15 issd 2024-07-16 [FOIL] 71 mo [2024] TSB-A-24(8)I recv 2018-05-07 issd 2024-04-24 [FOIL] 58 mo [2020] TSB-A-20(5)I recv 2015-12-11 issd 2020-10-20 [FOIL] 57 mo [2024] TSB-A-24(2)I recv 2019-07-05 issd 2024-04-18 [FOIL] 57 mo [2024] TSB-A-24(9)I recv 2019-06-27 issd 2024-04-24 [FOIL] 53 mo [2024] TSB-A-24(3)I recv 2019-11-12 issd 2024-04-19 [FOIL] 52 mo [2020] TSB-A-20(12)I recv 2016-06-16 issd 2020-10-27 [FOIL] 51 mo [2020] TSB-A-20(13)I,(4)C recv 2016-07-18 issd 2020-11-03 [FOIL] 51 mo [2024] TSB-A-24(13)I recv 2019-12-20 issd 2024-04-18 [FOIL] 49 mo [2024] TSB-A-24(5)I recv 2020-03-05 issd 2024-04-24 [FOIL] 49 mo [2024] TSB-A-24(12)I recv 2020-03-11 issd 2024-04-24 [FOIL] 44 mo [2020] TSB-A-20(3)I recv 2016-06-06 issd 2020-02-10 [FOIL] 44 mo [2020] TSB-A-20(9)I recv 2017-01-12 issd 2020-10-06 [FOIL] 43 mo [2020] TSB-A-20(4)I recv 2016-10-20 issd 2020-06-16 [FOIL] 42 mo [2020] TSB-A-20(7)I recv 2017-03-24 issd 2020-09-29 41 mo [2020] TSB-A-20(8)I recv 2017-04-13 issd 2020-10-06 [FOIL] 38 mo [2020] TSB-A-20(6)I recv 2017-07-17 issd 2020-09-29 [FOIL] 38 mo [2020] TSB-A-20(11)I recv 2017-08-23 issd 2020-10-27 [FOIL] 36 mo [2017] TSB-A-17(2)I recv 2014-03-06 issd 2017-03-01 35 mo [2020] TSB-A-20(2)I recv 2017-02-17 issd 2020-02-04 [FOIL]Before, the slowest list was dominated by the four phantom 105-month reissues. With those removed and the FOIL data added, the slowest 20 is now an almost solid block of 2020 and 2024 issuances. Pre-FOIL there was one entry over 36 months in the entire post-2000 era. Post-FOIL there are 18. ====================================================================== DISTRIBUTION OF PROCESSING TIMES ====================================================================== 0-3 mo 122 (33.2%) ################# 4-6 mo 109 (29.6%) ############### 7-12 mo 69 (18.8%) ######### 13-24 mo 36 ( 9.8%) ##### 25-36 mo 14 ( 3.8%) ## 37-60 mo 16 ( 4.3%) ## 61-120 mo 2 ( 0.5%)Historically about 63% of opinions came out within six months. The 37–60 month tail is almost entirely the FOIL-recovered set. ====================================================================== BY YEAR (received) - when did the wait start? ====================================================================== 2013 15.0 mo ( 5) ############### 2014 17.8 mo ( 8) ################## 2015 22.0 mo ( 9) ###################### 2016 34.0 mo ( 6) ################################## 2017 38.2 mo ( 9) ###################################### 2018 28.7 mo ( 3) ############################# 2019 44.0 mo ( 5) ############################################ 2020 30.2 mo ( 4) ############################## 2021 34.5 mo ( 2) ################################## 2022 21.5 mo ( 4) ######################This view is more meaningful than "by issue year" for what I was actually after — it shows what a petitioner who filed in year X really waited. File in 2013, wait about 15 months. File in 2019, wait 44. There's a clean monotonic climb from 2013 to 2019 with a clear inflection point. Submission year stops at 2022 because anything received in 2023 or later that hasn't been issued yet isn't in the database at all — those are still-accumulating, unresolved petitions. ====================================================================== THE 2024 BATCH - were these issued together? ====================================================================== 2024-04-18: 3 opinions TSB-A-24(2)I, (4)I, (13)I 2024-04-19: 1 opinion TSB-A-24(3)I 2024-04-24: 9 opinions TSB-A-24(5)I, (6)I, (7)I, (8)I, (9)I, (10)I, (11)I, (12)I, (14)I 2024-07-16: 1 opinion TSB-A-24(1)I,(7)SThis is the finding most worth flagging. 13 of the 14 opinions issued in 2024 came out in a single six-day window, April 18–24, with the fourteenth following on July 16. These weren't a backlog of uniformly old cases: their receipt dates span mid-2017 to November 2022, and the waits run from 17 to 85 months (averaging 44.8). The Department wasn't releasing them as they resolved — the entire pending queue went out at once, a petition filed in late 2022 alongside one filed in 2017, in a single window. That's hard to explain by ordinary workload management and looks more like a clearance operation.
======================================================================
SUMMARY
======================================================================
Total opinions: 395
Clean for stats: 368 (excluded: 18 zero-month parsing artifacts,
4 reissue phantoms, 5 Dec-31 fallbacks)
Overall average: 9.1 months
Overall median: 5.0 months
1980-1999 average: 4.9 months (199 opinions)
2000-2009 average: 7.4 months (79 opinions)
2010-2019 average: 10.4 months (62 opinions)
2020-2024 average: 41.1 months (28 opinions)
The headline shift: the 2020s average runs roughly 8.4× the 1980–1999 baseline (41.1 vs 4.9 months). The 2010s already showed a doubling, and then the 2020s break the chart entirely.
A couple of honest caveats on what these numbers can and can't say. The 2020–2024 average (41.1 months across 28 opinions) only covers opinions I have receipt dates for — the FOIL set plus the one self-disclosed 2020(7)I. Any 2020+ opinion that was issued quickly and wasn't in the FOIL request won't appear here, so the set is biased toward slow opinions. It's also the only observable processing-time data for that era, since the Department stopped publishing receipt dates around 2019. The bias arguably cuts both ways: the true 2020s average could be lower if there are fast-resolved opinions I haven't captured, or higher, since a FOIL request tends to surface exactly the cases the Department still had something outstanding to disclose. btw — if you've got a pile of records you wish you could actually run questions against, building corpora like this one (and doing the analysis on top) is the kind of side work I take on. Reverse-engineering the messy source, structuring it, and getting it to the point where a question has a real answer instead of an anecdote. Drop me a line. Comments: (2)On 04.03.26 - 9:00pm dave wrote:
On 04.06.26 - 7:46pm Dave wrote:
|
About Me More Blogs Main Site
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||