HN
login
The trouble with 'ntile()'blog.djnavarro.net

Please, please, please do not use ‘dplyr::ntile()’ in an analysis that will be submitted to a regulatory agency. It’s secretly an ill-conceived SQL tool that doesn’t do what you probably want it to, and it does not…

26 points•luu•10 days ago•3 comments•
#ntile#trouble#please#submitted#analysis#regulatory#agency#dplyr

3 comments

tmoertel8 days ago
Tl;dr: The trouble with ntile seems to be that, when the author of the post imagined what it did based solely on its name and she imagined wrong, she ran with her imaginary version way too long before checking the documentation, which was very clear about its behavior:

> Unlike other ranking functions, ntile() ignores ties: it will create evenly sized buckets even if the same value of x ends up in different buckets.

setr8 days ago
I’m pretty sure NTILE is just a generally cursed function. SQL NTILE also requires everything to be loaded in memory because of the odd rule that larger buckets precede smaller buckets, so it’s unusable on anything decently sized.

Not being able to specify how ties are handled (and choosing, as far as I can tell, a fairly useless definition) fits the bill.

tmoertel7 days ago
If it is "cursed" for a function to do what it is clearly documented to do, instead of what someone mistakenly imagines it to do, then what function isn't cursed?

> Not being able to specify how ties are handled (and choosing, as far as I can tell, a fairly useless definition) fits the bill.

Actually, you can completely specify how ties are handled—and should, in any study designed to be repeatable. The docs for dplyr::ntile tell us that:

> To rank by multiple columns at once, supply a data frame.

So, repeatable, completely specified tiebreaking is as easy as adding a tiebreaker column to the dataset, using whatever strategy makes sense for your study. For example, if we wanted random tiebreaking using R's built-in `runif`, all it takes is one extra line of code:

    data_table |>
      dplyr::mutate(
        tie_breaker = runif(dplyr::n()),
        ntile_bin = dplyr::ntile(tibble::tibble(partitioning_key, tie_breaker), n = 4))
Almost 100% of the original author's problems could have been avoided by just reading the docs.

Read the full thread on Hacker News →

Related stories