NeurIPS used uncalibrated AI detector for desk rejections

📰 Analysis

NeurIPS, a prominent AI conference, has been using an uncalibrated AI detector for desk rejections. This detector is used to evaluate the quality of submissions before they are sent to reviewers. The issue with this detector is that it is not calibrated, meaning it does not accurately reflect the quality of the submissions. This can lead to biases in the review process and potentially harm the careers of researchers. For AI/ML practitioners, this highlights the importance of using calibrated evaluation metrics and being aware of potential biases in AI systems. The incident also raises questions about the use of AI in academic publishing and the need for transparency in AI decision-making processes.