How confident are you in your ability to detect AI-generated text and how likely are you to read it?
I’ve noticed I’m very disenchanted with AI “slop” and I have gotten pretty confident (probably overconfident) in my ability to pick up on AI-generated content. As a result, there are things that I struggle to read, or at least read thoroughly, because I believe they are AI-generated. I’d love to hear whether or not you all are having similar experiences, if you are still engaging with the content if it is well done enough, and how you are managing it all. So how about it? Are you confident in your detection ability? Why or why not? Do you read it anyway? Why or why not?
Bonus Question: I use generative AI a lot, so I have to be careful not to have a double standard. Do you find yourself holding other content to a higher standard that you own when it comes to the size of the role AI played in its creation?
One thoughtful comment unlocks the ability to create threads.
5 comments
I should probably preface this with the fact that I'm a NixOS, GrapheneOS user that chooses not to participate in the AI training data extraction "apps" they call Social Media. I'm not sure how confident I am in my ability to detect AI written stuff but as long as it is presenting the data that I'm looking to incorporate in something I'm interested in doing with my time I'm ok with it.
On the bonus question I have Claude re-write a lot of things for me for two important reasons, first I was born without a filter for things I have, at times, wish I would have taken a bit more time before responding and two when Claude re-writes stuff for me it just reads a bit clearer and comes across slightly less direct.
The way I see language is that it's a crude tool that's useful, no doubt, but it's still just some stuff people made up and then formed a relationship with, usually many different kinds of relationships. The best advice I got was from the guy who taught me yoga as a spiritual process, he said, "say everything you want to say with half the words." That practice is powerful, so powerful I told Claude to do the same and then asked about it afterword which was interesting to say the least. ~Eric
I'm confident that I can detect badly AI generated text, because most of the time that's clearly different than badly written human text.
Not so confident about detecting AI generated text that was then edited and iterated on. For example for technical documentation, I think you can get it to a point where it's hard to distinguish from something a human would produce.
If it really is slop, then it makes a big difference if you clearly declare it. If you say, hey I generated this with AI, didn't have time to edit or iterate, but there might be some useful info in there, then I am much more likely to read it or at least skim it, than if you post it without disclaimer and I notice in the second sentence that it's low effort AI content.
I’ve integrated Pangram to check every answer on this forum, detection rate is very good for answers over 20 words!
I tested on a set of evals that were half forum comments from before 2022 so no slop and half paragraphs from known AI generated sources, I would estimate my own accuracy is like 75-80% on this set while Pangram got 98% correct!
Personally I find Claude slop much more off putting than Gemini/Chat slop for some reason.