Observation Accuracy Experiment v0.4 is Underway

Another one? And just after a solid week of identifying for the City Nature Challenge - I dont think I ever want to have to do an ID again!!!!!

Thanks @tonyrebelo - we very much appreciate all the IDing help. We know the timing is not great given City Nature Challenge. Thanks for all you do

I identified 13 species of the order Odonata, but I don't know how good a test this is, as the majority of them were among the most easily identified species of dragonflies, mostly from eastern United States. Much better tests would be to dive into the tropics if you want to know what percentage of species are being correctly (or at all) identified. Perhaps you are doing that also.

I appears that the plea page must be responded to, in order to turn off the red mail flag.

@arnim - a message just needs to be viewed for the red notice to go away, it doesn't need to be replied to.

Three of the nine I received were observations I had already made identifications on.
Two of those three were ones on which I made the "Improving" ID to the currently research-grade taxon.
How should I deal with this?

Thats right. If you've already ID'd an obs and you still stand by your ID, please skip. Otherwise. update your ID. Thank you!

A couple native species in the set this time yay!

I had one obs that was an images showing different species case. I did the usual, IDed (disagreeing) as common ancestor, left a comment and added my (the first) 'no' vote to 'Evidence related to a single subject'.

The new strategy for matching observers to observations seemed to be an improvement this time, at least for me.

However, it is less and less clear to me what purpose this experiment serves and what can be learned from it.

What does the "accuracy" being measured really tell us? The likelihood that any randomly selected observation will already have an ID that experienced IDers on iNat agree is correct? Whether a randomly selected observation can still have its ID refined? These seem to me to be fairly trivial.

The inclusion of non-research-grade observations in the last couple of rounds means that we are also looking at observations that do not yet have a community consensus -- i.e., I have several where the only ID is that of the observer, or it is a general ID like "flies" that could probably be refined by a specialist (which I am not), or it is at a high level because there is a wrong ID by an unresponsive observer and not enough people with relevant expertise have looked at it to override that ID.

What are we measuring in such cases? If I ID it as "flies" does that indicate it cannot be refined further? If I add a refining ID or add the additional ID that overrides a disagreement, what does this tell us except that the observation had not yet been reviewed?

I'll look at my set and add IDs to those where I can provide a meaningful contribution, but I honestly don't see any point in adding additional confirming IDs to common local plants or honeybees. There are lots of other ways I can spend my time that would do more to improve iNat's data.

Done!

How many of these are there going to be?

We can leave it without an ID, since it goes to Casual with the new DQA - was in the info from iNat.

(I did the usual, IDed (disagreeing) as common ancestor @richyfourtytwo fourtytwo

I got 6 observations which seems to be the average.

One of my observations was Rubus armeniacus, which is a hotly debated taxon! Maybe someone else will say it's R. bifrons and the iNaturalist curators will try to sort it out a bit more. It's really a mess.

I only got 2 observations to ID and I’d be happy to do more. Wish I got a couple of South American Mangora, as they are my favorite, but oh well. Maybe I didn't meet the requirements for those :(

@davidenrique agreeing IDs with included comments show up in your notices? My apologies, I didn't realize that would happen! I thought documenting all IDs related to this experiment would be valuable (even though it was more time consuming than just clicking the agree button); without that, it could be confusing to see multiple new IDs being added to observations that had been research grade for years, and those added IDs could be interpreted as trying to run up one's ID totals.

Several of the observations I was asked to ID in Observation Accuracy Experiment v0.4 were out of my state. I usually do not attempt things out of state because I do not know as much about what is possible and there are plenty of observations to work on from my state.

I have done my duty!

@huttonia Yeah, maybe next time. Turns out there weren’t any South American Mangora in the set of observations at all, so I got unlucky lol.

Is it acceptable to ask the observer for more info? I have one to ID that is a plant that seems to be something that does not grow naturally where the observation is sited (even though obscured). I have asked them if it is a cultivated plant (not marked as such), but I don't know if that is appropriate for this experiment.

@xris I know this isn't what you want per se, but there IS a way to restrict who adds observation fields to your observations. You can go your account settings> Content & Display > and change the "Who can add observation fields to my observations?" option to "curators" or "only you".
I don't recommend doing this, as observation fields can be useful, but that's an option if it's really affecting you too much.

@davidenrique I don't mind them doing it. We've corresponded about their Project, and I want to support it with my Observations. However, I have a lot of Observations that are similar with respect to their selected Observation Fields. I'm aware that they're doing this work. I don't need to be notified about it every single time!

@tiwane It doesn't when it's my own Observation.

Done and done! Was able to ID most of them but there was a really tricky Sterna tern that was eluding me, so had to leave it at genus. Thanks for reaching out, will be happy to help (if I can) with any further rounds of testing!

It took me a while to figure out that I had to edit the url from "inaturalist.org" to "inaturalist.ca" before I could even log in. Then the same thing happened again when I clicked on the link in the original message, to get to this blog to see if anyone else is having the same problems. Pretty annoying to have "incorrect password" returned over and over when I KNOW I'm entering it correctly (for .ca, not .org, it turns out).
Greg

I have been asked to ID an observation which has the location set to Private. However, without that info I can't ID to species level. Should I still attempt to ID? Any other instructions on how to deal with this case?
Thanks!

Would be great to have a 'multiple observations of the same organism' box to select in the Research Grade Qualification list. I often ID unknowns and there are often numerous observations from one account of the same organism. I leave a comment when I come across it to inform them of what to do but a lot of the time it's ignored. It must surely skew the data when 10 observations of the same organism all reach research grade?

@kmackau make a feature request.
I won the multi-species battle, you tackle this one?
(Meanwhile I have a copypasta for that)

Only one for me, as I am still not able to ID much. And I believe that it was two photos of different species. Does this mean that I should go through my observations to check for the same? Ha

Think about it - long, and hard @kmackau . Write the text, and edit.
Then go to the Forum and - add feature request. For another new DQA? That would be the elegant guidelined solution (especially if it also tips to Casual, until resolved by the observer)

PS - new route (mine was in the email to tiwane days) - https://forum.inaturalist.org/t/about-the-feature-requests-category-please-read-before-posting/69

My copypasta for these problem obs

See also (insert other obs numbers here, which is ... tiresome to do, but useful for the next identifier, and an eventual techie / DQA solution)
~
Please combine multiple pictures of the same individual of a species
https://forum.inaturalist.org/t/how-to-turn-multiple-observations-into-a-single-observation/9838

@dianastuder - looks like we shared quite a few observations to check. Not surprizing perhaps, but perhaps one way to rationalize number?
I dont mind the numbers. Would be happy to to say 100, if it was my interest group. One observation in my set (https://www.inaturalist.org/observations/21582820) took as long as all the others combined: a Giraffe in a zoo in Bakersfield - raised hackles: sun high for 19h22, no locality, two observations on at 19h30 is gloaming, no website zoo, farm or petting place shows Giraffe in the area. Looks like Maasai, but I wanted to check against zoo records, but cannot place it. [I have only identified the two subspecies of Giraffa giraffa, but this was only ID'd to a deprecated generic level because of a swap following the creation of 4 species from previously 1 sp and 7 subspecies, so I guess I was included for a California ID; my other way-out one was a sterile Pterocarpus to genus in Mexico, but their species are different from our African ones, so I just Dicotted it).

I'm confused, I clicked on v.04 and 9k observations came up. Am I doing something wrong and does it matter if I, ID those? I haven't had anything sent to me via email.

Please remove me from inclusion in any future versions of the accuracy experiment, I think that the experimental design where I am asked to ID plants in areas where I would NEVER id (Mexico, California, Arkansas for example from this round of 108 observations (5 hours)) means that the experiment is deeply flawed, even with me opting for the high level taxonomic groups that eliminate my input- while wasting my time.

@loarie A few points of clarification about ajott's question:

1.) If I have three improving IDs to a genus in a particular country, and an observation that I had previously ID'd to genus is part of the experiment, would I still count as a validator if I change/refine my ID (even though I was not sent it originally)?
2.) I the answer to 1.) is yes, when is validator eligibility determined/is it fixed? For example, I notice that before this experiment it appears I had exactly 3 leading and 2 improving IDs of genus X in country Y. One of my leading IDs was on an observation selected for the experiment, so now that leading ID has been converted to improving. Would that make me eligible validator for that genus in that country now? If so, if I refine that one ID to agree with the community taxa, would I lose eligibility because my active ID would be supporting?
3.) On another observation I was sent, it seems like there is a chance the original community taxon was either wrong or too confident and might either be backed up to a higher level or actually change to a different taxa. Would a change to the community taxa to one where many of those of us who were sent it would not be eligible as validators cause us who were sent it to lose eligibility as validators, and would others gain it? (this was my main question, the first two questions were just setting it up; it seems important because we almost certainly are more qualified to say what it is not than what it is)

Depending on the answer to question 3, I think it could be an interesting follow-up experiment to re-send just the subset of observations that were scored as 'incorrect' or maybe also 'uncertain' to a larger set of potential validators (perhaps a higher target redundancy and inclusion of a more expansive set of validators from the common ancestor/new taxa), to see if the community taxon changes further. This could be an interesting cross-check of the validity of the validators in disagreements, and might help refine the ambiguity of the 'uncertain' category.. Obviously you would archive a frozen version of the results before re-sending anything, and such a set would not be random.

@eyekosaeder Why not auto-email yourself once a month with a link? such as:
https://www.inaturalist.org/observations/identify?order_by=random&place_id=7207 (i.e. Needs ID random for Germany [or other area]). Skip those that dont interest you.

My subset size jumped to 82 with this one and most seem to be good matches for what and where I usually ID. There's one outlier in a different country but I have in the past indeed added IDs there so that's just icing on the cake. And even more fun: One of my own observations is included in this sample. Awesome to see the IDs pour in! It's RG at genus level (cannot be improved) and so far nobody has dared to take it to species yet (which is good, because I honestly don't think it can be narrowed based on the evidence provided).

My subset has 37 observations to ID. Piecemeal compared to what I usually identify. So just keep 'em coming. I have opted out of the CNC treadmill. In central Alberta it is barely spring when the CNC happens. I funnel my resources to more meaningful projects (such as this one) instead of IDing little snippets of greenery or leafless shrubs and trees. The observation subset is now fairly well matched to my identifying skills, which is great. From what I see now, most ID revisions happen to substandard observations that have blurry images, include more than one species, or lack locality data. It is too bad that such observations influence the end results negatively. I am looking forward to the outcome of this experiment.

I felt confident enough to provide a species-level ID on 11 of the the requested observations, genus/section-level ID on 4 observations, and I did not have the confidence to provide any ID for one. Thanks for letting me help out!

I am curious about something that came up. I am wondering how this experiment handles things like this.

https://www.inaturalist.org/observations/4580033

This observation is a cottontail rabbit track, confirmed 7 years ago by several certified wildlife trackers. However, the experiment 0.4 asks folks who have previously identified cottontail rabbits to identify this track. I think the AI doesn't have fine enough sorting capabilities to sort out track images from images of the actual animals. So, people who may not necessarily be trained in tracking are being asked to identify something that's outside their wheelhouse. (A track rather than an animal.) That in turn lowers the quality of the original observation from "cottontail rabbits" to "mammals." So, in effect, this has made the classification of this observation drop to a broader category than it originally was. Is this something that the folks doing the experiment are testing for? I'd be interested in seeing the results of this experiment. Or maybe an experiment designed specifically for animal tracks and sign?

I identified all 14 observations I was asked to look at. Most were the animals themselves. Curious if others got a mix of types of observations too? I did get a few tracks in the first rounds.

This is fun. Keep it up. Interesting thoughts are coming to mind out of this experiment.

That observation is still "Cottontail Rabbits". No-one has disagreed, so that hasn't changed.

@beartracker you can experiment with how the CID algorithm reacts by adding and removing your ID - a broader ID without disagreement has no effect on CID.

@ditchingit the 'non-blind'/transparent ID process here is the same as for every museum and herbarium specimen that gets IDed by a second, third, etc person: each new identifier has full access to/can see previous det slips, including who made the IDs and what those IDs were. I understand your point, but iNat's approach is the norm rather than the exception

I was asked to participate despite being a generally broad-only identifier, not sure I added anything of value haha. I got a couple blob style photos I couldn't make out and would ordinarily just hit reviewed and skip, had to put a non disagreeing "life" ID on one that already had cID at order which I'm sure will look very silly to anyone looking back at the obs without context.

Done. This time was much easier. I had 17 observations mostly from regions ans species I am familiar wiht. IDying was much easier than the last time. I had one observation without location whicht did not make much sense to me.
However, it was great fun for me to participate!
Is there anything wrong if I would use the filter to through observations in my aera and ID them?

Thanks for the great discussion and for participating. We've already had over 3k validators respond and over 94% of the sample validated. We can't tell you how grateful we are to be able to engage the expertise of so many of you to do this work. This incredible community of expertise really is what makes iNaturalist so special and we are so appreciative for your participation.

So far, the results are looking almost identical to past experiments with the research grade subset having 95% correct, 1% incorrect and 4% uncertain.

A lot of the conversation here is about whether the average accuracy of the iNat dataset is the right thing to measure. Given how biased it is towards common species, would it be better to orchestrate experiments to estimate the accuracy of rare species? I wanted to talk a bit about this topic.

The graph below shows dragonflies and damselflies (order Odonata) on iNaturalist. There are about 6000 described species of Odonata. On iNaturalist we've observed almost 4,000 of them. The black curve shows the number of observations by species sorted from most observations (Blue dasher) on down. This is on a log scale which shows you just how biased the iNat dataset is towards common species. Blue dasher has almost 100,000 observations, but only ~1000 species have at least 100 observations. Pale bluet is an example of a species with 100 observations. Many species like Dull Jewel have only 1 observation. Others like Swamp Groundling still have 0 observations.

The vertical line separates this subset of ~1k on the left side of the graph with >100 observations (lets call them common species) from the subset of around 5k on the right side of the graph with <100 observations (lets call them rare, data deficient species). This also roughly corresponds to the set of species in the Computer Vision and Geomodels vs those that don't have enough data. The pink line shows the number of IDers with at least 3 improving IDs (candidate validators) for each of these species. The ratio of candidate validators to observations is about 5%, ie a species with 100 observations has 5 candidate validators.

One of the interesting things about iNaturalist is that part of our mission is about getting lots of people connected to nature and part of our mission is about scaling global biodiversity monitoring. The former is focused more on the left side of the graph. Most of the dragonflies most of the people using iNaturalist are going to see are going to be species like Blue Dasher, and its really important that iNaturalist works well and provides good information about these species. The latter is focused on the right side of the graph, most of the discoveries and important contributions to science and conservation are probably coming from this frontier of rare species.

Much of our philosophy for bringing these rare species into focus is just to grow iNaturalist as a whole. That lifts the whole curve. As iNat grows, yes we'll get many many more encounters with Blue Dasher (which is really important for our mission of building broad advocacy for nature), but we'll also get more rare species too.

But because these experiments are using a random sample of iNat observations, they are biased towards the left side of the graph. For example, in Experiment v0.3, there were 134 common Odonata in the sample of which 128 were correct giving us an estimate of 95% with pretty high confidence. There were only 3 rare, data-deficient observations in the sample. All 3 were correct, but this estimate of 100% is very uncertain and low confidence due to low-sample size.

It sounds like there's quite a bit of interest about trying to estimate the accuracy of observations of these rare species. We can design a non-random sample that will do that. With fewer candidate validators, it will be harder to orchestrate and we may need different validator criteria. But we agree it would be very interesting and important towards better understanding the right side of the graph.

@misumeta I can't say for sure without checking, but it feels like each time I get more and more, not complaining, initiative leads to that. :)

this frontier of rare species

@loarie then please reconsider destroying placeholder text. Your rare species are often carefully named in 'shall we use this as a placeholder for you?' Yes, thanks!
That is what I use to - flag for curation - please add missing species.

We have discovered that iNatters can get around that working as intended, if they opt out of CID. Then the observer's intention is respected. But only ONE iNatter has shown me that.

An experiment that tells us that RG observations of common and easy to ID species are generally ID'd accurately is not very informative. It does not require monthly experiments to confirm the results.

There's a difference between "rare species" and "difficult to ID species". I actually suspect that the really rare (and unfamiliar) species are probably ID'd fairly accurately, because the average user does not know enough to suggest them, particularly if they are species not included in the CV.

While many commonly observed species are also easy to ID, there are plenty of species that are common but difficult to ID for one reason or another. For example, I've been reviewing European Xylocopa observations. These are big, conspicuous bees and consequently observed fairly frequently (23,000+ observations in Europe). In the areas where the ranges are known to overlap, the misidentification rate of RG observations (wrong species, or a species level ID where it should be left at genus or subgenus) is quite high. I haven't been keeping statistics, but I'm fairly sure it is well over the 5% error rate of the iNat-wide experiment.

While such taxa may not make up a large enough portion of the dataset to substantially affect the overall accuracy rate, this does not mean that misidentification of these groups (and lack of expert IDers) is not a problem. Most people are not using iNat's data set as a whole; rather, they are interested in specific taxa. It means very little to be told that most plants are correct if one is studying mosses or ferns or even one of the tricker groups of vascular plants (daisies anyone?). The fact that honeybees are generally correct is of little value for someone studying solitary bees. Etc.

For difficult taxa, a high accuracy rate is a likely to be not a reflection of how well the average user manages to ID the observations, but of the existence of a handful of indefatigable IDers who have gone through and corrected all the observations. Talking about "accuracy" in an abstract sense fails to acknowledge how much this depends on the knowledge and effort of specialist IDers, or the fact that if these people were to stop IDing the accuracy of that taxon would likely decrease substantially. For these taxa, the community is not largely self-correcting, because IDing requires knowledge and skills that the average user does not have.

INat is currently incredibly slow, trying to upload 50 observations for the last 3 days now. Thanks for the invitation to take part in the Accuracy Experiment v0.4. I will certainly consider such an invitation as soon as iNat restarts working at normal speed.

There is a general problem with images getting to "research grade" too easily. The following scenario occurs extremely frequently:
An observer posts an image they cannot identify. A second person offers an identification for the image. The observer then agrees with the suggestion, but actually has no personal knowledge to back that up. Bingo, the image is "Research Grade" but in reality only one person has offered an identification.

I suggest that where the poster has not initially identified the image, then their agreeing with an offered ID will not count. Two independant ID's should be required.

I worked through these very fast because I'm teaching a class. I hope I did OK. Most were very appropriate observations to ID. I did chuckle at the palm tree from United Arab Emeriates, not a place this botanist from Oregon usually ID's for, but I have to admit that I did ID a number of photos there once, when there was a problem school class. Unfortunately, all I could do with the palm was Palm Family.

Yes, I did chuckle a bit at a Rubus armeniacus/bifrons observation. We argue over the best name, so hard to know what the name posted on it may mean.

Do these need to be done by the end of the 13th or before the 13th? "By the 13th" can be interpreted both ways.

I hope by the end of the 13th - still have a few I wanted to check keys for but have to make it to the finish line for the semester first. Final exam week, graduation events over the weekend, and grades due by midday today has taken priority.

I just want to point out that 23:59:59 UTC is not the end of the day in most time zones, here in the eastern US in daylight savings time it is 1 second before 8:00 PM

We've update the post above with the results - thanks again everyone!

@traianbertau - but in none of the Eristalinus megacephalus currently in southern Africa https://www.inaturalist.org/observations/identify?quality_grade=needs_id%2Cresearch%2Ccasual&taxon_id=359895&place_id=113055 have you stated this, or have you enforced it (by disagreeing when you added a subgenus/genus level ID - I see you have in some other observations).

@tonyrebelo

Yes, I should have disagreed, but this is nothing one can support with a scientific paper, just a well known problem.

As @insectobserver123 pointed out above, 23:59:59 UTC is not the "end of the day" for half of the world, and it wasn't an intuitive interpretation for me. Since this is a global effort, I think it would be a good idea to be more specific about the deadline in the invitation email.

I deferred identifying about half of my batch - mainly ones in distant areas where I wanted to check for local lookalikes - until the final day. I thought I had finished with hours to spare, but apparently I identified dozens of them after the deadline.

Do you have stats about accuracy versus the amount of people who have identified it?

@kroeckx, do you mean are we storing the ID ledger on sample observations before the experiment kicks off? We are not - but its something we could do. Its partially possible to rewind/reconstruct an ID ledger to a previous point in time but not perfectly because (1) people can delete IDs and even observations, (2) while only one ID per person can be current at any time there are weird edge cases where someone could, for example, manually mark a more recent ID as not-current and an older ID as current that we can't reconstruct.