Managing tests through the feature life cycle

[jdesrosiers]

Sorry, I missed the discussion on #803 about moving the propertyDependencies tests to a new directory. I don't disagree with that move, but I realize that it isn't clear when these tests should be moved into the main suite. I think it should be when the feature reaches the "experimentation" phase because that's when we expect implementations to start implementing the keyword, but I imagine some of you are thinking it would be when the features moves to the "stable" stage. I want people's builds to break when a feature gets to the point where they should be implementing it. It's one way we can nudge implementers about new features. I just wanted to make sure we're on the same page on this.

This may be a bit of a tangent, but as an alternative, I was thinking about introducing a flag on tests that says what release or stage a test requires. Then people can easily filter out tests they want to ignore and we don't have to worry about separate directories for proposal tests.

Pinging @gregsdennis and @karenetheridge specifically for your thoughts.

[gregsdennis]

I'm not sure about that. I think while a feature is experimental, it should stay separate. It should only be merged into the main suite when incorporated into the spec and the spec is published.

I guess it depends on how we view the suite. Is it intended to be a representation of the spec, or is it just a tool for implementers? If the former, then features that haven't been merged and published definitely need to be in their own folders. If the latter, then I can see incorporation into the main suite being a message to implementers that this feature is coming.

---

A stage property could do if it's on every test at the schema level (or at least the proposal tests). My reasoning for putting it in a different folder is that there were tests spread throughout different files. Also, if I was the one upgrading a feature from experimental to stable, I'd find the folder approach makes it easier to not miss any.

[jdesrosiers]

I guess it depends on how we view the suite. Is it intended to be a representation of the spec, or is it just a tool for implementers?

I was definitely not thinking of it as a representation of the spec. I think it's more useful as a tool for developers.

A stage property could do if it's on every test at the schema level (or at least the proposal tests).

Actually, I was thinking of using the property for more than just stages. Once it gets added, it would record the release number where it was added. For example, it would go from "proposal" to "experimental" to "2027". That way something like bowtie can filter easily and say what level of support each implementation has by it's test results.

That's another think I'm just thinking about now! Bowtie might need to work a little differently for v1.

My reasoning for putting it in a different folder is that there were tests spread throughout different files. Also, if I was the one upgrading a feature from experimental to stable, I'd find the folder approach makes it easier to not miss any.

That's true, but it would be more maintenance. I'm not excited about having to move files when things are promoted. I honestly think filtering on a release property should be easy enough to use.

[karenetheridge]

My $0.02, which I think matches fairly closely with what Greg was saying:

• add some tests to proposals/ when there is enough detail fleshed out that it is possible to implement (even if some edge cases aren't figured out yet, the act of implementing can help resolve those issues)
• move from proposals/ to the main directory when the feature has been accepted and we do not anticipate any material changes are still to come

except for:

It should only be merged into the main suite when incorporated into the spec and the spec is published

This doesn't make sense because the v1 spec isn't "published" yet -- it's still being written. So by this definition, there would be nothing in the main directory for v1, and all of its tests would be in a side directory.

[jdesrosiers]

move from proposals/ to the main directory when the feature has been accepted and we do not anticipate any material changes are still to come

It's unclear what you mean by "accepted". What you describe sounds like the "experimentation" stage. At that point, no changes are anticipated and implementations need to start implementing them. It only gets added to the spec once enough implementations support it. But, if by "accepted", you mean it's added to the spec, that's different. So, can you clarify what you mean by "accepted"?