Inspiration

[martindurant]

First to be clear, I am 100% supportive of the ideas here, and the need for sharable data specification in the way you describe in the blog post.

Here are some more links for inspiration you might want to consult or mention.

• dandi archive ( https://dandiarchive.org/dandiset ) which is hosted n-D data, often in zarr format, but specialised to neuro. There are other "bring your data" services around with metadata requirements, e.g., https://ckan.org/ (which actually is more "deploy your own data hub")
• pangeo-forge ( https://pangeo-forge.readthedocs.io/en/latest/index.html ), an attempt at making existing datasets cloud-friendly. I expected better docs there for the "why", perhaps that design exists elsewhere. This should be mentioned only because you already spoke of the pangeo experience, and this was to be the next stage.
• many other attempts to standardize metadata description aside from source-coop (which I find not very complete); https://specs.frictionlessdata.io/ (rather tabular)
• various OOS catalogs like https://atlas.apache.org , https://www.unitycatalog.io/ which may be tied to specific tech stacks and data models
• some big data providers like NASA; just look at how many fields can be searched in one of my former go-to sites: https://archive.eso.org/eso/eso_archive_main.html ; this implies that any catalog model needs to be able not only to store that kind of detail, but to have specialised search interfaces for practitioners of different fields. They do a good job of linking each file (image) to a program ID with abstract and details (e.g., https://archive.eso.org/wdb/wdb/eso/sched_rep_arc/query?progid=075.D-0156(A) ) and any associated publications.

[davidgasquez]

I've been curating a list of "Open Data" projects and organizations here.

There are some amazing projects out there and would be amazing to figure out ways we could "bridge" the datasets somwehat to any new system / network.

[martindurant]

^ An interesting page with tons of relevant links - if a bit of a stream (my opinion). Many links are to documentation front-pages, though, so not clear what specifically can be learned from each.

I do take issue with the one concrete dataset description provided as an example ( https://handbook.davidgasquez.com/Open+Data#Datafile ). Yes, I realise this isn't meant to be a final template or anything

• no dataset description except a general string, some link and opaque "metadata"
• the source and pipeline are dependent on environment. How is this command to be interpreted? How does one connect to some database? It's not exactly provenance.
• the schema is obviously for a "table", and specialising to tables is sort of hinted throughout the doc
• the fields just have a simple description. As scientists, we would at the very least expect units; or better, link/encoding to a thorough ontology describing what these entities are

These are more or less my thoughts on frictionless too, which I suppose this is very closely mimicking.

[davidgasquez]

It is definitely a stream! I use it mostly for personal-documentation and to keep track of projects I'm interested on.

You're right with regards the Datafile! That is a remanent section that I should remove. This note started as a way to collect information about Frictionless but has moved a lot from that, including not updating the Datafile. I haven't though a lot about the specific of a potential datafile in a while as that touches a large amount of things (ontologies, LD, ...). I'd say that, right now, the thing I care the most is a way to make bridges bettwen the current packaging solutions to avoid the famous XKCD.

The thing you are spot on is that I've been mostly thinking about tabular datasets as I think (don't have any experience) you can represent or link to any kind of datasets from a tabular form (very loosely held opinion I'm not sure is even correct).

[martindurant]

the thing I care the most is a way to make bridges bettwen the current packaging solutions to avoid the famous XKCD.

I saw that link in the text :)

I've been mostly thinking about tabular datasets

This discussion thread and project started out of zarr and related ecosystem, which is already non-tabular, so that style of data will certainly be considered from the start, and possibly others.

To everyone: I wonder to what degree data can be described in the abstract, as many of these specs attempt, versus in code? Intake and tools like dbt tie data strongly to the code that will read or operate on them. Something like STAC tries to be abstract (if specialised to a specific field in this case), but in practice the implementation often still does leak into the abstraction. Generalised transformation pipelines probably cannot be represented without an implementation, even if that implementation is fairly generic like SQL.

[TomNicholas]

I wonder to what degree data can be described in the abstract, as many of these specs attempt, versus in code?

I think the entire solution to this data availability problem should be code-agnostic, specified entirely through protocols, schemas, and standards. I'm optimistic we can do this - it looks like S3-compatible storage (S3 is a protocol) containing Icechunk/Iceberg data (which have open specs), found via a global catalog with a common schema and federated via another open protocol.

[martindurant]

containing Icechunk/Iceberg data

So you are thinking to piggy-back on these technologies explicitly? It certainly limits scope, but of course most data in not in those formats. In some cases, of course, data in other formats can be indexed as it it were...
How do you describe provenance - is it via links to doicumentation/code? Makes me wonder if anyone ever read the very many HISTORY tags you often see in a FITS data unit (or dicom or your favourite thing).

[TomNicholas]

containing Icechunk/Iceberg data

So you are thinking to piggy-back on these technologies explicitly?

I'm not suggesting explicitly only supporting those formats (it should be a point in the protocol open to extension), but I am suggesting that those formats have a confluence of features that make the cataloging and distribution problem much easier. See dedicated discussion in #3 (comment).

of course, data in other formats can be indexed as it it were...

Yes, I think you could get a long way with Icechunk + VirtualiZarr alone, and those ideas could be applied to other data models.

[jsignell]

STAC tries to be abstract (if specialised to a specific field in this case), but in practice the implementation often still does leak into the abstraction

I think this is kind of by design. STAC is trying to be a practical standard where the implementation and the abstraction are eventually consistent but it is ok for one to get ahead of the other. The idea is that when you have an implementation running alongside an evolving specification the specification gets tested in real-time and you find any weaknesses or ambiguity which you can then go back and iron out in the spec.

[TomNicholas]

Some more inspiration, this time on the idea of including pub-sub functionality:

• NASA CMR Ingest subscriptions - these are literally a subscription which "allow a user to be notified when specific collections/granules [i.e. NASA CMR datasets] are created, updated, or deleted."
• The Element84 EarthSearch SNS notifications feature

These examples show there is a desire for a pub-sub functionality, but ideally it would build upon a cross-org general-purpose network.