Long-term maintainance with regards to PAW-Architecture

Naturally the PAW was designed with an intention of longevity. However, especially as development draws to a close we foresee a handful of changes that in time are likely to be desirable.

Cutting out the problem-children of Grimoirelab

As the PAW matures, we expect the library of custom scripting to grow much faster than the library of inherited Grimoirelab enrichment features. In time then, the role of a single mordred instance will shrink to merely performing a perceval-call followed by running some custom code. The development process of grimoirelab can be opaque at times, which could pose a particular issue when it comes to long-term support. To this end we highly recommend that once the library of custom features has grown to the point where it justifies the PAW’s existance in it’s own right, to replace Mordred by a custom piece of software; dropping the reliance on the full Grimoirelab stack. This proprietary Mordred replacement can then simply call Perceval to retrieve the data, use SortingHat for helping solve identities, and store the data in the data store in its own way, possibly also harvesting their enrichment features from the GrimoireELK repository.

The role of ElasticSearch

ElasticSearch in the PAW currently functions as a store of internal memory, data is entered from Perceval with the help of GrimoireELK, and retrieved with ElasticWrap, possibly via the graphql-API. This means that all data has to be copied over into ElasticSearch before it can be utilised in the PAW. The only qualification ElasticSearch has to fullfill this function over any other is the absent requirement for formally structured data, allowing virtually any data to be stored as needed without reconfiguration. If all the data is already stored in a format that supports this the duplication is unnecessary and wasteful. Replacing ElasticSearch with a different technology may, depending on how different the query interface is, not be trivial however. If mordred and grimoireELK are still in use, mordred is responsible for adding the perceval-retrieved JSON data to the datastore, while the enrichment scripts of grimoireELK appear in no way standardized in terms of their data retrieval and updating pipelines. For ElasticWrap however, one would only need to modify 2 element. The main call to search_pit would have to be replaced by whatever call is needed to get results from the new data store most efficiently. Additionaly, the construction of the query would have to be refactored to correctly query the new data store. Once these changes are implemented, this new support should propagate to the datasources and Graphql-API.

Kibiter support

Kibiter, at time of writing, is no longer supported by the PAW due to Kibiter requiring an outdated version of ElasticSearch. In theory this issue should be solved in time by the CHAOSS team themselves, though even if it is, Kibiter has barely seen any attention during the development process, so while the initial exploration did not note any reason our changes to GrimoireLab would somehow break Kibiter compatibility, there have been no successful integration tests showing that Multi-Mordred does not somehow fundamentally confuse Kibiter in an irreconcileable way.