Dependencies, Versioning, and Wheels

[hunhoffe]

The build/distribution system is pretty complex at the moment. To build reliable wheels that provide a consistent development experience, I think what needs to happen is:

• The README(s) are updated to encourage folks to use the tag of the repo that matches the version they are using; this can help keep programming examples in sync with the wheels (I believe this may help with issues such as Examples dont work with fresh install. #2804)
• The process of updating LLVM should be done automatically, unless there is an issue, and well documented in order to keep the dependencies in sync. When we bump MLIR, we also want to bump the eudsl extras to keep them in-sync. I've just started work towards this here: [WIP] Workflow for bumping llvm #2806
• Since the eudsl extras are kind of a pain to install (the environment variable precludes normal pip install; the flag that is supported to install via a requirements file is very specific to particular pip versions). There are two options:
  • 1. Encoding them as a dependency that gets installed as the wheels are installed. This is probably the best option.
  • 2. Packaging them directly into the wheels (which is more resilient to if the extras wheels are also purged after a specific time limit) -- I've spent a small amount of time exploring the feasibility of this option: [WIP] Try to include extras in wheels #2786

I was thinking that instead of building our own MLIR wheels, we should use distributions already publicly available, e.g.: https://github.com/llvm/eudsl/tree/main/projects/mlir-wheel However, these wheels expire after 30 days, so we would need to find a way to have them hosted or built on our own on the fly to ensure more consistent long-term dev environment (or also phase out our wheels every 30 days).

@jgmelber and @erwei-xilinx and @andrej please to feel free to weigh in!

[andrej]

Thanks for thinking about this. Providing a smooth install experience is important and if we can make it easier for ourselves and more resilient to do so that would be good.

Encoding them as a dependency that gets installed as the wheels are installed. This is probably the best option.

This would require the special flag for the prefix inside the requirements.txt that only recent pip versions support, is that correct?

[hunhoffe]

So unfortunately it's a little worse than that. The flag works with pip install in recent versions but would need custom parsing in the setup.py for the wheels because the parsing utility used for requirements with wheels does not support the flag (even in recent versions). So if you want to be able to 1. install from wheels, and 2. install from local environment w/ dependencies from a unified requirements file, you must:

• Use the flag in the requirements.txt file so we only have once requirements.txt for local builds, and requirements are automatically installed from wheel.
• Filter the flag out with custom wheel building setup logic and use the environment variable for custom installation instructions of dependencies. The custom dependency parsing unfortunately adds fragility to the wheel build, but not too much. This is why I think it's the best option. Option two listed about is to include the files directly during wheel building, which in some ways in simpler to implement and more stable, but it also adds content to the wheel. I'd rather keep the wheel lean.

Some additional options towards refining the llvm bumping from a meeting today. The goals are currently to:

1. Run monthly, not daily
2. Fetch version to bump to based on ecosystem and not llvm itself or allow a user to trigger manually with a custom commit hash
3. Automatically assign failing bumps to folks based on a rotating schedule in order to share work sensibly