Dagster on Arch Linux (part the 1st): Setting Up

Packages you’ll need: #

pacman -S make patch python3 extras/postgresql nodejs-lts-gallium

Note: yarn will have issues
on node v17. Using nvm here is probably a good idea.

You’ll need to install pip with ensurepip, as python3 on arch doesn’t come with pip

python3 -m ensurepip --upgrade

Install pyenv:

sudo pacman -S pyenv

Run this to set up the shell. Icky, but convenient.

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zprofile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zprofile
echo 'eval "$(pyenv init --path)"' >> ~/.zprofile

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.profile
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.profile
echo 'eval "$(pyenv init --path)"' >> ~/.profile

echo 'eval "$(pyenv init -)"' >> ~/.zshrc

You’ll need to run . ~/.profile, or you can log out and back in to reload
the login shell config.

Install pyenv-virtualenv

git clone https://github.com/pyenv/pyenv-virtualenv.git \
$(pyenv root)/plugins/pyenv-virtualenv

Now you should be able to run:

which python
#=> /home/manny/.pyenv/shims/python

If this is pointing to .penv/shims then pyenv is working as expected.

Following the dagster docs :

pyenv install 3.7.4
pyenv virtualenv 3.7.4 dagster37
pyenv activate dagster37

Cloning the Repo and Setting up #

git clone git@github.com:dagster-io/dagster.git && \
    cd dagster

Now you can run make install_dev

You’ll be waiting a while for:

Building wheel for pandas (PEP 517): started
Building wheel for pandas (PEP 517): still running...
Building wheel for pandas (PEP 517): still running...
Building wheel for pandas (PEP 517): still running...
Building wheel for pandas (PEP 517): still running...
Building wheel for pandas (PEP 517): still running...

If this is successful, you should be able to run tests and start hacking!

python -m pytest python_modules/dagster/dagster_tests

I have 8 cores so I ask pytest-xdist to use them. The Dagster folks have it
installed as part of the setup.

Add -n auto to the invocation:

python -m pytest -n auto python_modules/dagster/dagster_tests

I’ve got test failures, but that’s for the next post!