Error in pyarrow while installing Apache Beam in Raspberry Pi

Question

I'm using a Raspberry Pi 3 B+ and should be installing the Apache Beam SDK to connect it to Google Cloud Platform services such as Pub/Sub, Dataflow, and BigQuery. I've got Raspbian GNU/Linux 10 (buster) installed as my OS. I've been following the instructions very carefully on a community tutorial in GCP:

https://cloud.google.com/community/tutorials/ardu-pi-serial-part-2

Ran the following commands:

git clone https://github.com/GoogleCloudPlatform/community.git
cd community/tutorials/ardu-pi-serial-part-2
virtualenv venv
source venv/bin/activate
pip install -r beam-requirements.txt
pip install apache-beam[gcp]

Then I get the error while installing apache-beam:

creating build/temp.linux-armv7l-2.7
-- Running cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/home/pi/community/tutorials/ardu-pi-serial-part-2/venv/bin/python2  - 
DPYARROW_BOOST_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /tmp/pip-install-L4MeVK/pyarrow
unable to execute 'cmake': No such file or directory
error: command 'cmake' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for pyarrow
Building wheel for googledatastore (setup.py) ... done
Created wheel for googledatastore: filename=googledatastore-7.0.2-py2-none-any.whl size=18154 
sha256=d819df28a4cf473e25913fd34bb56506651a3c81b26d66fc2b6b3afede341f8d
Stored in directory: 
/home/pi/.cache/pip/wheels/a6/07/b7/048bd604cfadf321bc995c1d2b6e92770bf852e2209b9dcc97
Building wheel for proto-google-cloud-datastore-v1 (setup.py) ... done
Created wheel for proto-google-cloud-datastore-v1: filename=proto_google_cloud_datastore_v1-0.90.4- 
py2-none-any.whl size=23752 sha256=70834e2d3c64fdfbf04a6326b27b0ec85d01ffd408f15df7b0cefa055265c9b5
Stored in directory: 
/home/pi/.cache/pip/wheels/80/86/e7/3e30f012839d7608dc34c5bb5087356307a028c7e1c44c3075
Building wheel for grpc-google-iam-v1 (setup.py) ... done
Created wheel for grpc-google-iam-v1: filename=grpc_google_iam_v1-0.12.3-py2-none-any.whl size=18500 
sha256=0c5278ed422f0a616fdc3b54de4a750348441ce9f945cb0a1eaa2d5e9b7d94f3
Stored in directory: 
/home/pi/.cache/pip/wheels/77/4d/90/443c1cecdcfbf9d97f6e567304aacddc234cf1b26e48c6f0e5
Building wheel for googleapis-common-protos (setup.py) ... done
Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.51.0-py2-none-any.whl 
size=77593 sha256=910992bfc3cf5b58189e21114ccc2c7bca83a91bedd02cdb06d8a2e783b97007
Stored in directory: 
/home/pi/.cache/pip/wheels/56/af/44/f0c28e985bc224ffb90612f7cdeef432ba4fbd5d15485ab271
Successfully built apache-beam crcmod dill fastavro future hdfs httplib2 pymongo oauth2client avro 
pyvcf googledatastore proto-google-cloud-datastore-v1 grpc-google-iam-v1 googleapis-common-protos
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

What could be the reason this is happening and how would I fix this?

Answer 1

There is an excellent answer for this at streamlit. It worked for me. Another forum member claimed it worked for them.

It is a substantial build: disk space to build: ~ 5.6 GB for llvm, ~0.6 GB for arrow disk space of the install: ~ 0.7 GB

Because I had installed some of the Python packages previously (Cython, most specifically) as the pi user, but not with sudo, I had to re-install those packages using sudo for the last step of pyarrow installation to work:

sudo python setup.py install

Note that you don't need to install streamlit if all you want is pyarrow. But that was the motivation for pyarrow in that environment.

I've copied the majority of the post from @shadanan here (verbatim):

Here are the steps I used:

Prepare your LD_LIBRARY_PATH. You’ll want to put this in .bashrc or .zshrc.

$ export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"

Install llvm-10 from source 1 – unfortunately, arrow requires llvm-10 at a minimum and the latest llvm in the raspbian package repos was llvm-9. If you can get llvm-10 in some other way, do it that way instead of this. It took a whole night to build llvm-10 on my poor Raspberry Pi.

$ git clone https://github.com/llvm/llvm-project.git
$ cd llvm-project
$ git fetch --tags
$ git checkout llvmorg-10.0.1
$ mkdir build
$ cd build
$ cmake -G "Unix Makefiles" \
    -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi" \
    -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=On \
    ../llvm
$ make -j4
$ sudo make install 

Install Apache Arrow from source. I adapted instructions from here: https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 1

$ wget https://github.com/apache/arrow/archive/apache-arrow-1.0.1.tar.gz
$ tar xzf apache-arrow-1.0.1.tar.gz
$ cd arrow-apache-arrow-1.0.1/cpp
$ mkdir build
$ cd build
$ export ARROW_HOME=/usr/local
$ cmake \
    -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
    -DCMAKE_INSTALL_LIBDIR=lib \
    -DARROW_WITH_BZ2=ON \
    -DARROW_WITH_ZLIB=ON \
    -DARROW_WITH_ZSTD=ON \
    -DARROW_WITH_LZ4=ON \
    -DARROW_WITH_SNAPPY=ON \
    -DARROW_PARQUET=ON \
    -DARROW_PYTHON=ON \
    -DARROW_BUILD_TESTS=OFF \
    ..
$ make -j4
$ sudo make install
$ cd ../../python
$ python3 setup.py build_ext --build-type=release --with-parquet
$ sudo python3 setup.py install 

Install streamlit.

$ sudo pip3 install streamlit

Run streamlit as follows. You can also put the LD_PRELOAD in your .bashrc or .zshrc. I think there’s a way to avoid having this 2, but it looked like I’d have to rebuild either llvm-10 or arrow, and I didn’t want to spend any more time on this.

$ LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 streamlit run <python_script.py>

If someone else goes through this and can simplify these steps, I’m sure the community would appreciate it.

Answer 2

if it really is cmake that it can't find:

sudo apt-get install cmake

But at that point it will probably complain that it can't find arrow (the C++ libraries).

in which case, see: https://github.com/apache/arrow/blob/master/docs/source/developers/cpp/building.rst

with the key parts (given Raspberry Pi):

sudo apt-get install build-essential cmake
git clone https://github.com/apache/arrow.git
cd arrow/cpp
mkdir release
cd release
cmake ..
make
make install

you may run out of swap in which case see: How to install latest Scipy version on Raspberry Pi but once arrow C++ is built and installed you can try

pip install pyarrow

I can't get pyarrow to install, it's looking for

-- Found the Arrow core static library: /usr/local/lib/libarrow.a -- Could NOT find ArrowPython (missing: ArrowPython_DIR) -- Checking for module 'arrow-python' -- No package 'arrow-python' found CMake Error at /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message): Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR ARROW_PYTHON_LIB_DIR) (found version "1.0.1")

, but hope this gets you closer...