Windows Build: bazel errror

Deep Speech
#1

Am trying to build libdeepspeech.so on Windows from source. Folling the description on github, I installed MSYS2 and Bazel, and did run configure.py to use CUDA.

When running the command :

bazel build --workspace_status_command=“bash native_client/bazel_workspace_status_cmd.sh” -c opt --config=cuda --copt=/arch:AVX --copt=/arch:AVX2 //native_client:libdeepspeech.so

I always the the error:

ERROR: D:/projects/deepspeech/tensorflow/native_client/BUILD:3:1: file ‘@org_tensorflow//tensorflow:tensorflow.bzl’ does not contain symbol ‘if_cuda’
ERROR: D:/projects/deepspeech/tensorflow/native_client/BUILD:148:10: Traceback (most recent call last):
File “D:/projects/deepspeech/tensorflow/native_client/BUILD”, line 61
tf_cc_shared_object(name = “libdeepspeech.so”, srcs = …"]})), <5 more arguments>)
File “D:/projects/deepspeech/tensorflow/native_client/BUILD”, line 148, in tf_cc_shared_object
if_cuda
name ‘if_cuda’ is not defined
ERROR: Skipping ‘//native_client:libdeepspeech.so’: no such target ‘//native_client:libdeepspeech.so’: target ‘libdeepspeech.so’ not declared in package ‘native_client’ defined by D:/projects/deepspeech/tensorflow/native_client/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target ‘//native_client:libdeepspeech.so’: target ‘libdeepspeech.so’ not declared in package ‘native_client’ defined by D:/projects/deepspeech/tensorflow/native_client/BUILD
INFO: Elapsed time: 0.147s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded)

Unfortunately I don’t know Bazel, and my simplistic attempts of adding something like

def if_cuda(a, b[])
return a

did of course not solve anything.

Any help how to resolve this issue would be greatly appreciated.

#2

Hello,

Looks like your symbolic link is wrong

#3

First, I thought that as well. But then I copied the native_client folder to the tensorflow folder (as was suggested elsewhere). However, that did not change the problem.

So, I was focusing on the if_cuda error, but was not able to solve that either.

#4

If you shared your configure steps it would help … Your error obviously shows it’s not configured.

#5

According to the steps in the readme file README.MD I simply run

python configure.py

in the tensorflow folder.

Whether I accept all defaults, or elect to use CUDA makes no difference to the bazel error. And configure.py completes without any errors or warnings.

#6

I don’t know where you got that suggestion, but you need to symlink, not copy.

Care to share more complete log and steps ? Maybe there’s a mistake you did somewhere else …

#7

Don’t know whether this helps, but this is the .tf_configure.bazelrc file that is created by configure.py:

build --action_env PYTHON_BIN_PATH=“C:/Program Files/Python37/python.exe”
build --action_env PYTHON_LIB_PATH=“C:/Program Files/Python37/lib/site-packages”
build --python_path=“C:/Program Files/Python37/python.exe”
build:xla --define with_xla_support=true
build --action_env TF_NEED_OPENCL_SYCL=“0”
build --action_env TF_NEED_ROCM=“0”
build --action_env TF_NEED_CUDA=“1”
build --action_env TF_NEED_TENSORRT=“0”
build --action_env CUDA_TOOLKIT_PATH=“C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0”
build --action_env TF_CUDA_COMPUTE_CAPABILITIES=“3.5,7.0”
build --action_env TF_CUDA_CLANG=“0”
build --config=cuda
build:opt --copt=/arch:AVX
build:opt --define with_default_optimizations=true
build --config monolithic
build --copt=-w --host_copt=-w
build --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI
build --verbose_failures
build --distinct_host_configuration=false
build --define=override_eigen_strong_inline=true
build:v2 --define=tf_api_version=2
test --flaky_test_attempts=3
test --test_size_filters=small,medium
test --test_tag_filters=-benchmark-test,-no_oss,-oss_serial
test --build_tag_filters=-benchmark-test,-no_oss
test --test_tag_filters=-no_windows,-gpu
test --build_tag_filters=-no_windows,-gpu
build --action_env TF_CONFIGURE_IOS=“0”

If there is anything else that would be useful, please let me know and I’m happy to post.

Btw, I also replaced the native_client directory with a symbolic link. Did not change anything in the error.

#8

Your original post had two errors, you still have the two ?

It is surprising with this if you get the if_cuda error. Can you ensure you did the proper checkout ?

#9

@pvk444 As you can see, if_cuda comes from the local_config_cuda component, so that’s why it’s important we get proper context of your configure steps, since this is what is going to create this https://github.com/mozilla/tensorflow/blob/r1.14/tensorflow/tensorflow.bzl#L24-L28

#10

I deleted DeepSpeech and tensorflow completely, and cloned again from github as described in the README.MD

After running configure.py again, I still get exactly the same errors. Here is what I used for the configuration:

python configure.py
WARNING: Running Bazel server needs to be killed, because the startup options are different.
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command “bazel shutdown”.
You have bazel 0.25.2 installed.
Please specify the location of python. [Default is C:\Program Files\Python37\python.exe]:

Found possible Python library paths:
C:\Program Files\Python37\lib\site-packages
Please input the desired Python library path to use. Default is [C:\Program Files\Python37\lib\site-packages]

Do you wish to build TensorFlow with XLA JIT support? [y/N]:
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with ROCm support? [y/N]:
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Found CUDA 10.0 in:
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/lib/x64
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/include
Found cuDNN 7 in:
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/lib/x64
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/include

Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 3.5,7.0]:

Please specify optimization flags to use during compilation when bazel option “–config=opt” is specified [Default is /arch:AVX]:

Would you like to override eigen strong inline for some C++ compilation to reduce the compilation time? [Y/n]:
Eigen strong inline overridden.

Preconfigured Bazel build configs. You can use any of the below by adding “–config=<>” to your build command. See .bazelrc for more details.
–config=mkl # Build with MKL support.
–config=monolithic # Config for mostly static monolithic build.
–config=gdr # Build with GDR support.
–config=verbs # Build with libverbs support.
–config=ngraph # Build with Intel nGraph support.
–config=numa # Build with NUMA support.
–config=dynamic_kernels # (Experimental) Build kernels into separate shared objects.
Preconfigured Bazel build configs to DISABLE default on features:
–config=noaws # Disable AWS S3 filesystem support.
–config=nogcp # Disable GCP support.
–config=nohdfs # Disable HDFS support.
–config=noignite # Disable Apache Ignite support.
–config=nokafka # Disable Apache Kafka support.
–config=nonccl # Disable NVIDIA NCCL support.

And not sure whether this has any relevance or even whether I read it correctly but given

load(
“@local_config_cuda//cuda:build_defs.bzl”,
“cuda_default_copts”,
“if_cuda”,
)

I was trying to find build_defs.bzl in a “cuda” folder. I found a entire bunch of build_defs.bzl files but none in a “cuda” folder. The closest was a build_defs.bzl.tpl file in tensorflow\third_party\gpus\cuda\ or then cuda_build_defs.bzl in tensorflow\tensorflow\core\platform\default.

#11

Sorry to insist, but have you cloned our tensorflow repository and used branch r1.14 ?

Upstream recommends 0.24.1, could you stick to this ?

It should be somewhere under some bazel-*/external, like for me bazel-tensorflow/external/local_config_cuda/

But at some point it does not seems to be a DeepSpeech issue … I don’t have much experience with building with CUDA on Windows.

#12

BTW, is there a specific reason you want to do this ? What’s wrong with our prebuilt binaries ?

#13

At a later stage, I was intending to see whether I can port it to support a Neuro Stick. For now, I just wanted to test the waters, so to speak

#14

Ok. That seems to have moved things forward. Btw, the readme for the windows build did not include a required version number for bazel. When running the building process originally, it said the I needed to use a lower version, so I selected the one that was suggested. Was obvioulsy still too high.

However, now I run into the following problem:

WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
ERROR: D:/projects/deepspeech/tensorflow/native_client/BUILD:68:1: no such target ‘//tensorflow:darwin’: target ‘darwin’ not declared in package ‘tensorflow’ defined by D:/projects/deepspeech/tensorflow/tensorflow/BUILD and referenced by ‘//native_client:libdeepspeech.so’
ERROR: Analysis of target ‘//native_client:libdeepspeech.so’ failed; build aborted: no such target ‘//tensorflow:darwin’: target ‘darwin’ not declared in package ‘tensorflow’ defined by D:/projects/deepspeech/tensorflow/tensorflow/BUILD
INFO: Elapsed time: 2.788s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (5 packages loaded, 4 targets configured)

No idea why it would want to select darwin … I’m on Windows 10.

#15

Darwin is now macos : https://github.com/mozilla/DeepSpeech/commit/d52ab9df0b5cc76aeca866def524d228a95ce4f0#diff-70256643755d3d1cdb540a032753f47e

Please check again that you are working with the most recent version of DS and TF r 1.14
If you are working with the correct versions please try running

bazel clean

#16

Obvious symptom that you don’t have r1.14

#17

Almost there … the initial

bazel clean

didn’t do anything. But then I deleted once again DS and tensorflow, restarted and pulled the packages as per README.MD.

And ta-ta … it started to compile. However, after a few minutes I got now the following error:

WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Analysed target //native_client:libdeepspeech.so (83 packages loaded, 4153 targets configured).
INFO: Found 1 target…
ERROR: Process exited with status 127: Process exited with status 127
/bin/bash: native_clientbazel_workspace_status_cmd.sh: No such file or directory
Target //native_client:libdeepspeech.so failed to build
INFO: Elapsed time: 48.977s, Critical Path: 1.09s
INFO: 7 processes: 7 local.
FAILED: Build did NOT complete successfully

After fiddling around with the system and trying all sorts of “//” and “” type combinations, it turns out that despite having MSYS2 on the path, the bash command which was invoked was from the ubuntu system (which is installed as part of the windows - linux support). So I moved the MSYS2 path to the front of the system path, and voila, it’s compiling perfectly.

Thanks carlfm01 and lissyx for your help. Would have been completely lost without your support!

Bazel build error
#18

Obvious, but wrong. For this exercise, I ALWAYS used

git clone --branch r1.14 https://github.com/mozilla/tensorflow

as described in the Windows Build README.MD. So something else must have gone wrong.

#19

Err, I should not reply when too tired. tensorflow:darwin is a symptom of wrong DeepSpeech checkout, since wtih r1.14 this has moved to tensorflow:macos

#20

Try that

--- a/native_client/BUILD
+++ b/native_client/BUILD
@@ -1,8 +1,10 @@
 # Description: Deepspeech native client library.

+load("@local_config_cuda//cuda:build_defs.bzl", "if_cuda")
+
 load(
     "@org_tensorflow//tensorflow:tensorflow.bzl",
-    "if_cuda",
+    #"if_cuda",
     "tf_cc_shared_object",
 )
 load(