Description
Hello! Thank you so much for putting up this beautiful work!
After training on the args.txt configuration, I proceeded to evaluate as per the instructions. I obtained the following error. Do you know what is going on?
2020-06-26 00:53:56.788567: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-06-26 00:53:56.795400: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599845000 Hz
2020-06-26 00:53:56.795894: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xecc6dc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-26 00:53:56.795930: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-06-26 00:53:56.801185: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-06-26 00:53:56.923729: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xecf9b60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-26 00:53:56.923796: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P40, Compute Capability 6.1
2020-06-26 00:53:56.925923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:84:00.0
2020-06-26 00:53:56.927409: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.928514: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.929673: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.930791: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.932004: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.933170: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.933611: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /share/apps/python3/3.7.3/intel/lib:/share/apps/gcc/6.3.0/lib64:/share/apps/gcc/6.3.0/lib:/share/apps/mpc/1.0.3/gnu/lib:/share/apps/mpfr/3.1.5/gnu/lib:/share/apps/gmp/6.1.2/gnu/lib:/share/apps/intel/19.0.1/mkl/lib/intel64:/share/apps/intel/19.0.1/lib/intel64:/share/apps/centos/7/usr/lib64:/opt/slurm/lib64
2020-06-26 00:53:56.933650: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-06-26 00:53:56.933691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-26 00:53:56.933715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-06-26 00:53:56.933736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
�[1mPreprocess data...�[0m
�[1mLoading data...�[0m
took 76.19 seconds
�[1mLoading word vectors...�[0m
0
{'': 0, '': 1, '': 2, '': 3, 'are': 4, 'there': 5, 'more': 6, 'big': 7, 'green': 8, 'things': 9, 'than': 10, 'large': 11, 'purple': 12, 'shiny': 13, 'cubes': 14, 'how': 15, 'many': 16, 'other': 17, 'of': 18, 'the': 19, 'same': 20, 'shape': 21, 'as': 22, 'tiny': 23, 'cyan': 24, 'matte': 25, 'object': 26, 'is': 27, 'color': 28, 'sphere': 29, 'cube': 30, 'what': 31, 'material': 32, 'that': 33, 'right': 34, 'brown': 35, 'cylinder': 36, 'and': 37, 'left': 38, 'gray': 39, 'on': 40, 'side': 41, 'small': 42, 'rubber': 43, 'behind': 44, 'thing': 45, 'to': 46, 'metallic': 47, 'size': 48, 'any': 49, 'have': 50, 'block': 51, 'blue': 52, 'yellow': 53, 'a': 54, ';': 55, 'it': 56, 'ball': 57, 'its': 58, 'in': 59, 'front': 60, 'does': 61, 'number': 62, 'red': 63, 'spheres': 64, 'made': 65, 'metal': 66, 'cylinders': 67, 'both': 68, 'balls': 69, 'or': 70, 'blocks': 71, 'objects': 72, 'visible': 73, 'another': 74, 'has': 75, 'greater': 76, 'fewer': 77, 'less': 78, 'either': 79, 'anything': 80, 'else': 81, 'do': 82, 'an': 83, 'equal': 84}
85
{'yes': 0, '2': 1, 'no': 2, 'rubber': 3, 'large': 4, '0': 5, 'sphere': 6, 'gray': 7, 'cube': 8, 'blue': 9, 'brown': 10, '1': 11, 'yellow': 12, 'purple': 13, 'cylinder': 14, 'small': 15, 'green': 16, 'metal': 17, '3': 18, '4': 19, 'cyan': 20, '6': 21, 'red': 22, '5': 23, '8': 24, '7': 25, '9': 26, '10': 27}
28
{'': 0, '': 1, '': 2, '': 3, 'are': 4, 'there': 5, 'more': 6, 'big': 7, 'green': 8, 'things': 9, 'than': 10, 'large': 11, 'purple': 12, 'shiny': 13, 'cubes': 14, 'yes': 15, 'how': 16, 'many': 17, 'other': 18, 'of': 19, 'the': 20, 'same': 21, 'shape': 22, 'as': 23, 'tiny': 24, 'cyan': 25, 'matte': 26, 'object': 27, '2': 28, 'is': 29, 'color': 30, 'sphere': 31, 'cube': 32, 'no': 33, 'what': 34, 'material': 35, 'that': 36, 'right': 37, 'brown': 38, 'cylinder': 39, 'and': 40, 'left': 41, 'rubber': 42, 'gray': 43, 'on': 44, 'side': 45, 'small': 46, 'behind': 47, 'thing': 48, '0': 49, 'to': 50, 'metallic': 51, 'size': 52, 'any': 53, 'have': 54, 'block': 55, 'blue': 56, 'yellow': 57, 'a': 58, ';': 59, 'it': 60, 'ball': 61, 'its': 62, 'in': 63, 'front': 64, 'does': 65, 'number': 66, 'red': 67, 'spheres': 68, 'made': 69, 'metal': 70, 'cylinders': 71, '1': 72, 'both': 73, 'balls': 74, 'or': 75, 'blocks': 76, 'objects': 77, 'visible': 78, 'another': 79, 'has': 80, 'greater': 81, 'fewer': 82, 'less': 83, '3': 84, '4': 85, 'either': 86, 'anything': 87, 'else': 88, 'do': 89, '6': 90, 'an': 91, 'equal': 92, '5': 93, '8': 94, '7': 95, '9': 96, '10': 97}
98
took 0.00 seconds
�[1mVectorizing data...�[0m
took 13.70 seconds
took �[1m�[34m89.90�[0m seconds
�[1mBuilding model...�[0m
took �[1m�[34m17.59�[0m seconds
Traceback (most recent call last):
File "main.py", line 802, in
main()
File "main.py", line 691, in main
epoch = loadWeights(sess, saver, init)
File "main.py", line 190, in loadWeights
config.restoreEpoch, config.lr = lastLoggedEpoch()
File "main.py", line 62, in lastLoggedEpoch
epoch = int(lastLine[0])
ValueError: invalid literal for int() with base 10: 'epoch'
Thank you very much in advance for the help! :) Take care :)