Open
Description
Trying out automl_gs in a new conda env using the titanic dataset. After each iteration I get the error:
ValueError: Parent directory of model_weights.hdf5 doesn't exist, can't save.
Same behavior running from command line or within ipython following the example notebook. To clarify, it's finding titanic.csv fine, the error seems to be when saving the intermediate results. Full traceback available below.
Traceback
$ automl_gs titanic.csv Survived
/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/automl_gs/utils_automl.py:270: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
metrics = yaml.load(f)
Solving a binary_classification problem, maximizing accuracy using tensorflow.
Modeling with field specifications:
Pclass: categorical
Name: ignore
Sex: categorical
Age: numeric
Siblings/Spouses Aboard: categorical
Parents/Children Aboard: categorical
Fare: numeric
/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/automl_gs/utils_automl.py:126: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
hps = yaml.load(f)
0%| | 0/100 [00:00<?, ?trial/s/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/automl_gs/utils_automl.py:199: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
metrics = yaml.load(f)[problem_type]
Traceback (most recent call last):████████████████████████████████████ | 16/20 [00:06<00:01, 2.35epoch/s]
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)██████████████████████████████████████████████████████████████████| 20/20 [00:07<00:00, 2.83epoch/s]
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: ; No such file or directory
[[Node: save/SaveV2 = SaveV2[dtypes=[DT_STRING, DT_STRING, DT_STRING, DT_STRING, DT_STRING, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_22, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, _arg_Const_1_0_10, _arg_Const_22_0_13, _arg_Const_2_0_14, _arg_Const_3_0_15, _arg_Const_14_0_4, _arg_Const_17_0_7, _arg_Const_20_0_11, _arg_Const_4_0_16, _arg_Const_5_0_17, _arg_Const_6_0_18, _arg_Const_7_0_19, _arg_Const_8_0_20, _arg_Const_11_0_1, _arg_Const_9_0_21, hidden_1/bias/Read/ReadVariableOp, hidden_1/bias/AdamW/Read/ReadVariableOp, hidden_1/bias/AdamW_1/Read/ReadVariableOp, hidden_1/kernel/Read/ReadVariableOp, hidden_1/kernel/AdamW/Read/ReadVariableOp, hidden_1/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_10_0_0, bn_1/beta/Read/ReadVariableOp, bn_1/beta/AdamW/Read/ReadVariableOp, bn_1/beta/AdamW_1/Read/ReadVariableOp, bn_1/gamma/Read/ReadVariableOp, bn_1/gamma/AdamW/Read/ReadVariableOp, bn_1/gamma/AdamW_1/Read/ReadVariableOp, bn_1/moving_mean/Read/ReadVariableOp, bn_1/moving_variance/Read/ReadVariableOp, _arg_Const_12_0_2, hidden_2/bias/Read/ReadVariableOp, hidden_2/bias/AdamW/Read/ReadVariableOp, hidden_2/bias/AdamW_1/Read/ReadVariableOp, hidden_2/kernel/Read/ReadVariableOp, hidden_2/kernel/AdamW/Read/ReadVariableOp, hidden_2/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_13_0_3, bn_2/beta/Read/ReadVariableOp, bn_2/beta/AdamW/Read/ReadVariableOp, bn_2/beta/AdamW_1/Read/ReadVariableOp, bn_2/gamma/Read/ReadVariableOp, bn_2/gamma/AdamW/Read/ReadVariableOp, bn_2/gamma/AdamW_1/Read/ReadVariableOp, bn_2/moving_mean/Read/ReadVariableOp, bn_2/moving_variance/Read/ReadVariableOp, _arg_Const_15_0_5, hidden_3/bias/Read/ReadVariableOp, hidden_3/bias/AdamW/Read/ReadVariableOp, hidden_3/bias/AdamW_1/Read/ReadVariableOp, hidden_3/kernel/Read/ReadVariableOp, hidden_3/kernel/AdamW/Read/ReadVariableOp, hidden_3/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_16_0_6, bn_3/beta/Read/ReadVariableOp, bn_3/beta/AdamW/Read/ReadVariableOp, bn_3/beta/AdamW_1/Read/ReadVariableOp, bn_3/gamma/Read/ReadVariableOp, bn_3/gamma/AdamW/Read/ReadVariableOp, bn_3/gamma/AdamW_1/Read/ReadVariableOp, bn_3/moving_mean/Read/ReadVariableOp, bn_3/moving_variance/Read/ReadVariableOp, _arg_Const_18_0_8, hidden_4/bias/Read/ReadVariableOp, hidden_4/bias/AdamW/Read/ReadVariableOp, hidden_4/bias/AdamW_1/Read/ReadVariableOp, hidden_4/kernel/Read/ReadVariableOp, hidden_4/kernel/AdamW/Read/ReadVariableOp, hidden_4/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_19_0_9, bn_4/beta/Read/ReadVariableOp, bn_4/beta/AdamW/Read/ReadVariableOp, bn_4/beta/AdamW_1/Read/ReadVariableOp, bn_4/gamma/Read/ReadVariableOp, bn_4/gamma/AdamW/Read/ReadVariableOp, bn_4/gamma/AdamW_1/Read/ReadVariableOp, bn_4/moving_mean/Read/ReadVariableOp, bn_4/moving_variance/Read/ReadVariableOp, _arg_Const_21_0_12, output/bias/Read/ReadVariableOp, output/bias/AdamW/Read/ReadVariableOp, output/bias/AdamW_1/Read/ReadVariableOp, output/kernel/Read/ReadVariableOp, output/kernel/AdamW/Read/ReadVariableOp, output/kernel/AdamW_1/Read/ReadVariableOp, training/TFOptimizer/beta1_power, training/TFOptimizer/beta2_power)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1620, in save
{self.saver_def.filename_tensor_name: checkpoint_file})
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/checkpointable/util.py", line 1047, in run
fetches=fetches, feed_dict=feed_dict, **kwargs)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: ; No such file or directory
[[Node: save/SaveV2 = SaveV2[dtypes=[DT_STRING, DT_STRING, DT_STRING, DT_STRING, DT_STRING, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_22, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, _arg_Const_1_0_10, _arg_Const_22_0_13, _arg_Const_2_0_14, _arg_Const_3_0_15, _arg_Const_14_0_4, _arg_Const_17_0_7, _arg_Const_20_0_11, _arg_Const_4_0_16, _arg_Const_5_0_17, _arg_Const_6_0_18, _arg_Const_7_0_19, _arg_Const_8_0_20, _arg_Const_11_0_1, _arg_Const_9_0_21, hidden_1/bias/Read/ReadVariableOp, hidden_1/bias/AdamW/Read/ReadVariableOp, hidden_1/bias/AdamW_1/Read/ReadVariableOp, hidden_1/kernel/Read/ReadVariableOp, hidden_1/kernel/AdamW/Read/ReadVariableOp, hidden_1/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_10_0_0, bn_1/beta/Read/ReadVariableOp, bn_1/beta/AdamW/Read/ReadVariableOp, bn_1/beta/AdamW_1/Read/ReadVariableOp, bn_1/gamma/Read/ReadVariableOp, bn_1/gamma/AdamW/Read/ReadVariableOp, bn_1/gamma/AdamW_1/Read/ReadVariableOp, bn_1/moving_mean/Read/ReadVariableOp, bn_1/moving_variance/Read/ReadVariableOp, _arg_Const_12_0_2, hidden_2/bias/Read/ReadVariableOp, hidden_2/bias/AdamW/Read/ReadVariableOp, hidden_2/bias/AdamW_1/Read/ReadVariableOp, hidden_2/kernel/Read/ReadVariableOp, hidden_2/kernel/AdamW/Read/ReadVariableOp, hidden_2/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_13_0_3, bn_2/beta/Read/ReadVariableOp, bn_2/beta/AdamW/Read/ReadVariableOp, bn_2/beta/AdamW_1/Read/ReadVariableOp, bn_2/gamma/Read/ReadVariableOp, bn_2/gamma/AdamW/Read/ReadVariableOp, bn_2/gamma/AdamW_1/Read/ReadVariableOp, bn_2/moving_mean/Read/ReadVariableOp, bn_2/moving_variance/Read/ReadVariableOp, _arg_Const_15_0_5, hidden_3/bias/Read/ReadVariableOp, hidden_3/bias/AdamW/Read/ReadVariableOp, hidden_3/bias/AdamW_1/Read/ReadVariableOp, hidden_3/kernel/Read/ReadVariableOp, hidden_3/kernel/AdamW/Read/ReadVariableOp, hidden_3/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_16_0_6, bn_3/beta/Read/ReadVariableOp, bn_3/beta/AdamW/Read/ReadVariableOp, bn_3/beta/AdamW_1/Read/ReadVariableOp, bn_3/gamma/Read/ReadVariableOp, bn_3/gamma/AdamW/Read/ReadVariableOp, bn_3/gamma/AdamW_1/Read/ReadVariableOp, bn_3/moving_mean/Read/ReadVariableOp, bn_3/moving_variance/Read/ReadVariableOp, _arg_Const_18_0_8, hidden_4/bias/Read/ReadVariableOp, hidden_4/bias/AdamW/Read/ReadVariableOp, hidden_4/bias/AdamW_1/Read/ReadVariableOp, hidden_4/kernel/Read/ReadVariableOp, hidden_4/kernel/AdamW/Read/ReadVariableOp, hidden_4/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_19_0_9, bn_4/beta/Read/ReadVariableOp, bn_4/beta/AdamW/Read/ReadVariableOp, bn_4/beta/AdamW_1/Read/ReadVariableOp, bn_4/gamma/Read/ReadVariableOp, bn_4/gamma/AdamW/Read/ReadVariableOp, bn_4/gamma/AdamW_1/Read/ReadVariableOp, bn_4/moving_mean/Read/ReadVariableOp, bn_4/moving_variance/Read/ReadVariableOp, _arg_Const_21_0_12, output/bias/Read/ReadVariableOp, output/bias/AdamW/Read/ReadVariableOp, output/bias/AdamW_1/Read/ReadVariableOp, output/kernel/Read/ReadVariableOp, output/kernel/AdamW/Read/ReadVariableOp, output/kernel/AdamW_1/Read/ReadVariableOp, training/TFOptimizer/beta1_power, training/TFOptimizer/beta2_power)]]
Caused by op 'save/SaveV2', defined at:
File "model.py", line 46, in <module>
model_train(df, encoders, args, model)
File "/Volumes/Backstaff/scratch/automl/automl_train/pipeline.py", line 377, in model_train
batch_size=256)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1363, in fit
validation_steps=validation_steps)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 291, in fit_loop
callbacks.on_train_end()
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/callbacks.py", line 158, in on_train_end
callback.on_train_end(logs)
File "/Volumes/Backstaff/scratch/automl/automl_train/pipeline.py", line 398, in on_train_end
self.model.save_weights('model_weights.hdf5')
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1365, in save_weights
self._checkpointable_saver.save(filepath, session=session)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/checkpointable/util.py", line 1178, in save
self._last_save_saver = saver_lib.Saver(var_list=named_variables)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1281, in __init__
self.build()
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1293, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1330, in _build
build_save=build_save, build_restore=build_restore)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 775, in _build_internal
save_tensor = self._AddSaveOps(filename_tensor, saveables)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 275, in _AddSaveOps
save = self.save_op(filename_tensor, saveables)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 193, in save_op
tensors)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1687, in save_v2
shape_and_slices=shape_and_slices, tensors=tensors, name=name)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()
NotFoundError (see above for traceback): ; No such file or directory
[[Node: save/SaveV2 = SaveV2[dtypes=[DT_STRING, DT_STRING, DT_STRING, DT_STRING, DT_STRING, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_22, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, _arg_Const_1_0_10, _arg_Const_22_0_13, _arg_Const_2_0_14, _arg_Const_3_0_15, _arg_Const_14_0_4, _arg_Const_17_0_7, _arg_Const_20_0_11, _arg_Const_4_0_16, _arg_Const_5_0_17, _arg_Const_6_0_18, _arg_Const_7_0_19, _arg_Const_8_0_20, _arg_Const_11_0_1, _arg_Const_9_0_21, hidden_1/bias/Read/ReadVariableOp, hidden_1/bias/AdamW/Read/ReadVariableOp, hidden_1/bias/AdamW_1/Read/ReadVariableOp, hidden_1/kernel/Read/ReadVariableOp, hidden_1/kernel/AdamW/Read/ReadVariableOp, hidden_1/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_10_0_0, bn_1/beta/Read/ReadVariableOp, bn_1/beta/AdamW/Read/ReadVariableOp, bn_1/beta/AdamW_1/Read/ReadVariableOp, bn_1/gamma/Read/ReadVariableOp, bn_1/gamma/AdamW/Read/ReadVariableOp, bn_1/gamma/AdamW_1/Read/ReadVariableOp, bn_1/moving_mean/Read/ReadVariableOp, bn_1/moving_variance/Read/ReadVariableOp, _arg_Const_12_0_2, hidden_2/bias/Read/ReadVariableOp, hidden_2/bias/AdamW/Read/ReadVariableOp, hidden_2/bias/AdamW_1/Read/ReadVariableOp, hidden_2/kernel/Read/ReadVariableOp, hidden_2/kernel/AdamW/Read/ReadVariableOp, hidden_2/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_13_0_3, bn_2/beta/Read/ReadVariableOp, bn_2/beta/AdamW/Read/ReadVariableOp, bn_2/beta/AdamW_1/Read/ReadVariableOp, bn_2/gamma/Read/ReadVariableOp, bn_2/gamma/AdamW/Read/ReadVariableOp, bn_2/gamma/AdamW_1/Read/ReadVariableOp, bn_2/moving_mean/Read/ReadVariableOp, bn_2/moving_variance/Read/ReadVariableOp, _arg_Const_15_0_5, hidden_3/bias/Read/ReadVariableOp, hidden_3/bias/AdamW/Read/ReadVariableOp, hidden_3/bias/AdamW_1/Read/ReadVariableOp, hidden_3/kernel/Read/ReadVariableOp, hidden_3/kernel/AdamW/Read/ReadVariableOp, hidden_3/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_16_0_6, bn_3/beta/Read/ReadVariableOp, bn_3/beta/AdamW/Read/ReadVariableOp, bn_3/beta/AdamW_1/Read/ReadVariableOp, bn_3/gamma/Read/ReadVariableOp, bn_3/gamma/AdamW/Read/ReadVariableOp, bn_3/gamma/AdamW_1/Read/ReadVariableOp, bn_3/moving_mean/Read/ReadVariableOp, bn_3/moving_variance/Read/ReadVariableOp, _arg_Const_18_0_8, hidden_4/bias/Read/ReadVariableOp, hidden_4/bias/AdamW/Read/ReadVariableOp, hidden_4/bias/AdamW_1/Read/ReadVariableOp, hidden_4/kernel/Read/ReadVariableOp, hidden_4/kernel/AdamW/Read/ReadVariableOp, hidden_4/kernel/AdamW_1/Read/ReadVariableOp, _arg_Const_19_0_9, bn_4/beta/Read/ReadVariableOp, bn_4/beta/AdamW/Read/ReadVariableOp, bn_4/beta/AdamW_1/Read/ReadVariableOp, bn_4/gamma/Read/ReadVariableOp, bn_4/gamma/AdamW/Read/ReadVariableOp, bn_4/gamma/AdamW_1/Read/ReadVariableOp, bn_4/moving_mean/Read/ReadVariableOp, bn_4/moving_variance/Read/ReadVariableOp, _arg_Const_21_0_12, output/bias/Read/ReadVariableOp, output/bias/AdamW/Read/ReadVariableOp, output/bias/AdamW_1/Read/ReadVariableOp, output/kernel/Read/ReadVariableOp, output/kernel/AdamW/Read/ReadVariableOp, output/kernel/AdamW_1/Read/ReadVariableOp, training/TFOptimizer/beta1_power, training/TFOptimizer/beta2_power)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "model.py", line 46, in <module>
model_train(df, encoders, args, model)
File "/Volumes/Backstaff/scratch/automl/automl_train/pipeline.py", line 377, in model_train
batch_size=256)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1363, in fit
validation_steps=validation_steps)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 291, in fit_loop
callbacks.on_train_end()
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/callbacks.py", line 158, in on_train_end
callback.on_train_end(logs)
File "/Volumes/Backstaff/scratch/automl/automl_train/pipeline.py", line 398, in on_train_end
self.model.save_weights('model_weights.hdf5')
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1365, in save_weights
self._checkpointable_saver.save(filepath, session=session)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/checkpointable/util.py", line 1186, in save
global_step=checkpoint_number)
File "/Users/dnowacki/miniconda3/envs/automl/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1637, in save
raise exc
ValueError: Parent directory of model_weights.hdf5 doesn't exist, can't save.
Metrics:
trial_id: 3e5c75e7-53be-4e75-8558-17b511440ba9
epoch: 20
time_completed: 2019-03-26 22:03:19
log_loss: 0.6697867036089022
accuracy: 0.6142322097378277
auc: 0.8345666587733839
precision: 0.30711610486891383
recall: 0.5
f1: 0.3805104408352668
1%|▊ | 1/100 [00:07<12:16, 7.44s/trial]