02/04 19:52:52] pbsdev@pbsdev-centos7-mky18-server:~ 49% time sudo $CV2_PTL_INSTALL_DIR/bin/pbs_benchpress --db-name $CV2_PBS_BUILD_DIR/test/tests/ptl_test_results.json -f $CV2_PTL_INSTALL_DIR/tests/functional/pbs_hook_modifyvnode_state_changes.py -p "moms=mom1:mom2" 2021-02-04 20:11:24,022 INFO input command: pbs_benchpress --db-name /home/pbsdev/pbs/build/test/tests/ptl_test_results.json -f /opt/ptl/tests/functional/pbs_hook_modifyvnode_state_changes.py -p moms=mom1:mom2 2021-02-04 20:11:24,022 INFO param: moms=mom1:mom2 2021-02-04 20:11:24,022 INFO ptl version: 19.1.1.1093.g7f30851.modified 2021-02-04 20:11:24,025 INFO platform: Linux pbsdev-centos7-mky18-server 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 x86_64 2021-02-04 20:11:24,026 INFO python version: 3.6.8 2021-02-04 20:11:24,026 INFO user: root 2021-02-04 20:11:24,026 INFO -------------------------------------------------------------------------------- 2021-02-04 20:11:24,026 INFO Cleaning up temporary files 2021-02-04 20:11:24,027 INFO Cleaning up /var/tmp dir 2021-02-04 20:11:24,027 INFO Cleaning up /tmp dir 2021-02-04 20:11:24,038 INFO ====================================================================== 2021-02-04 20:11:24,038 INFO suite name: TestPbsModifyvnodeStateChanges 2021-02-04 20:11:24,038 INFO suite docstring: Test the modifyvnode hook by inducing various vnode state changes and inspecting the pbs log for expected values. 2021-02-04 20:11:24,038 INFO ====================================================================== 2021-02-04 20:11:24,038 INFO ================================================== 2021-02-04 20:11:24,038 INFO Entered TestPbsModifyvnodeStateChanges setUpClass 2021-02-04 20:11:24,038 INFO ================================================== 2021-02-04 20:11:24,039 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser 2021-02-04 20:11:24,041 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser1 2021-02-04 20:11:24,044 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser2 2021-02-04 20:11:24,047 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser3 2021-02-04 20:11:24,050 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser4 2021-02-04 20:11:24,053 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser5 2021-02-04 20:11:24,056 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser6 2021-02-04 20:11:24,058 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser7 2021-02-04 20:11:24,061 INFOCLI2 pbsdev-centos7-mky18-server: id pbsother 2021-02-04 20:11:24,063 INFOCLI2 pbsdev-centos7-mky18-server: id pbstest 2021-02-04 20:11:24,066 INFOCLI2 pbsdev-centos7-mky18-server: id tstusr00 2021-02-04 20:11:24,068 INFOCLI2 pbsdev-centos7-mky18-server: id tstusr01 2021-02-04 20:11:24,071 INFOCLI2 pbsdev-centos7-mky18-server: id pbsoper 2021-02-04 20:11:24,073 INFOCLI2 pbsdev-centos7-mky18-server: id pbsmgr 2021-02-04 20:11:24,075 INFOCLI2 pbsdev-centos7-mky18-server: id pbsadmin 2021-02-04 20:11:24,078 INFOCLI2 pbsdev-centos7-mky18-server: id pbsdata 2021-02-04 20:11:24,080 INFOCLI2 pbsdev-centos7-mky18-server: id pbsroot 2021-02-04 20:11:24,082 INFOCLI2 pbsdev-centos7-mky18-server: id root 2021-02-04 20:11:24,085 INFOCLI2 pbsdev-centos7-mky18-server: id pbsbuild 2021-02-04 20:11:24,088 INFOCLI2 pbsdev-centos7-mky18-server: id root 2021-02-04 20:11:24,091 INFO FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server 2021-02-04 20:11:24,100 INFO server pbsdev-centos7-mky18-server: server operating mode set to cli 2021-02-04 20:11:24,100 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:24,106 ERROR err: ['Connection refused', 'qstat: cannot connect to server pbsdev- (errno=111)'] 2021-02-04 20:11:24,106 INFO server pbsdev-centos7-mky18-server: version unknown 2021-02-04 20:11:24,107 INFO expect action: created new action kicksched 2021-02-04 20:11:24,107 INFO expect action: added action kicksched to server pbsdev-centos7-mky18-server 2021-02-04 20:11:24,107 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:24,323 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2021-02-04 20:11:24,333 ERROR server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local is down 2021-02-04 20:11:24,335 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:11:24,346 INFO running init script to restart pbs on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', '/opt/pbs/libexec/pbs_init.d', 'restart'] 2021-02-04 20:11:24,346 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/libexec/pbs_init.d restart 2021-02-04 20:11:30,276 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:30,314 INFO FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server 2021-02-04 20:11:30,317 INFOCLI2 pbsdev-centos7-mky18-server: which chown 2021-02-04 20:11:30,319 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:30,328 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:30,339 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:30,353 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:30,362 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:30,371 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_logs 2021-02-04 20:11:30,381 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:30,394 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:30,403 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:30,412 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:30,425 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:30,433 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:30,442 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_logs 2021-02-04 20:11:30,452 INFO FQDN name pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local differs from name provided mom1 2021-02-04 20:11:30,454 INFOCLI2 mom1: ssh mom1 ls -1 /opt/pbs/python/bin/python 2021-02-04 20:11:30,469 INFOCLI2 mom1: ssh mom1 python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:11:30,543 INFO FQDN name pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local differs from name provided mom2 2021-02-04 20:11:30,544 INFOCLI2 mom2: ssh mom2 ls -1 /opt/pbs/python/bin/python 2021-02-04 20:11:30,560 INFOCLI2 mom2: ssh mom2 python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:11:30,631 INFO FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server 2021-02-04 20:11:30,634 INFO manager on pbsdev-centos7-mky18-server: set server {'pbs_license_min': 2} 2021-02-04 20:11:30,635 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server pbs_license_min=2 2021-02-04 20:11:30,652 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:30,871 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:11:30,882 INFO running init script to stop pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:11:30,883 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbsc8y5380q Contents of /tmp/PtlPbsc8y5380q: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:11:36,498 INFOCLI2 pbsdev-centos7-mky18-server: which rm 2021-02-04 20:11:36,501 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbsc8y5380q 2021-02-04 20:11:36,506 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:11:36,517 INFO running init script to start pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:11:36,517 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbsdt3uqlkb Contents of /tmp/PtlPbsdt3uqlkb: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:11:40,517 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbsdt3uqlkb 2021-02-04 20:11:40,524 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2021-02-04 20:11:40,533 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:40,543 INFO =================================================== 2021-02-04 20:11:40,543 INFO Completed TestPbsModifyvnodeStateChanges setUpClass 2021-02-04 20:11:40,543 INFO =================================================== 2021-02-04 20:11:40,575 INFO mom pbsdev-centos7-mky18-mom1 log match: searching for "alps_client" - with existence - No match 2021-02-04 20:11:41,081 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -l /etc/sgi-compute-node-release 2021-02-04 20:11:41,102 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -l /etc/sgi-known-distributions 2021-02-04 20:11:41,122 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -l /etc/sgi-compute-node-release 2021-02-04 20:11:41,141 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -l /etc/sgi-known-distributions 2021-02-04 20:11:41,159 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo 2021-02-04 20:11:41,163 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs 2021-02-04 20:11:41,168 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:11:41,179 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:11:41,189 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:11:41,201 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:11:41,211 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:11:41,221 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:11:41,233 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:11:41,243 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:11:41,254 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:11:41,265 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:11:41,276 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:11:41,287 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:11:41,298 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:11:41,309 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:11:41,319 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:11:41,330 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:11:41,341 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:11:41,352 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:11:41,364 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:11:41,374 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:11:41,384 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:11:41,395 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:11:41,406 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:11:41,417 INFOCLI2 mom2: ssh mom2 which cat 2021-02-04 20:11:41,435 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo 2021-02-04 20:11:41,467 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs 2021-02-04 20:11:41,500 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:11:41,500 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:11:41,516 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:11:41,541 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:11:41,541 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:11:41,560 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:11:41,584 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:11:41,584 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:11:41,605 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:11:41,630 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:11:41,654 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:11:41,678 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:11:41,703 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:11:41,729 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:11:41,753 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:11:41,778 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:11:41,804 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:11:41,828 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:11:41,853 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:11:41,878 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:11:41,902 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:11:41,927 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:11:41,952 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:11:41,976 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:11:42,001 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:11:42,025 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:11:42,050 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:11:42,075 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:11:42,100 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:11:42,125 INFOCLI2 mom1: ssh mom1 which cat 2021-02-04 20:11:42,142 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo 2021-02-04 20:11:42,174 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs 2021-02-04 20:11:42,207 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:11:42,207 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:11:42,225 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:11:42,251 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:11:42,251 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:11:42,269 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:11:42,295 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:11:42,295 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:11:42,313 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:11:42,340 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:11:42,370 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:11:42,396 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:11:42,422 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:11:42,448 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:11:42,475 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:11:42,502 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:11:42,528 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:11:42,553 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:11:42,580 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:11:42,607 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:11:42,634 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:11:42,665 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:11:42,692 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:11:42,716 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:11:42,744 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:11:42,773 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:11:42,800 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:11:42,827 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:11:42,854 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:11:42,880 INFO test name: test_check_node_state_constants_00 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)... 2021-02-04 20:11:42,880 INFO test start time: Thu Feb 4 20:11:42 2021 2021-02-04 20:11:42,880 INFO test docstring: Test: verify expected node state constants and associated reverse map are defined in the pbs module and contain the expected values. 2021-02-04 20:11:42,881 INFO ============================================= 2021-02-04 20:11:42,881 INFO Entered TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:11:42,881 INFO ============================================= 2021-02-04 20:11:42,881 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:42,892 INFO status on pbsdev-centos7-mky18-server: server 2021-02-04 20:11:42,893 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:43,107 INFO manager on pbsdev-centos7-mky18-server: unset server managers 2021-02-04 20:11:43,108 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2021-02-04 20:11:43,127 INFO manager on pbsdev-centos7-mky18-server: unset server operators 2021-02-04 20:11:43,128 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators 2021-02-04 20:11:43,147 INFO status on pbsdev-centos7-mky18-server: server ['managers', 'operators'] 2021-02-04 20:11:43,147 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:43,359 INFO manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')} 2021-02-04 20:11:43,359 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@* 2021-02-04 20:11:43,378 INFO server pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:11:43,378 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:43,593 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:11:43,593 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:43,808 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:11:43,809 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:43,809 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:43,820 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:43,830 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:43,841 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:43,858 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:43,868 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:43,878 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:11:43,878 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:11:43,891 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:43,901 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:43,911 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:43,923 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:43,939 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:43,950 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:43,961 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:11:43,961 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:11:43,961 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:44,176 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:11:44,176 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:11:44,176 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:11:44,188 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:44,196 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:44,207 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:44,218 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:44,235 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:44,244 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:44,256 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:11:44,256 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:11:44,256 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:11:44,266 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:44,480 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:11:44,480 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:44,696 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:11:44,696 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:44,696 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:44,707 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:44,717 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:44,728 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:44,744 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:44,754 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:44,765 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:11:44,765 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:11:44,778 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:44,787 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:44,798 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:44,809 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:44,825 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:44,836 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:44,848 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:11:44,849 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:11:44,849 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:45,063 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:11:45,063 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:11:45,064 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:11:45,075 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:45,084 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:45,096 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:45,108 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:45,125 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:45,138 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:45,148 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:11:45,148 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:11:45,149 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:11:45,159 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:45,373 INFO manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'scheduler_iteration', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision', 'sync_mom_hookfiles_timeout', 'max_job_sequence_id'] 2021-02-04 20:11:45,373 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,scheduler_iteration,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision,sync_mom_hookfiles_timeout,max_job_sequence_id 2021-02-04 20:11:45,386 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:11:45,405 INFO manager on pbsdev-centos7-mky18-server: delete hook s1234 2021-02-04 20:11:45,405 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c delete hook s1234 2021-02-04 20:11:45,423 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:45,634 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:11:45,635 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:11:45,647 INFO manager on pbsdev-centos7-mky18-server: delete queue workq 2021-02-04 20:11:45,647 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq 2021-02-04 20:11:45,657 INFO manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'} 2021-02-04 20:11:45,657 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True 2021-02-04 20:11:45,666 INFO status on pbsdev-centos7-mky18-server: queue workq 2021-02-04 20:11:45,667 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:45,881 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:45,881 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:45,890 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:45,900 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:45,912 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:45,928 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:45,940 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:45,950 INFO manager on pbsdev-centos7-mky18-server: delete node @default 2021-02-04 20:11:45,951 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default 2021-02-04 20:11:45,962 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:11:45,981 INFO manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'} 2021-02-04 20:11:45,982 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false 2021-02-04 20:11:46,001 INFO manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'} 2021-02-04 20:11:46,001 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq 2021-02-04 20:11:46,012 INFO status on pbsdev-centos7-mky18-server: resource 2021-02-04 20:11:46,012 INFO manager on pbsdev-centos7-mky18-server: list resource 2021-02-04 20:11:46,012 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource 2021-02-04 20:11:46,022 INFOCLI status on pbsdev-centos7-mky18-server: server license_count 2021-02-04 20:11:46,022 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:46,031 INFO server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed 2021-02-04 20:11:46,032 INFO status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:46,032 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:46,247 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:11:46,265 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:11:46,315 INFO Set {'PBS_HOME': '/var/spool/pbs', 'PBS_EXEC': '/opt/pbs', 'PBS_SERVER': 'pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local', 'PBS_START_SCHED': '0', 'PBS_START_COMM': '0', 'PBS_START_SERVER': '0', 'PBS_START_MOM': '1', 'PBS_CORE_LIMIT': 'unlimited', 'PBS_SCP': '/usr/bin/scp', 'PBS_LOG_HIGHRES_TIMESTAMP': '1'} in /etc/pbs.conf 2021-02-04 20:11:46,315 INFOCLI2 pbsdev-centos7-mky18-server: which chmod 2021-02-04 20:11:46,318 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/chmod 644 /tmp/PtlPbs54xu546f 2021-02-04 20:11:46,336 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:11:46,520 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which chown 2021-02-04 20:11:46,534 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/chown root /etc/pbs.conf 2021-02-04 20:11:46,559 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which chgrp 2021-02-04 20:11:46,577 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/chgrp root /etc/pbs.conf 2021-02-04 20:11:46,602 INFOCLI2 pbsdev-centos7-mky18-server: which rm 2021-02-04 20:11:46,604 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbs54xu546f 2021-02-04 20:11:46,621 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:11:46,646 INFO running init script to restart pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'restart'] 2021-02-04 20:11:46,647 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsnuzpql_2 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsnuzpql_2 2021-02-04 20:11:46,672 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsnuzpql_2 Contents of /tmp/PtlPbsnuzpql_2: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d restart ---------------------------------------- 2021-02-04 20:11:47,786 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsnuzpql_2 2021-02-04 20:11:47,837 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import sys; print(sys.platform)" 2021-02-04 20:11:47,885 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:47,911 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:47,911 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:47,920 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:11:47,921 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:11:47,938 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:11:47,985 INFO Set {'PBS_HOME': '/var/spool/pbs', 'PBS_EXEC': '/opt/pbs', 'PBS_SERVER': 'pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local', 'PBS_START_SCHED': '0', 'PBS_START_COMM': '0', 'PBS_START_SERVER': '0', 'PBS_START_MOM': '1', 'PBS_CORE_LIMIT': 'unlimited', 'PBS_SCP': '/usr/bin/scp', 'PBS_LOG_HIGHRES_TIMESTAMP': '1'} in /etc/pbs.conf 2021-02-04 20:11:47,986 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/chmod 644 /tmp/PtlPbspip9brgh 2021-02-04 20:11:48,003 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:11:48,184 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which chown 2021-02-04 20:11:48,199 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/chown root /etc/pbs.conf 2021-02-04 20:11:48,223 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which chgrp 2021-02-04 20:11:48,241 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/chgrp root /etc/pbs.conf 2021-02-04 20:11:48,266 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbspip9brgh 2021-02-04 20:11:48,284 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:11:48,309 INFO running init script to restart pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'restart'] 2021-02-04 20:11:48,311 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsp2zqc4zd pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbsp2zqc4zd 2021-02-04 20:11:48,335 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbsp2zqc4zd Contents of /tmp/PtlPbsp2zqc4zd: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d restart ---------------------------------------- 2021-02-04 20:11:49,446 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsp2zqc4zd 2021-02-04 20:11:49,498 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import sys; print(sys.platform)" 2021-02-04 20:11:49,545 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:49,570 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:49,570 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:49,579 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:11:49,582 ERROR Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value 2021-02-04 20:11:49,589 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:11:49,600 INFO scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:11:49,601 INFO manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm'] 2021-02-04 20:11:49,601 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm 2021-02-04 20:11:49,615 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:49,628 INFOCLI2 pbsdev-centos7-mky18-server: which cp 2021-02-04 20:11:49,631 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbscdkufyrv /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:49,643 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:49,653 INFO scheduler pbsdev-centos7-mky18-server: reverting holidays file to default 2021-02-04 20:11:49,653 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:49,664 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2021-02-04 20:11:49,675 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:49,686 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:49,697 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:49,708 INFO Scheduler sent signal -HUP 2021-02-04 20:11:49,709 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:11:49,719 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148 2021-02-04 20:11:49,730 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default 2021-02-04 20:11:49,755 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:11:49,766 INFO status on pbsdev-centos7-mky18-server: sched default 2021-02-04 20:11:49,766 INFO manager on pbsdev-centos7-mky18-server: list sched default 2021-02-04 20:11:49,766 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:49,777 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:49,789 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:49,802 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:49,818 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:49,830 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:49,861 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:49,888 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:49,888 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:49,898 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:11:49,899 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:11:49,923 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:11:49,948 INFOCLI pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:11:50,037 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:11:50,092 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbshuyki0mr pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmpueq3zibvPtlPbstmpcopy 2021-02-04 20:11:50,116 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpueq3zibvPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:11:50,141 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpueq3zibvPtlPbstmpcopy 2021-02-04 20:11:50,157 INFO MoM sent signal -HUP 2021-02-04 20:11:50,158 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:50,182 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 12996 2021-02-04 20:11:50,225 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:50,248 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,249 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,258 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:11:50,259 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,259 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,272 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,273 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,282 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:50,292 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 got: state = state-unknown,down 2021-02-04 20:11:51,295 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:51,308 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2 got: state = state-unknown,down 2021-02-04 20:11:52,312 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:52,325 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ... OK 2021-02-04 20:11:52,325 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:11:52,326 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:11:52,354 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:52,378 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,379 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,388 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:11:52,388 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:11:52,413 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:11:52,438 INFOCLI pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:11:52,525 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:11:52,575 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbs96iz2riq pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpgkfw2h_qPtlPbstmpcopy 2021-02-04 20:11:52,598 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpgkfw2h_qPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:11:52,623 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpgkfw2h_qPtlPbstmpcopy 2021-02-04 20:11:52,639 INFO MoM sent signal -HUP 2021-02-04 20:11:52,640 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:52,664 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 12946 2021-02-04 20:11:52,707 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:11:52,731 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,731 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,741 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:11:52,742 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,742 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,756 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,756 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,766 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:52,776 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 got: state = state-unknown,down 2021-02-04 20:11:53,743 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:53,756 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2 got: state = state-unknown,down 2021-02-04 20:11:54,759 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:54,771 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ... OK 2021-02-04 20:11:54,771 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:11:54,771 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:11:54,781 INFO manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'} 2021-02-04 20:11:54,781 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off 2021-02-04 20:11:54,798 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2021-02-04 20:11:54,809 INFO ============================================== 2021-02-04 20:11:54,809 INFO Completed TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:11:54,809 INFO ============================================== 2021-02-04 20:11:54,823 INFO ================================================ 2021-02-04 20:11:54,823 INFO Entered TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:11:54,824 INFO ================================================ 2021-02-04 20:11:54,824 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:54,834 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:11:54,834 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:55,048 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:11:55,048 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:55,049 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:55,060 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:55,070 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:55,081 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:55,096 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:55,106 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:55,117 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:11:55,118 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:11:55,130 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:55,140 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:55,150 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:55,162 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:55,179 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:55,190 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:55,200 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:11:55,200 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:11:55,201 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:55,419 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:11:55,420 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:11:55,420 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:11:55,433 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:55,444 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:55,455 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:55,469 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:55,485 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:55,497 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:55,509 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:11:55,509 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:55,509 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:55,520 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:55,533 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:55,545 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:55,560 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:55,573 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:55,585 INFO ================================================= 2021-02-04 20:11:55,585 INFO Completed TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:11:55,586 INFO ================================================= 2021-02-04 20:11:55,587 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:11:55,642 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:11:55,697 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:11:55,752 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:11:55,807 INFO ok 2021-02-04 20:11:55,808 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo 2021-02-04 20:11:55,812 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs 2021-02-04 20:11:55,816 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:11:55,826 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:11:55,838 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:11:55,848 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:11:55,857 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:11:55,868 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:11:55,879 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:11:55,890 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:11:55,901 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:11:55,912 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:11:55,924 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:11:55,935 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:11:55,945 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:11:55,957 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:11:55,968 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:11:55,979 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:11:55,990 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:11:56,001 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:11:56,012 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:11:56,024 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:11:56,035 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:11:56,045 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:11:56,057 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:11:56,067 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo 2021-02-04 20:11:56,100 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs 2021-02-04 20:11:56,132 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:11:56,133 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:11:56,150 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:11:56,175 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:11:56,176 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:11:56,195 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:11:56,219 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:11:56,219 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:11:56,237 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:11:56,262 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:11:56,287 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:11:56,312 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:11:56,337 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:11:56,362 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:11:56,387 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:11:56,412 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:11:56,438 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:11:56,463 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:11:56,488 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:11:56,514 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:11:56,538 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:11:56,564 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:11:56,589 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:11:56,614 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:11:56,639 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:11:56,664 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:11:56,688 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:11:56,713 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:11:56,738 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:11:56,763 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo 2021-02-04 20:11:56,796 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs 2021-02-04 20:11:56,829 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:11:56,829 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:11:56,846 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:11:56,872 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:11:56,872 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:11:56,890 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:11:56,916 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:11:56,916 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:11:56,935 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:11:56,959 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:11:56,987 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:11:57,014 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:11:57,040 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:11:57,067 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:11:57,093 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:11:57,119 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:11:57,143 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:11:57,172 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:11:57,198 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:11:57,225 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:11:57,252 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:11:57,276 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:11:57,304 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:11:57,332 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:11:57,359 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:11:57,386 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:11:57,410 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:11:57,439 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:11:57,466 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:11:57,493 INFO test name: test_check_node_state_lookup_00 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)... 2021-02-04 20:11:57,493 INFO test start time: Thu Feb 4 20:11:57 2021 2021-02-04 20:11:57,493 INFO test docstring: Test: check for the existence and values of the pbs.REVERSE_STATE_CHANGES dictionary run a hook that converts a state change hex into a string, then search for it in the server log. 2021-02-04 20:11:57,493 INFO ============================================= 2021-02-04 20:11:57,494 INFO Entered TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:11:57,494 INFO ============================================= 2021-02-04 20:11:57,494 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:57,503 INFO status on pbsdev-centos7-mky18-server: server 2021-02-04 20:11:57,504 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:57,716 INFO manager on pbsdev-centos7-mky18-server: unset server managers 2021-02-04 20:11:57,716 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2021-02-04 20:11:57,736 INFO manager on pbsdev-centos7-mky18-server: unset server operators 2021-02-04 20:11:57,736 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators 2021-02-04 20:11:57,755 INFO status on pbsdev-centos7-mky18-server: server ['managers', 'operators'] 2021-02-04 20:11:57,756 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:57,966 INFO manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')} 2021-02-04 20:11:57,967 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@* 2021-02-04 20:11:57,985 INFO server pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:11:57,986 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:58,201 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:11:58,201 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:58,415 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:11:58,416 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:58,416 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:58,425 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:58,435 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:58,446 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:58,461 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:58,472 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:58,483 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:11:58,483 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:11:58,494 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:58,504 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:58,515 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:58,526 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:58,542 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:58,553 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:58,563 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:11:58,563 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:11:58,563 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:58,777 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:11:58,778 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:11:58,778 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:11:58,791 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:58,800 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:58,811 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:58,823 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:58,839 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:58,850 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:58,861 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:11:58,861 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:11:58,862 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:11:58,871 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:59,089 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:11:59,089 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:59,306 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:11:59,306 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:11:59,307 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:11:59,318 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:59,328 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:59,339 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:59,356 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:59,367 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:59,377 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:11:59,377 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:11:59,389 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:59,399 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:59,410 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:59,421 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:59,437 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:59,447 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:59,458 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:11:59,459 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:11:59,459 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:59,673 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:11:59,673 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:11:59,673 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:11:59,685 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:11:59,695 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:11:59,706 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:11:59,719 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:11:59,735 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:11:59,745 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:11:59,756 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:11:59,757 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:11:59,757 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:11:59,768 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:11:59,985 INFO manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision'] 2021-02-04 20:11:59,985 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision 2021-02-04 20:11:59,997 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:12:00,015 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:00,025 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:12:00,025 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:12:00,035 INFO manager on pbsdev-centos7-mky18-server: delete queue workq 2021-02-04 20:12:00,035 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq 2021-02-04 20:12:00,047 INFO manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'} 2021-02-04 20:12:00,047 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True 2021-02-04 20:12:00,058 INFO status on pbsdev-centos7-mky18-server: queue workq 2021-02-04 20:12:00,059 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:00,271 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:12:00,271 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:12:00,281 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:00,290 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:00,303 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:00,321 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:00,334 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:00,345 INFO manager on pbsdev-centos7-mky18-server: delete node @default 2021-02-04 20:12:00,346 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default 2021-02-04 20:12:00,358 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:12:00,376 INFO manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'} 2021-02-04 20:12:00,376 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false 2021-02-04 20:12:00,396 INFO manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'} 2021-02-04 20:12:00,396 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq 2021-02-04 20:12:00,407 INFO status on pbsdev-centos7-mky18-server: resource 2021-02-04 20:12:00,407 INFO manager on pbsdev-centos7-mky18-server: list resource 2021-02-04 20:12:00,407 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource 2021-02-04 20:12:00,417 INFOCLI status on pbsdev-centos7-mky18-server: server license_count 2021-02-04 20:12:00,417 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:00,629 INFO server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed 2021-02-04 20:12:00,629 INFO status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:00,629 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:00,877 ERROR Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value 2021-02-04 20:12:00,882 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:12:00,892 INFO scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:12:00,892 INFO manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm'] 2021-02-04 20:12:00,893 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm 2021-02-04 20:12:00,906 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:00,917 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbsb_dgr0cv /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:00,928 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:00,939 INFO scheduler pbsdev-centos7-mky18-server: reverting holidays file to default 2021-02-04 20:12:00,939 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:00,949 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2021-02-04 20:12:00,961 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:00,971 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:00,982 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:00,993 INFO Scheduler sent signal -HUP 2021-02-04 20:12:00,994 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:12:01,004 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148 2021-02-04 20:12:01,015 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default 2021-02-04 20:12:01,039 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:12:01,050 INFO status on pbsdev-centos7-mky18-server: sched default 2021-02-04 20:12:01,050 INFO manager on pbsdev-centos7-mky18-server: list sched default 2021-02-04 20:12:01,051 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:01,061 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:01,072 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:01,083 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:01,100 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:01,110 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:01,140 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:01,165 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,166 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,175 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:12:01,176 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:12:01,201 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:12:01,225 INFOCLI pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:12:01,317 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:12:01,373 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbs1nptahq8 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmpussqkshePtlPbstmpcopy 2021-02-04 20:12:01,397 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpussqkshePtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:12:01,421 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpussqkshePtlPbstmpcopy 2021-02-04 20:12:01,438 INFO MoM sent signal -HUP 2021-02-04 20:12:01,439 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:01,464 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 12996 2021-02-04 20:12:01,507 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:01,532 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,532 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,541 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:12:01,541 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,542 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,555 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,555 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,565 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:01,575 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 got: state = state-unknown,down 2021-02-04 20:12:02,577 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:02,591 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2 got: state = state-unknown,down 2021-02-04 20:12:03,593 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:03,605 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ... OK 2021-02-04 20:12:03,606 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:03,606 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:03,636 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:03,662 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:03,662 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:03,673 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:12:03,673 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:12:03,699 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:12:03,726 INFOCLI pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:12:03,816 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:12:03,867 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbs_iw17ra4 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpukldks58PtlPbstmpcopy 2021-02-04 20:12:03,893 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpukldks58PtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:12:03,920 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpukldks58PtlPbstmpcopy 2021-02-04 20:12:03,936 INFO MoM sent signal -HUP 2021-02-04 20:12:03,936 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:03,964 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 12946 2021-02-04 20:12:04,009 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:04,034 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,035 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,046 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:12:04,046 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,046 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,060 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,060 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,070 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:04,081 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 got: state = state-unknown,down 2021-02-04 20:12:05,087 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:05,100 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2 got: state = state-unknown,down 2021-02-04 20:12:06,102 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:06,113 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ... OK 2021-02-04 20:12:06,114 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:06,114 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:06,123 INFO manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'} 2021-02-04 20:12:06,123 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off 2021-02-04 20:12:06,142 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2021-02-04 20:12:06,155 INFO ============================================== 2021-02-04 20:12:06,155 INFO Completed TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:12:06,155 INFO ============================================== 2021-02-04 20:12:06,156 INFO manager on pbsdev-centos7-mky18-server: set server {'log_events': 4095} 2021-02-04 20:12:06,156 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server log_events=4095 2021-02-04 20:12:06,166 INFO status on pbsdev-centos7-mky18-server: hook 2021-02-04 20:12:06,166 INFO manager on pbsdev-centos7-mky18-server: list hook 2021-02-04 20:12:06,167 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:12:06,184 INFO manager on pbsdev-centos7-mky18-server: create hook x1234 2021-02-04 20:12:06,185 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c create hook x1234 2021-02-04 20:12:06,203 INFO status on pbsdev-centos7-mky18-server: hook x1234 2021-02-04 20:12:06,203 INFO manager on pbsdev-centos7-mky18-server: list hook x1234 2021-02-04 20:12:06,203 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook x1234 2021-02-04 20:12:06,222 INFO status on pbsdev-centos7-mky18-server: hook x1234 2021-02-04 20:12:06,223 INFO manager on pbsdev-centos7-mky18-server: list hook x1234 2021-02-04 20:12:06,223 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook x1234 2021-02-04 20:12:06,241 INFO manager on pbsdev-centos7-mky18-server: set hook x1234 {'event': 'modifyvnode', 'enabled': 'True', 'debug': 'True'} 2021-02-04 20:12:06,241 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook x1234 event=modifyvnode,enabled=True,debug=True 2021-02-04 20:12:06,261 INFO status on pbsdev-centos7-mky18-server: server sync_mom_hookfiles_timeout 2021-02-04 20:12:06,261 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:06,270 INFO Setting sync_mom_hookfiles_timeout to 15s 2021-02-04 20:12:06,271 INFO manager on pbsdev-centos7-mky18-server: set server {'sync_mom_hookfiles_timeout': 15} 2021-02-04 20:12:06,271 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server sync_mom_hookfiles_timeout=15 2021-02-04 20:12:06,282 INFO manager on pbsdev-centos7-mky18-server: import hook x1234 {'content-type': 'application/x-python', 'content-encoding': 'default', 'input-file': '/tmp/PtlPbsv0nc1wqr'} 2021-02-04 20:12:06,282 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c import hook x1234 application/x-python default /tmp/PtlPbsv0nc1wqr 2021-02-04 20:12:06,300 INFO server pbsdev-centos7-mky18-server: imported hook body --- import pbs e = pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "pbs.__file__:" + pbs.__file__) # this is backwards as it's a reverse lookup. for value, key in pbs.REVERSE_NODE_STATE.items(): pbs.logmsg(pbs.LOG_DEBUG, "key:%s value:%s" % (key, value)) e.accept() --- 2021-02-04 20:12:06,320 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:06,345 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:06,345 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:06,355 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:06,365 INFO expect on server pbsdev-centos7-mky18-server: state ~ free|provisioning|offline|job-busy node pbsdev-centos7-mky18-mom1 ... OK 2021-02-04 20:12:06,381 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:06,406 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:12:06,408 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsi3nrzgau pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsi3nrzgau 2021-02-04 20:12:06,432 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsi3nrzgau Contents of /tmp/PtlPbsi3nrzgau: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:12:07,495 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsi3nrzgau 2021-02-04 20:12:07,529 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:07,554 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:12:07,556 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbszmncwu22 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbszmncwu22 2021-02-04 20:12:07,580 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbszmncwu22 Contents of /tmp/PtlPbszmncwu22: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:12:07,680 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbszmncwu22 2021-02-04 20:12:07,716 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:07,746 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:12:08,898 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - attempt 2 2021-02-04 20:12:10,060 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - attempt 3 2021-02-04 20:12:11,195 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:12:11,203 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence - No match 2021-02-04 20:12:12,338 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:12:12,344 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_FREE value:0" - with existence... OK 2021-02-04 20:12:12,347 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE value:1" - with existence... OK 2021-02-04 20:12:12,350 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DOWN value:2" - with existence... OK 2021-02-04 20:12:12,353 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DELETED value:4" - with existence... OK 2021-02-04 20:12:12,359 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_STALE value:32" - with existence... OK 2021-02-04 20:12:12,364 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOBBUSY value:16" - with existence... OK 2021-02-04 20:12:12,367 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOB_EXCLUSIVE value:64" - with existence... OK 2021-02-04 20:12:12,370 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_RESV_EXCLUSIVE value:8192" - with existence... OK 2021-02-04 20:12:12,372 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_BUSY value:128" - with existence... OK 2021-02-04 20:12:12,375 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNKNOWN value:256" - with existence... OK 2021-02-04 20:12:12,377 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEEDS_HELLOSVR value:512" - with existence... OK 2021-02-04 20:12:12,380 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_INIT value:1024" - with existence... OK 2021-02-04 20:12:12,384 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_PROV value:2048" - with existence... OK 2021-02-04 20:12:12,387 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_WAIT_PROV value:4096" - with existence... OK 2021-02-04 20:12:12,392 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNRESOLVABLE value:8" - with existence... OK 2021-02-04 20:12:12,395 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_SLEEP value:262144" - with existence... OK 2021-02-04 20:12:12,399 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE_BY_MOM value:16384" - with existence... OK 2021-02-04 20:12:12,402 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MARKEDDOWN value:32768" - with existence... OK 2021-02-04 20:12:12,405 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_ADDRS value:65536" - with existence... OK 2021-02-04 20:12:12,407 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MAINTENANCE value:131072" - with existence... OK 2021-02-04 20:12:12,410 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_CREDENTIALS value:524288" - with existence... OK 2021-02-04 20:12:12,413 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_VNODE_UNAVAILABLE value:409903" - with existence... OK 2021-02-04 20:12:12,435 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:12,461 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:12,461 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:12,472 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:12,482 INFO expect on server pbsdev-centos7-mky18-server: state ~ free|provisioning|offline|job-busy node pbsdev-centos7-mky18-mom2 ... OK 2021-02-04 20:12:12,501 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:12,528 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:12:12,529 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbs5dgi0u1v pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbs5dgi0u1v 2021-02-04 20:12:12,555 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbs5dgi0u1v Contents of /tmp/PtlPbs5dgi0u1v: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:12:13,621 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbs5dgi0u1v 2021-02-04 20:12:13,654 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:13,681 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:12:13,682 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbs1on8oko1 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbs1on8oko1 2021-02-04 20:12:13,707 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbs1on8oko1 Contents of /tmp/PtlPbs1on8oko1: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:12:13,805 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbs1on8oko1 2021-02-04 20:12:13,839 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:13,868 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:12:15,028 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - attempt 2 2021-02-04 20:12:16,185 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - attempt 3 2021-02-04 20:12:17,330 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:12:17,337 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence - No match 2021-02-04 20:12:18,492 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:12:18,499 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_FREE value:0" - with existence... OK 2021-02-04 20:12:18,502 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE value:1" - with existence... OK 2021-02-04 20:12:18,505 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DOWN value:2" - with existence... OK 2021-02-04 20:12:18,508 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DELETED value:4" - with existence... OK 2021-02-04 20:12:18,511 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_STALE value:32" - with existence... OK 2021-02-04 20:12:18,514 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOBBUSY value:16" - with existence... OK 2021-02-04 20:12:18,518 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOB_EXCLUSIVE value:64" - with existence... OK 2021-02-04 20:12:18,521 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_RESV_EXCLUSIVE value:8192" - with existence... OK 2021-02-04 20:12:18,524 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_BUSY value:128" - with existence... OK 2021-02-04 20:12:18,526 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNKNOWN value:256" - with existence... OK 2021-02-04 20:12:18,529 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEEDS_HELLOSVR value:512" - with existence... OK 2021-02-04 20:12:18,531 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_INIT value:1024" - with existence... OK 2021-02-04 20:12:18,533 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_PROV value:2048" - with existence... OK 2021-02-04 20:12:18,536 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_WAIT_PROV value:4096" - with existence... OK 2021-02-04 20:12:18,540 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNRESOLVABLE value:8" - with existence... OK 2021-02-04 20:12:18,543 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_SLEEP value:262144" - with existence... OK 2021-02-04 20:12:18,546 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE_BY_MOM value:16384" - with existence... OK 2021-02-04 20:12:18,549 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MARKEDDOWN value:32768" - with existence... OK 2021-02-04 20:12:18,552 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_ADDRS value:65536" - with existence... OK 2021-02-04 20:12:18,556 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MAINTENANCE value:131072" - with existence... OK 2021-02-04 20:12:18,559 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_CREDENTIALS value:524288" - with existence... OK 2021-02-04 20:12:18,561 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_VNODE_UNAVAILABLE value:409903" - with existence... OK 2021-02-04 20:12:18,562 INFO ================================================ 2021-02-04 20:12:18,562 INFO Entered TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:12:18,562 INFO ================================================ 2021-02-04 20:12:18,562 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:18,572 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:12:18,572 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:18,791 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:12:18,791 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:12:18,791 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:12:18,803 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:18,814 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:18,826 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:18,843 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:18,854 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:18,866 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:12:18,867 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:12:18,879 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:18,890 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:18,900 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:18,913 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:18,929 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:18,940 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:18,952 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:12:18,952 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:12:18,953 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:19,170 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:12:19,170 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:12:19,170 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:12:19,184 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:19,194 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:19,204 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:19,216 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:19,232 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:19,244 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:19,256 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:12:19,257 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:12:19,257 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:12:19,269 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:19,281 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:19,295 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:19,312 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:19,323 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:19,334 INFO ================================================= 2021-02-04 20:12:19,335 INFO Completed TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:12:19,335 INFO ================================================= 2021-02-04 20:12:19,335 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:12:19,392 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:12:19,449 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:12:19,506 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:12:19,564 INFO ok 2021-02-04 20:12:19,565 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo 2021-02-04 20:12:19,570 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs 2021-02-04 20:12:19,576 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:12:19,588 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:12:19,603 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:12:19,615 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:12:19,628 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:12:19,640 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:12:19,652 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:12:19,666 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:12:19,677 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:12:19,688 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:12:19,700 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:12:19,711 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:12:19,722 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:12:19,734 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:12:19,745 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:12:19,756 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:12:19,768 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:12:19,779 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:12:19,790 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:12:19,802 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:12:19,813 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:12:19,825 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:12:19,836 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:12:19,847 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo 2021-02-04 20:12:19,879 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs 2021-02-04 20:12:19,913 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:12:19,913 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:12:19,931 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:12:19,957 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:12:19,957 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:12:19,975 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:12:20,000 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:12:20,000 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:12:20,017 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:12:20,044 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:12:20,070 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:12:20,095 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:12:20,121 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:12:20,146 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:12:20,172 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:12:20,197 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:12:20,223 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:12:20,248 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:12:20,275 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:12:20,302 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:12:20,327 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:12:20,353 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:12:20,378 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:12:20,404 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:12:20,430 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:12:20,455 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:12:20,481 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:12:20,506 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:12:20,532 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:12:20,558 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo 2021-02-04 20:12:20,590 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs 2021-02-04 20:12:20,624 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:12:20,624 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:12:20,642 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:12:20,668 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:12:20,668 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:12:20,685 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:12:20,711 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:12:20,712 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:12:20,729 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:12:20,754 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:12:20,780 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:12:20,806 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:12:20,831 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:12:20,857 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:12:20,882 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:12:20,908 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:12:20,934 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:12:20,960 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:12:20,985 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:12:21,011 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:12:21,037 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:12:21,064 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:12:21,090 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:12:21,116 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:12:21,143 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:12:21,170 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:12:21,197 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:12:21,224 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:12:21,250 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:12:21,278 INFO test name: test_hook_state_changes_00 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)... 2021-02-04 20:12:21,279 INFO test start time: Thu Feb 4 20:12:21 2021 2021-02-04 20:12:21,279 INFO test docstring: Test: induce a variety of vnode state changes with debug turned on and inspect the pbs log for expected entries 2021-02-04 20:12:21,279 INFO ============================================= 2021-02-04 20:12:21,279 INFO Entered TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:12:21,279 INFO ============================================= 2021-02-04 20:12:21,279 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:21,289 INFO status on pbsdev-centos7-mky18-server: server 2021-02-04 20:12:21,289 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:21,505 INFO manager on pbsdev-centos7-mky18-server: unset server managers 2021-02-04 20:12:21,505 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2021-02-04 20:12:21,526 INFO manager on pbsdev-centos7-mky18-server: unset server operators 2021-02-04 20:12:21,526 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators 2021-02-04 20:12:21,547 INFO status on pbsdev-centos7-mky18-server: server ['managers', 'operators'] 2021-02-04 20:12:21,548 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:21,761 INFO manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')} 2021-02-04 20:12:21,761 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@* 2021-02-04 20:12:21,780 INFO server pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:12:21,781 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:21,995 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:12:21,995 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:22,209 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:12:22,210 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:12:22,210 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:12:22,221 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:22,231 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:22,244 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:22,260 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:22,272 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:22,284 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:12:22,284 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:12:22,298 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:22,309 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:22,320 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:22,331 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:22,348 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:22,359 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:22,371 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:12:22,371 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:12:22,371 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:22,587 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:12:22,587 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:12:22,588 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:12:22,601 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:22,611 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:22,621 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:22,633 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:22,649 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:22,661 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:22,671 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:12:22,671 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:12:22,671 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:12:22,682 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:22,898 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:12:22,899 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:23,078 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:12:23,079 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:12:23,079 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:12:23,091 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:23,101 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:23,112 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:23,129 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:23,140 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:23,150 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:12:23,151 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:12:23,164 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:23,175 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:23,185 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:23,197 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:23,214 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:23,226 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:23,237 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:12:23,238 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:12:23,238 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:23,452 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:12:23,452 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:12:23,453 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:12:23,467 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:23,476 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:23,487 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:23,499 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:23,516 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:23,527 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:23,538 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:12:23,538 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:12:23,538 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:12:23,549 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:23,763 INFO manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision', 'sync_mom_hookfiles_timeout'] 2021-02-04 20:12:23,764 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision,sync_mom_hookfiles_timeout 2021-02-04 20:12:23,778 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:12:23,796 INFO manager on pbsdev-centos7-mky18-server: delete hook x1234 2021-02-04 20:12:23,796 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c delete hook x1234 2021-02-04 20:12:23,815 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:24,028 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:12:24,028 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:12:24,040 INFO manager on pbsdev-centos7-mky18-server: delete queue workq 2021-02-04 20:12:24,041 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq 2021-02-04 20:12:24,054 INFO manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'} 2021-02-04 20:12:24,054 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True 2021-02-04 20:12:24,067 INFO status on pbsdev-centos7-mky18-server: queue workq 2021-02-04 20:12:24,068 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:24,282 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:12:24,283 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:12:24,295 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:24,306 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:24,317 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:24,335 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:24,348 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:24,360 INFO manager on pbsdev-centos7-mky18-server: delete node @default 2021-02-04 20:12:24,360 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default 2021-02-04 20:12:24,375 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:12:24,394 INFO manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'} 2021-02-04 20:12:24,395 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false 2021-02-04 20:12:24,417 INFO manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'} 2021-02-04 20:12:24,417 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq 2021-02-04 20:12:24,428 INFO status on pbsdev-centos7-mky18-server: resource 2021-02-04 20:12:24,428 INFO manager on pbsdev-centos7-mky18-server: list resource 2021-02-04 20:12:24,428 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource 2021-02-04 20:12:24,439 INFOCLI status on pbsdev-centos7-mky18-server: server license_count 2021-02-04 20:12:24,440 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:24,655 INFO server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed 2021-02-04 20:12:24,655 INFO status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:24,655 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:24,911 ERROR Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value 2021-02-04 20:12:24,917 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:12:24,927 INFO scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:12:24,927 INFO manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm'] 2021-02-04 20:12:24,927 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm 2021-02-04 20:12:24,941 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:24,952 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbsojn6smmo /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:24,965 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:24,975 INFO scheduler pbsdev-centos7-mky18-server: reverting holidays file to default 2021-02-04 20:12:24,975 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:24,986 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2021-02-04 20:12:24,998 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:25,009 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:25,020 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:25,032 INFO Scheduler sent signal -HUP 2021-02-04 20:12:25,032 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:12:25,043 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148 2021-02-04 20:12:25,054 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default 2021-02-04 20:12:25,079 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:12:25,091 INFO status on pbsdev-centos7-mky18-server: sched default 2021-02-04 20:12:25,091 INFO manager on pbsdev-centos7-mky18-server: list sched default 2021-02-04 20:12:25,091 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:12:25,103 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:12:25,113 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:12:25,125 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:12:25,142 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:12:25,152 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:12:25,182 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:25,208 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,208 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,217 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:12:25,218 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:12:25,243 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:12:25,268 INFOCLI pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:12:25,355 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:12:25,411 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbspu_ezd39 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmp19gjncqvPtlPbstmpcopy 2021-02-04 20:12:25,435 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmp19gjncqvPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:12:25,460 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmp19gjncqvPtlPbstmpcopy 2021-02-04 20:12:25,477 INFO MoM sent signal -HUP 2021-02-04 20:12:25,477 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:25,503 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 13677 2021-02-04 20:12:25,546 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:25,571 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,571 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,581 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:12:25,581 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,581 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,595 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,596 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,606 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:25,616 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 got: state = state-unknown,down 2021-02-04 20:12:26,621 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:26,635 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2 got: state = state-unknown,down 2021-02-04 20:12:27,636 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:27,649 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ... OK 2021-02-04 20:12:27,649 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:27,649 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:27,679 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:27,705 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:27,706 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:27,716 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:12:27,717 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:12:27,742 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:12:27,768 INFOCLI pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:12:27,857 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:12:27,908 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbsp__atm72 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpm2mf37buPtlPbstmpcopy 2021-02-04 20:12:27,932 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpm2mf37buPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:12:27,958 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpm2mf37buPtlPbstmpcopy 2021-02-04 20:12:27,974 INFO MoM sent signal -HUP 2021-02-04 20:12:27,975 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:28,001 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 13627 2021-02-04 20:12:28,045 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:28,073 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,073 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,083 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:12:28,084 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,084 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,098 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,099 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,111 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:28,121 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 got: state = state-unknown,down 2021-02-04 20:12:29,126 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:29,140 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2 got: state = state-unknown,down 2021-02-04 20:12:30,142 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:30,152 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ... OK 2021-02-04 20:12:30,152 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:12:30,153 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:12:30,162 INFO manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'} 2021-02-04 20:12:30,163 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off 2021-02-04 20:12:30,181 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2021-02-04 20:12:30,192 INFO ============================================== 2021-02-04 20:12:30,192 INFO Completed TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:12:30,192 INFO ============================================== 2021-02-04 20:12:30,192 INFO manager on pbsdev-centos7-mky18-server: set server {'log_events': 4095} 2021-02-04 20:12:30,193 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server log_events=4095 2021-02-04 20:12:30,205 INFO status on pbsdev-centos7-mky18-server: hook 2021-02-04 20:12:30,205 INFO manager on pbsdev-centos7-mky18-server: list hook 2021-02-04 20:12:30,205 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:12:30,225 INFO manager on pbsdev-centos7-mky18-server: create hook m1234 2021-02-04 20:12:30,225 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c create hook m1234 2021-02-04 20:12:30,246 INFO status on pbsdev-centos7-mky18-server: hook m1234 2021-02-04 20:12:30,246 INFO manager on pbsdev-centos7-mky18-server: list hook m1234 2021-02-04 20:12:30,246 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook m1234 2021-02-04 20:12:30,266 INFO status on pbsdev-centos7-mky18-server: hook m1234 2021-02-04 20:12:30,266 INFO manager on pbsdev-centos7-mky18-server: list hook m1234 2021-02-04 20:12:30,266 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook m1234 2021-02-04 20:12:30,286 INFO manager on pbsdev-centos7-mky18-server: set hook m1234 {'event': 'modifyvnode', 'enabled': 'True', 'debug': 'True'} 2021-02-04 20:12:30,287 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook m1234 event=modifyvnode,enabled=True,debug=True 2021-02-04 20:12:30,307 INFO status on pbsdev-centos7-mky18-server: server sync_mom_hookfiles_timeout 2021-02-04 20:12:30,307 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:12:30,317 INFO Setting sync_mom_hookfiles_timeout to 15s 2021-02-04 20:12:30,317 INFO manager on pbsdev-centos7-mky18-server: set server {'sync_mom_hookfiles_timeout': 15} 2021-02-04 20:12:30,317 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server sync_mom_hookfiles_timeout=15 2021-02-04 20:12:30,329 INFO manager on pbsdev-centos7-mky18-server: import hook m1234 {'content-type': 'application/x-python', 'content-encoding': 'default', 'input-file': '/tmp/PtlPbsii2_muph'} 2021-02-04 20:12:30,329 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c import hook m1234 application/x-python default /tmp/PtlPbsii2_muph 2021-02-04 20:12:30,349 INFO server pbsdev-centos7-mky18-server: imported hook body --- import pbs import sys try: e = pbs.event() v = e.vnode v_o = e.vnode_o lsct = v.last_state_change_time lsct_o = v_o.last_state_change_time state_str_buf_v = ",".join(v.extract_state_strs()) state_str_buf_v_o = ",".join(v_o.extract_state_strs()) state_int_buf_v = ','.join([str(_) for _ in v.extract_state_ints()]) state_int_buf_v_o = ','.join( [str(_) for _ in v_o.extract_state_ints()]) # print show_vnode_state record (bi consumable) svs1_data = "v.state_hex=%s v_o.state_hex=%s v.state_strs=%s " "v_o.state_strs=%s" % (hex(v.state), hex(v_o.state), state_str_buf_v, state_str_buf_v_o) svs2_data = "v.state_ints=%s v_o.state_ints=%s v.lsct=%s " "v_o.lsct=%s" % (state_int_buf_v, state_int_buf_v_o, str(lsct), str(lsct_o)) svs_data = "%s %s" % (svs1_data, svs2_data) pbs.logmsg(pbs.LOG_DEBUG, "show_vnode_state;name=%s %s" % (v.name, svs_data)) # print additional hook parameter values pbs.logmsg(pbs.LOG_DEBUG, "name: v=%s, v_o=%s" % (v.name, v_o.name)) pbs.logmsg(pbs.LOG_DEBUG, "state: v=%s, v_o=%s" % (hex(v.state), hex(v_o.state))) pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: v=%s, v_o=%s" % ( str(lsct), str(lsct_o))) pbs.logmsg(pbs.LOG_DEBUG, "comment: v=%s, v_o=%s" % (v.comment, v_o.comment)) pbs.logmsg(pbs.LOG_DEBUG, "aoe: v=%s, v_o=%s" % (v.current_aoe, v_o.current_aoe)) pbs.logmsg(pbs.LOG_DEBUG, "in_mvn_host: v=%s, v_o=%s" % ( v.in_multivnode_host, v_o.in_multivnode_host)) pbs.logmsg(pbs.LOG_DEBUG, "jobs: v=%s, v_o=%s" % (v.jobs, v_o.jobs)) pbs.logmsg(pbs.LOG_DEBUG, "Mom: v=%s, v_o=%s" % (v.Mom, v_o.Mom)) pbs.logmsg(pbs.LOG_DEBUG, "ntype: v=%s, v_o=%s" % (hex(v.ntype), hex(v_o.ntype))) pbs.logmsg(pbs.LOG_DEBUG, "pcpus: v=%s, v_o=%s" % (v.pcpus, v_o.pcpus)) pbs.logmsg(pbs.LOG_DEBUG, "pnames: v=%s, v_o=%s" % (v.pnames, v_o.pnames)) pbs.logmsg(pbs.LOG_DEBUG, "Port: v=%s, v_o=%s" % (v.Port, v_o.Port)) pbs.logmsg(pbs.LOG_DEBUG, "Priority: v=%s, v_o=%s" % (v.Priority, v_o.Priority)) pbs.logmsg(pbs.LOG_DEBUG, "provision_enable: v=%s, v_o=%s" % ( v.provision_enable, v_o.provision_enable)) pbs.logmsg(pbs.LOG_DEBUG, "queue: v=%s, v_o=%s" % (v.queue, v_o.queue)) pbs.logmsg(pbs.LOG_DEBUG, "res_assigned: v=%s, v_o=%s" % ( v.resources_assigned, v_o.resources_assigned)) pbs.logmsg(pbs.LOG_DEBUG, "res_avail: v=%s, v_o=%s" % ( v.resources_available, v_o.resources_available)) pbs.logmsg(pbs.LOG_DEBUG, "resv: v=%s, v_o=%s" % (v.resv, v_o.resv)) pbs.logmsg(pbs.LOG_DEBUG, "resv_enable: v=%s, v_o=%s" % ( v.resv_enable, v_o.resv_enable)) pbs.logmsg(pbs.LOG_DEBUG, "sharing: v=%s, v_o=%s" % (v.sharing, v_o.sharing)) # sanity test some values if (lsct < lsct_o) or (lsct_o <= 0): e.reject("last_state_change_time: bad timestamp value") else: pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: good times") if (v.name != v_o.name) or (not v.name): e.reject( "name: vnode and vnode_o name values are null or mismatched") else: pbs.logmsg(pbs.LOG_DEBUG, "name: good names") if (isinstance(v.state, int)) and (isinstance(v_o.state, int)): pbs.logmsg(pbs.LOG_DEBUG, "state: good states") else: e.reject("state: bad state value") if len(v.extract_state_strs()) == len(v.extract_state_ints()): pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v sets") else: e.reject("state sets: bad v sets") if len(v_o.extract_state_strs()) == len(v_o.extract_state_ints()): pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v_o sets") else: e.reject("state sets: bad v_o sets") e.accept() except SystemExit: pass except: pbs.event().reject("%s hook failed with %s" % ( pbs.event().hook_name, sys.exc_info()[:2])) --- 2021-02-04 20:12:30,349 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:12:30,349 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:12:30,376 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:30,402 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:12:30,403 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbserrfy9h4 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbserrfy9h4 2021-02-04 20:12:30,427 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbserrfy9h4 Contents of /tmp/PtlPbserrfy9h4: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:12:31,496 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbserrfy9h4 2021-02-04 20:12:31,518 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:12:31,522 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:12:31,526 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:12:31,529 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:12:31,532 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:12:31,535 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:12:31,539 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:12:31,543 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:12:31,547 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:12:31,551 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:12:31,554 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:12:31,557 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK 2021-02-04 20:12:31,560 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:12:31,563 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK 2021-02-04 20:12:31,579 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:31,605 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:12:31,607 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsvxr6sku0 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsvxr6sku0 2021-02-04 20:12:31,631 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsvxr6sku0 Contents of /tmp/PtlPbsvxr6sku0: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:12:31,731 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsvxr6sku0 2021-02-04 20:12:31,769 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:31,798 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:12:32,946 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:12:34,108 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3 2021-02-04 20:12:35,256 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:12:35,262 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:12:35,265 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:12:35,268 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:12:35,271 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:12:35,273 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:12:35,276 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:12:35,278 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:12:35,281 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:12:35,284 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:12:35,288 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:12:35,292 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:12:35,294 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:12:35,297 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:12:35,316 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:35,340 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:12:35,340 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:35,351 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:12:35,362 INFO expect on server pbsdev-centos7-mky18-server: state ~ free|provisioning|offline|job-busy node pbsdev-centos7-mky18-mom1 ... OK 2021-02-04 20:12:35,378 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:35,404 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:12:35,406 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbs1j1dpaet pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbs1j1dpaet 2021-02-04 20:12:35,432 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbs1j1dpaet Contents of /tmp/PtlPbs1j1dpaet: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:12:36,502 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbs1j1dpaet 2021-02-04 20:12:36,536 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:12:36,562 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:12:36,563 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbs9t4d9bnu pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbs9t4d9bnu 2021-02-04 20:12:36,588 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbs9t4d9bnu Contents of /tmp/PtlPbs9t4d9bnu: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:12:36,688 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbs9t4d9bnu 2021-02-04 20:12:36,726 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:12:36,754 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:12:36,757 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:12:36,760 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:12:36,763 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:12:36,766 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:12:36,769 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:12:36,772 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:12:36,775 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:12:36,778 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:12:36,781 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:12:36,785 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:12:37,945 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - attempt 2 2021-02-04 20:12:39,102 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - attempt 3 2021-02-04 20:12:40,264 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:12:40,270 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence - No match 2021-02-04 20:12:41,427 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:12:41,434 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence - No match 2021-02-04 20:12:42,590 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK 2021-02-04 20:12:42,596 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence - No match 2021-02-04 20:12:43,756 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:12:43,764 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence - No match 2021-02-04 20:12:44,905 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK 2021-02-04 20:12:44,911 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:12:44,914 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:12:44,917 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:12:44,917 INFO manager on pbsdev-centos7-mky18-server: set node pbsdev-centos7-mky18-mom1 {'state': (2, 'offline')} 2021-02-04 20:12:44,918 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set node pbsdev-centos7-mky18-mom1 state+=offline 2021-02-04 20:12:44,938 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:12:44,941 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:12:44,944 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:12:44,947 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:12:44,950 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:12:44,955 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:12:44,960 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:12:44,964 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:12:44,968 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:12:44,974 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:12:44,978 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x1" - with existence... OK 2021-02-04 20:12:44,981 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_OFFLINE,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:12:44,985 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=1,409903" - with existence... OK 2021-02-04 20:12:44,986 INFO manager on pbsdev-centos7-mky18-server: set node pbsdev-centos7-mky18-mom1 {'state': (3, 'offline')} 2021-02-04 20:12:44,986 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set node pbsdev-centos7-mky18-mom1 state-=offline 2021-02-04 20:12:45,005 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:12:45,009 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:12:45,012 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:12:45,016 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:12:45,020 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:12:45,023 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:12:45,026 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:12:45,030 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:12:45,033 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:12:45,036 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:12:45,040 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:12:45,044 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:12:45,047 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:12:45,048 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rsub -R 202102042013.00 -E 202102042013.01 --hosts pbsdev-centos7-mky18-mom1 2021-02-04 20:12:45,064 INFO submit to pbsdev-centos7-mky18-server as root: reservation M12.pbsdev-centos7-mky18-server CONFIRMED OrderedDict([('reserve_start', 1612469580.0479345), ('reserve_end', 1612469581.0479345)]) 2021-02-04 20:12:45,068 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:12:46,223 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:12:47,388 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3 2021-02-04 20:12:48,570 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 4 2021-02-04 20:12:49,739 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 5 2021-02-04 20:12:50,903 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 6 2021-02-04 20:12:52,070 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 7 2021-02-04 20:12:53,196 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 8 2021-02-04 20:12:54,364 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 9 2021-02-04 20:12:55,516 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 10 2021-02-04 20:12:56,666 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 11 2021-02-04 20:12:57,833 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 12 2021-02-04 20:12:59,001 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 13 2021-02-04 20:13:00,164 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 14 2021-02-04 20:13:01,325 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:01,333 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:01,336 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:01,339 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:01,341 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:01,344 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:01,347 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:01,351 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:01,355 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:01,359 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:01,362 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2000" - with existence - No match 2021-02-04 20:13:02,523 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2000" - with existence... OK 2021-02-04 20:13:02,531 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_RESV_EXCLUSIVE" - with existence - No match 2021-02-04 20:13:03,700 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_RESV_EXCLUSIVE" - with existence... OK 2021-02-04 20:13:03,708 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=8192" - with existence - No match 2021-02-04 20:13:04,869 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=8192" - with existence... OK 2021-02-04 20:13:04,875 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:04,878 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:04,881 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:04,881 INFO checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom1 start=1612469550.3600194 end=1612469584.8817654 2021-02-04 20:13:04,920 INFO server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom1" - on all matches - with existence... OK 2021-02-04 20:13:04,938 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:13:04,964 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:13:04,966 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbslhgi7vz4 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbslhgi7vz4 2021-02-04 20:13:04,991 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbslhgi7vz4 Contents of /tmp/PtlPbslhgi7vz4: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:13:06,059 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbslhgi7vz4 2021-02-04 20:13:06,079 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:06,082 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:06,085 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:06,089 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:06,092 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:06,096 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:06,099 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:06,102 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:06,107 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:06,113 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:06,117 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:13:06,120 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK 2021-02-04 20:13:06,122 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:13:06,125 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK 2021-02-04 20:13:06,144 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:13:06,169 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:13:06,171 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsyxev3kdw pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbsyxev3kdw 2021-02-04 20:13:06,197 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbsyxev3kdw Contents of /tmp/PtlPbsyxev3kdw: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:13:06,296 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsyxev3kdw 2021-02-04 20:13:06,334 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:06,363 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:13:07,531 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:13:08,709 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3 2021-02-04 20:13:09,876 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:09,885 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:09,888 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:09,890 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:09,893 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:09,895 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:09,897 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:09,900 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:09,904 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:09,907 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:09,911 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:13:09,914 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:09,917 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:09,920 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:09,942 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:09,968 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:09,968 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:09,981 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:09,991 INFO expect on server pbsdev-centos7-mky18-server: state ~ free|provisioning|offline|job-busy node pbsdev-centos7-mky18-mom2 ... OK 2021-02-04 20:13:10,007 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:13:10,032 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:13:10,034 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbshtz05k_6 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbshtz05k_6 2021-02-04 20:13:10,057 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbshtz05k_6 Contents of /tmp/PtlPbshtz05k_6: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:13:11,123 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbshtz05k_6 2021-02-04 20:13:11,158 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:13:11,184 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:13:11,186 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsb6jgrid9 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbsb6jgrid9 2021-02-04 20:13:11,211 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbsb6jgrid9 Contents of /tmp/PtlPbsb6jgrid9: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:13:11,308 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsb6jgrid9 2021-02-04 20:13:11,344 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:11,372 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:11,375 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:11,378 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:11,380 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:11,383 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:11,387 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:11,391 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:11,395 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:11,399 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:11,402 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:11,406 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:13:12,566 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - attempt 2 2021-02-04 20:13:13,733 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - attempt 3 2021-02-04 20:13:14,888 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:13:14,897 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence - No match 2021-02-04 20:13:16,069 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:13:16,075 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence - No match 2021-02-04 20:13:17,232 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK 2021-02-04 20:13:17,239 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence - No match 2021-02-04 20:13:18,406 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:13:18,413 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence - No match 2021-02-04 20:13:19,578 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK 2021-02-04 20:13:19,584 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:19,587 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:19,590 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:19,590 INFO manager on pbsdev-centos7-mky18-server: set node pbsdev-centos7-mky18-mom2 {'state': (2, 'offline')} 2021-02-04 20:13:19,590 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set node pbsdev-centos7-mky18-mom2 state+=offline 2021-02-04 20:13:19,607 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:19,610 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:19,613 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:19,616 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:19,620 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:19,624 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:19,627 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:19,630 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:19,634 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:19,638 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:19,641 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x1" - with existence... OK 2021-02-04 20:13:19,644 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_OFFLINE,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:13:19,647 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=1,409903" - with existence... OK 2021-02-04 20:13:19,647 INFO manager on pbsdev-centos7-mky18-server: set node pbsdev-centos7-mky18-mom2 {'state': (3, 'offline')} 2021-02-04 20:13:19,647 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set node pbsdev-centos7-mky18-mom2 state-=offline 2021-02-04 20:13:19,666 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:19,670 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:19,673 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:19,676 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:19,679 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:19,682 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:19,687 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:19,690 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:19,694 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:19,697 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:19,700 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:19,703 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:19,706 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:19,707 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rsub -R 202102042013.34 -E 202102042013.35 --hosts pbsdev-centos7-mky18-mom2 2021-02-04 20:13:19,722 INFO submit to pbsdev-centos7-mky18-server as root: reservation M13.pbsdev-centos7-mky18-server CONFIRMED OrderedDict([('reserve_start', 1612469614.7069578), ('reserve_end', 1612469615.7069578)]) 2021-02-04 20:13:19,726 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:13:20,897 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:13:22,071 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3 2021-02-04 20:13:23,209 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 4 2021-02-04 20:13:24,391 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 5 2021-02-04 20:13:25,564 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 6 2021-02-04 20:13:26,730 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 7 2021-02-04 20:13:27,904 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 8 2021-02-04 20:13:29,072 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 9 2021-02-04 20:13:30,240 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 10 2021-02-04 20:13:31,422 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 11 2021-02-04 20:13:32,590 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 12 2021-02-04 20:13:33,770 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 13 2021-02-04 20:13:34,953 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 14 2021-02-04 20:13:36,138 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:36,146 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:36,148 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:36,151 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:36,153 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:36,156 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:36,159 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:36,162 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:36,165 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:36,169 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:36,172 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2000" - with existence - No match 2021-02-04 20:13:37,334 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2000" - with existence... OK 2021-02-04 20:13:37,342 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_RESV_EXCLUSIVE" - with existence - No match 2021-02-04 20:13:38,515 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_RESV_EXCLUSIVE" - with existence... OK 2021-02-04 20:13:38,522 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=8192" - with existence - No match 2021-02-04 20:13:39,689 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=8192" - with existence... OK 2021-02-04 20:13:39,695 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:39,698 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:39,700 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:39,701 INFO checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom2 start=1612469584.9210525 end=1612469619.7010572 2021-02-04 20:13:39,737 INFO server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom2" - on all matches - with existence... OK 2021-02-04 20:13:39,738 INFO ================================================ 2021-02-04 20:13:39,738 INFO Entered TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:13:39,739 INFO ================================================ 2021-02-04 20:13:39,739 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:39,749 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:13:39,749 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:39,964 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:13:39,965 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:13:39,965 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:13:39,976 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:39,986 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:39,998 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:40,014 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:40,026 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:40,036 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:13:40,037 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:13:40,050 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:40,061 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:40,072 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:40,084 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:40,101 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:40,112 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:40,123 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:13:40,124 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:13:40,124 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:40,134 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:13:40,134 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:13:40,134 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:13:40,147 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:40,159 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:40,169 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:40,181 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:40,198 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:40,209 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:40,221 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:13:40,221 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:13:40,221 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:13:40,232 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:40,243 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:40,255 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:40,271 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:40,283 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:40,294 INFO ================================================= 2021-02-04 20:13:40,294 INFO Completed TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:13:40,294 INFO ================================================= 2021-02-04 20:13:40,295 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:13:40,349 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:13:40,407 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:13:40,461 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:13:40,520 INFO ok 2021-02-04 20:13:40,521 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo 2021-02-04 20:13:40,525 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs 2021-02-04 20:13:40,530 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:13:40,542 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:13:40,554 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:13:40,568 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:13:40,581 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:13:40,595 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:13:40,608 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:13:40,620 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:13:40,634 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:13:40,645 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:13:40,657 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:13:40,671 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:13:40,682 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:13:40,696 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:13:40,708 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:13:40,719 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:13:40,732 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:13:40,744 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:13:40,755 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:13:40,770 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:13:40,781 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:13:40,795 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:13:40,807 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:13:40,817 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo 2021-02-04 20:13:40,852 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs 2021-02-04 20:13:40,888 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:13:40,888 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:13:40,910 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:13:40,938 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:13:40,938 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:13:40,955 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:13:40,983 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:13:40,983 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:13:41,002 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:13:41,027 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:13:41,053 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:13:41,080 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:13:41,106 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:13:41,131 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:13:41,156 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:13:41,182 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:13:41,209 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:13:41,234 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:13:41,261 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:13:41,286 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:13:41,312 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:13:41,337 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:13:41,362 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:13:41,387 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:13:41,414 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:13:41,439 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:13:41,464 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:13:41,490 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:13:41,516 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:13:41,542 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo 2021-02-04 20:13:41,575 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs 2021-02-04 20:13:41,609 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:13:41,609 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:13:41,627 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:13:41,651 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:13:41,652 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:13:41,670 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:13:41,695 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:13:41,695 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:13:41,714 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:13:41,739 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:13:41,765 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:13:41,790 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:13:41,815 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:13:41,841 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:13:41,865 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:13:41,890 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:13:41,916 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:13:41,942 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:13:41,968 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:13:41,993 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:13:42,018 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:13:42,044 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:13:42,070 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:13:42,096 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:13:42,121 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:13:42,147 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:13:42,173 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:13:42,198 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:13:42,224 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:13:42,250 INFO test name: test_hook_state_changes_01 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)... 2021-02-04 20:13:42,251 INFO test start time: Thu Feb 4 20:13:42 2021 2021-02-04 20:13:42,251 INFO test docstring: Test each mom for state changes: 1. sigkill the mom; check server log for expected messages 2. start the mom; check server log for expected messages 3. verify the chain of reported current/previous states is unbroken 2021-02-04 20:13:42,251 INFO ============================================= 2021-02-04 20:13:42,251 INFO Entered TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:13:42,251 INFO ============================================= 2021-02-04 20:13:42,252 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:42,262 INFO status on pbsdev-centos7-mky18-server: server 2021-02-04 20:13:42,262 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:42,478 INFO manager on pbsdev-centos7-mky18-server: unset server managers 2021-02-04 20:13:42,478 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2021-02-04 20:13:42,498 INFO manager on pbsdev-centos7-mky18-server: unset server operators 2021-02-04 20:13:42,499 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators 2021-02-04 20:13:42,519 INFO status on pbsdev-centos7-mky18-server: server ['managers', 'operators'] 2021-02-04 20:13:42,519 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:42,736 INFO manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')} 2021-02-04 20:13:42,736 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@* 2021-02-04 20:13:42,756 INFO server pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:13:42,756 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:42,970 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:13:42,970 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:43,183 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:13:43,183 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:13:43,183 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:13:43,194 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:43,204 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:43,217 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:43,237 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:43,249 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:43,260 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:13:43,260 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:13:43,274 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:43,284 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:43,297 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:43,309 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:43,325 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:43,338 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:43,351 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:13:43,351 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:13:43,351 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:43,566 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:13:43,566 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:13:43,567 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:13:43,579 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:43,588 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:43,598 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:43,611 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:43,628 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:43,639 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:43,650 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:13:43,650 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:13:43,650 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:13:43,660 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:43,876 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:13:43,877 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:44,092 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:13:44,092 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:13:44,092 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:13:44,102 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:44,112 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:44,124 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:44,141 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:44,152 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:44,165 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:13:44,165 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:13:44,178 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:44,190 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:44,201 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:44,214 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:44,232 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:44,243 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:44,255 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:13:44,256 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:13:44,256 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:44,475 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:13:44,476 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:13:44,476 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:13:44,490 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:44,499 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:44,510 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:44,522 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:44,539 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:44,550 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:44,562 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:13:44,562 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:13:44,562 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:13:44,572 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:44,785 INFO manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision', 'sync_mom_hookfiles_timeout'] 2021-02-04 20:13:44,785 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision,sync_mom_hookfiles_timeout 2021-02-04 20:13:44,799 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:13:44,818 INFO manager on pbsdev-centos7-mky18-server: delete hook m1234 2021-02-04 20:13:44,818 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c delete hook m1234 2021-02-04 20:13:44,837 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:45,051 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:13:45,051 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:13:45,062 INFO manager on pbsdev-centos7-mky18-server: delete queue workq 2021-02-04 20:13:45,062 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq 2021-02-04 20:13:45,073 INFO manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'} 2021-02-04 20:13:45,073 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True 2021-02-04 20:13:45,084 INFO status on pbsdev-centos7-mky18-server: queue workq 2021-02-04 20:13:45,085 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:45,304 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:13:45,304 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:13:45,316 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:45,327 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:45,337 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:45,354 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:45,365 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:45,377 INFO manager on pbsdev-centos7-mky18-server: delete node @default 2021-02-04 20:13:45,378 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default 2021-02-04 20:13:45,390 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:13:45,409 INFO manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'} 2021-02-04 20:13:45,409 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false 2021-02-04 20:13:45,429 INFO manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'} 2021-02-04 20:13:45,430 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq 2021-02-04 20:13:45,441 INFO status on pbsdev-centos7-mky18-server: resource 2021-02-04 20:13:45,441 INFO manager on pbsdev-centos7-mky18-server: list resource 2021-02-04 20:13:45,442 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource 2021-02-04 20:13:45,451 INFOCLI status on pbsdev-centos7-mky18-server: server license_count 2021-02-04 20:13:45,452 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:45,664 INFO server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed 2021-02-04 20:13:45,664 INFO status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:45,664 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:45,917 ERROR Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value 2021-02-04 20:13:45,922 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:13:45,934 INFO scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:13:45,935 INFO manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm'] 2021-02-04 20:13:45,935 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm 2021-02-04 20:13:45,949 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:45,963 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbs4sr0et08 /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:45,975 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:45,985 INFO scheduler pbsdev-centos7-mky18-server: reverting holidays file to default 2021-02-04 20:13:45,985 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:45,998 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2021-02-04 20:13:46,010 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:46,020 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:46,035 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:46,047 INFO Scheduler sent signal -HUP 2021-02-04 20:13:46,047 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:13:46,060 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148 2021-02-04 20:13:46,072 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default 2021-02-04 20:13:46,100 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:13:46,112 INFO status on pbsdev-centos7-mky18-server: sched default 2021-02-04 20:13:46,112 INFO manager on pbsdev-centos7-mky18-server: list sched default 2021-02-04 20:13:46,112 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:46,122 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:46,137 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:46,148 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:46,164 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:46,176 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:46,206 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:46,231 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,231 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,241 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:13:46,241 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:13:46,266 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:13:46,291 INFOCLI pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:13:46,379 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:13:46,432 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbskz8kqrue pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmpg3jb7fkvPtlPbstmpcopy 2021-02-04 20:13:46,456 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpg3jb7fkvPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:13:46,482 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpg3jb7fkvPtlPbstmpcopy 2021-02-04 20:13:46,499 INFO MoM sent signal -HUP 2021-02-04 20:13:46,499 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:46,524 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 14425 2021-02-04 20:13:46,570 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:46,597 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,598 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,608 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:13:46,608 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,608 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,621 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,622 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,633 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:46,644 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 got: state = state-unknown,down 2021-02-04 20:13:47,647 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:47,660 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2 got: state = state-unknown,down 2021-02-04 20:13:48,662 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:48,674 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ... OK 2021-02-04 20:13:48,674 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:13:48,674 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:13:48,704 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:48,732 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:48,732 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:48,742 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:13:48,742 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:13:48,768 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:13:48,793 INFOCLI pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:13:48,881 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:13:48,933 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbsw333id_y pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpi3fuo_1iPtlPbstmpcopy 2021-02-04 20:13:48,957 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpi3fuo_1iPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:13:48,983 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpi3fuo_1iPtlPbstmpcopy 2021-02-04 20:13:49,000 INFO MoM sent signal -HUP 2021-02-04 20:13:49,001 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:49,026 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 14375 2021-02-04 20:13:49,070 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:49,095 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,095 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,105 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:13:49,105 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,106 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,120 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,120 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,133 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:49,143 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 got: state = state-unknown,down 2021-02-04 20:13:50,146 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:50,161 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2 got: state = state-unknown,down 2021-02-04 20:13:51,163 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:51,173 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ... OK 2021-02-04 20:13:51,173 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:13:51,173 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:13:51,185 INFO manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'} 2021-02-04 20:13:51,185 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off 2021-02-04 20:13:51,204 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2021-02-04 20:13:51,215 INFO ============================================== 2021-02-04 20:13:51,216 INFO Completed TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:13:51,216 INFO ============================================== 2021-02-04 20:13:51,216 INFO manager on pbsdev-centos7-mky18-server: set server {'log_events': 4095} 2021-02-04 20:13:51,216 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server log_events=4095 2021-02-04 20:13:51,228 INFO status on pbsdev-centos7-mky18-server: hook 2021-02-04 20:13:51,229 INFO manager on pbsdev-centos7-mky18-server: list hook 2021-02-04 20:13:51,229 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:13:51,248 INFO manager on pbsdev-centos7-mky18-server: create hook p1234 2021-02-04 20:13:51,249 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c create hook p1234 2021-02-04 20:13:51,269 INFO status on pbsdev-centos7-mky18-server: hook p1234 2021-02-04 20:13:51,269 INFO manager on pbsdev-centos7-mky18-server: list hook p1234 2021-02-04 20:13:51,269 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook p1234 2021-02-04 20:13:51,287 INFO status on pbsdev-centos7-mky18-server: hook p1234 2021-02-04 20:13:51,288 INFO manager on pbsdev-centos7-mky18-server: list hook p1234 2021-02-04 20:13:51,288 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook p1234 2021-02-04 20:13:51,307 INFO manager on pbsdev-centos7-mky18-server: set hook p1234 {'event': 'modifyvnode', 'enabled': 'True', 'debug': 'True'} 2021-02-04 20:13:51,308 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook p1234 event=modifyvnode,enabled=True,debug=True 2021-02-04 20:13:51,329 INFO status on pbsdev-centos7-mky18-server: server sync_mom_hookfiles_timeout 2021-02-04 20:13:51,329 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:51,339 INFO Setting sync_mom_hookfiles_timeout to 15s 2021-02-04 20:13:51,339 INFO manager on pbsdev-centos7-mky18-server: set server {'sync_mom_hookfiles_timeout': 15} 2021-02-04 20:13:51,339 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server sync_mom_hookfiles_timeout=15 2021-02-04 20:13:51,350 INFO manager on pbsdev-centos7-mky18-server: import hook p1234 {'content-type': 'application/x-python', 'content-encoding': 'default', 'input-file': '/tmp/PtlPbssedlvx_j'} 2021-02-04 20:13:51,350 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c import hook p1234 application/x-python default /tmp/PtlPbssedlvx_j 2021-02-04 20:13:51,373 INFO server pbsdev-centos7-mky18-server: imported hook body --- import pbs import sys try: e = pbs.event() v = e.vnode v_o = e.vnode_o lsct = v.last_state_change_time lsct_o = v_o.last_state_change_time state_str_buf_v = ",".join(v.extract_state_strs()) state_str_buf_v_o = ",".join(v_o.extract_state_strs()) state_int_buf_v = ','.join([str(_) for _ in v.extract_state_ints()]) state_int_buf_v_o = ','.join( [str(_) for _ in v_o.extract_state_ints()]) # print show_vnode_state record (bi consumable) svs1_data = "v.state_hex=%s v_o.state_hex=%s v.state_strs=%s " "v_o.state_strs=%s" % (hex(v.state), hex(v_o.state), state_str_buf_v, state_str_buf_v_o) svs2_data = "v.state_ints=%s v_o.state_ints=%s v.lsct=%s " "v_o.lsct=%s" % (state_int_buf_v, state_int_buf_v_o, str(lsct), str(lsct_o)) svs_data = "%s %s" % (svs1_data, svs2_data) pbs.logmsg(pbs.LOG_DEBUG, "show_vnode_state;name=%s %s" % (v.name, svs_data)) # print additional hook parameter values pbs.logmsg(pbs.LOG_DEBUG, "name: v=%s, v_o=%s" % (v.name, v_o.name)) pbs.logmsg(pbs.LOG_DEBUG, "state: v=%s, v_o=%s" % (hex(v.state), hex(v_o.state))) pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: v=%s, v_o=%s" % ( str(lsct), str(lsct_o))) pbs.logmsg(pbs.LOG_DEBUG, "comment: v=%s, v_o=%s" % (v.comment, v_o.comment)) pbs.logmsg(pbs.LOG_DEBUG, "aoe: v=%s, v_o=%s" % (v.current_aoe, v_o.current_aoe)) pbs.logmsg(pbs.LOG_DEBUG, "in_mvn_host: v=%s, v_o=%s" % ( v.in_multivnode_host, v_o.in_multivnode_host)) pbs.logmsg(pbs.LOG_DEBUG, "jobs: v=%s, v_o=%s" % (v.jobs, v_o.jobs)) pbs.logmsg(pbs.LOG_DEBUG, "Mom: v=%s, v_o=%s" % (v.Mom, v_o.Mom)) pbs.logmsg(pbs.LOG_DEBUG, "ntype: v=%s, v_o=%s" % (hex(v.ntype), hex(v_o.ntype))) pbs.logmsg(pbs.LOG_DEBUG, "pcpus: v=%s, v_o=%s" % (v.pcpus, v_o.pcpus)) pbs.logmsg(pbs.LOG_DEBUG, "pnames: v=%s, v_o=%s" % (v.pnames, v_o.pnames)) pbs.logmsg(pbs.LOG_DEBUG, "Port: v=%s, v_o=%s" % (v.Port, v_o.Port)) pbs.logmsg(pbs.LOG_DEBUG, "Priority: v=%s, v_o=%s" % (v.Priority, v_o.Priority)) pbs.logmsg(pbs.LOG_DEBUG, "provision_enable: v=%s, v_o=%s" % ( v.provision_enable, v_o.provision_enable)) pbs.logmsg(pbs.LOG_DEBUG, "queue: v=%s, v_o=%s" % (v.queue, v_o.queue)) pbs.logmsg(pbs.LOG_DEBUG, "res_assigned: v=%s, v_o=%s" % ( v.resources_assigned, v_o.resources_assigned)) pbs.logmsg(pbs.LOG_DEBUG, "res_avail: v=%s, v_o=%s" % ( v.resources_available, v_o.resources_available)) pbs.logmsg(pbs.LOG_DEBUG, "resv: v=%s, v_o=%s" % (v.resv, v_o.resv)) pbs.logmsg(pbs.LOG_DEBUG, "resv_enable: v=%s, v_o=%s" % ( v.resv_enable, v_o.resv_enable)) pbs.logmsg(pbs.LOG_DEBUG, "sharing: v=%s, v_o=%s" % (v.sharing, v_o.sharing)) # sanity test some values if (lsct < lsct_o) or (lsct_o <= 0): e.reject("last_state_change_time: bad timestamp value") else: pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: good times") if (v.name != v_o.name) or (not v.name): e.reject( "name: vnode and vnode_o name values are null or mismatched") else: pbs.logmsg(pbs.LOG_DEBUG, "name: good names") if (isinstance(v.state, int)) and (isinstance(v_o.state, int)): pbs.logmsg(pbs.LOG_DEBUG, "state: good states") else: e.reject("state: bad state value") if len(v.extract_state_strs()) == len(v.extract_state_ints()): pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v sets") else: e.reject("state sets: bad v sets") if len(v_o.extract_state_strs()) == len(v_o.extract_state_ints()): pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v_o sets") else: e.reject("state sets: bad v_o sets") e.accept() except SystemExit: pass except: pbs.event().reject("%s hook failed with %s" % ( pbs.event().hook_name, sys.exc_info()[:2])) --- 2021-02-04 20:13:51,373 INFO MoM sent signal -KILL 2021-02-04 20:13:51,373 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:51,400 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -KILL 14425 2021-02-04 20:13:51,431 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:51,435 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:51,438 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:51,443 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:51,446 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:51,450 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:51,454 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:51,458 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:51,462 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:51,466 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:51,469 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:13:51,472 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK 2021-02-04 20:13:51,477 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:13:51,480 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK 2021-02-04 20:13:51,498 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:13:51,524 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:13:51,526 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsu80m4bot pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsu80m4bot 2021-02-04 20:13:51,554 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsu80m4bot Contents of /tmp/PtlPbsu80m4bot: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:13:51,666 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsu80m4bot 2021-02-04 20:13:51,707 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:51,744 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:13:52,890 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:13:54,074 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3 2021-02-04 20:13:55,265 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:55,271 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:55,275 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:55,277 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:55,280 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:55,282 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:55,284 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:55,286 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:55,289 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:55,292 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:55,296 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:13:55,298 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:55,301 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:55,303 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:55,303 INFO checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom1 start=1612469631.3735516 end=1612469635.3038225 2021-02-04 20:13:55,335 INFO server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom1" - on all matches - with existence... OK 2021-02-04 20:13:55,336 INFO MoM sent signal -KILL 2021-02-04 20:13:55,336 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:55,364 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -KILL 14375 2021-02-04 20:13:55,393 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:55,396 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:55,399 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:55,401 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:55,404 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:55,407 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:55,410 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:55,413 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:55,417 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:55,421 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:55,424 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:13:55,427 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK 2021-02-04 20:13:55,430 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK 2021-02-04 20:13:55,433 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK 2021-02-04 20:13:55,449 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:13:55,474 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:13:55,475 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsrzwtcmwx pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbsrzwtcmwx 2021-02-04 20:13:55,500 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbsrzwtcmwx Contents of /tmp/PtlPbsrzwtcmwx: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:13:55,607 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsrzwtcmwx 2021-02-04 20:13:55,644 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:13:55,674 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:13:56,867 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:13:58,046 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3 2021-02-04 20:13:59,233 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:13:59,239 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:13:59,242 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:13:59,245 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:13:59,248 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:13:59,250 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:13:59,252 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:13:59,255 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:13:59,259 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:13:59,262 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:13:59,265 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:13:59,268 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:13:59,271 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:13:59,274 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:13:59,275 INFO checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom2 start=1612469635.3362179 end=1612469639.2750354 2021-02-04 20:13:59,310 INFO server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom2" - on all matches - with existence... OK 2021-02-04 20:13:59,310 INFO ================================================ 2021-02-04 20:13:59,310 INFO Entered TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:13:59,310 INFO ================================================ 2021-02-04 20:13:59,311 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:59,321 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:13:59,321 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:59,536 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:13:59,536 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:13:59,537 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:13:59,548 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:59,559 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:59,570 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:59,585 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:59,596 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:59,607 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:13:59,608 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:13:59,619 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:59,630 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:59,641 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:59,653 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:59,672 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:59,683 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:13:59,694 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:13:59,694 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:13:59,694 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:13:59,910 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:13:59,911 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:13:59,911 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:13:59,925 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:13:59,936 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:13:59,947 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:13:59,960 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:13:59,979 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:13:59,990 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:00,003 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:14:00,003 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:14:00,004 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:14:00,013 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:00,026 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:00,038 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:00,056 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:00,067 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:00,078 INFO ================================================= 2021-02-04 20:14:00,079 INFO Completed TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:14:00,079 INFO ================================================= 2021-02-04 20:14:00,079 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:14:00,137 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:14:00,192 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:14:00,247 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:14:00,303 INFO ok 2021-02-04 20:14:00,303 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo 2021-02-04 20:14:00,308 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs 2021-02-04 20:14:00,312 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:14:00,324 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:14:00,335 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:14:00,346 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:14:00,358 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:14:00,369 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:14:00,380 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:14:00,392 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:14:00,403 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:14:00,414 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:14:00,426 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:14:00,436 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:14:00,448 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:14:00,460 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:14:00,471 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:14:00,483 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:14:00,494 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:14:00,505 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:14:00,517 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:14:00,528 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:14:00,539 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:14:00,551 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:14:00,562 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:14:00,574 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo 2021-02-04 20:14:00,610 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs 2021-02-04 20:14:00,647 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:14:00,647 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:14:00,664 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:14:00,691 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:14:00,691 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:14:00,708 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:14:00,733 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:14:00,733 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:14:00,751 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:14:00,776 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:14:00,803 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:14:00,829 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:14:00,854 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:14:00,879 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:14:00,904 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:14:00,930 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:14:00,956 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:14:00,981 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:14:01,007 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:14:01,033 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:14:01,059 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:14:01,084 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:14:01,110 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:14:01,135 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:14:01,161 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:14:01,187 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:14:01,213 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:14:01,237 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:14:01,263 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:14:01,289 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo 2021-02-04 20:14:01,322 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs 2021-02-04 20:14:01,358 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:14:01,358 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:14:01,376 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:14:01,402 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:14:01,402 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:14:01,422 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:14:01,449 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:14:01,449 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:14:01,467 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:14:01,494 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:14:01,522 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:14:01,546 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:14:01,571 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:14:01,598 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:14:01,626 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:14:01,653 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:14:01,679 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:14:01,705 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:14:01,732 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:14:01,759 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:14:01,786 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:14:01,812 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:14:01,840 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:14:01,867 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:14:01,894 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:14:01,922 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:14:01,947 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:14:01,973 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:14:02,001 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:14:02,028 INFO test name: test_hook_state_changes_02 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)... 2021-02-04 20:14:02,028 INFO test start time: Thu Feb 4 20:14:02 2021 2021-02-04 20:14:02,029 INFO test docstring: Test: stop and start the pbs server; look for proper log messages 2021-02-04 20:14:02,029 INFO ============================================= 2021-02-04 20:14:02,029 INFO Entered TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:14:02,029 INFO ============================================= 2021-02-04 20:14:02,029 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:02,039 INFO status on pbsdev-centos7-mky18-server: server 2021-02-04 20:14:02,039 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:02,256 INFO manager on pbsdev-centos7-mky18-server: unset server managers 2021-02-04 20:14:02,256 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2021-02-04 20:14:02,277 INFO manager on pbsdev-centos7-mky18-server: unset server operators 2021-02-04 20:14:02,278 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators 2021-02-04 20:14:02,297 INFO status on pbsdev-centos7-mky18-server: server ['managers', 'operators'] 2021-02-04 20:14:02,298 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:02,512 INFO manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')} 2021-02-04 20:14:02,512 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@* 2021-02-04 20:14:02,533 INFO server pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:14:02,533 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:02,749 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:14:02,749 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:02,967 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:14:02,967 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:14:02,967 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:14:02,981 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:02,991 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:03,003 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:03,019 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:03,031 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:03,043 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:14:03,043 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:14:03,056 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:03,066 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:03,076 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:03,088 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:03,105 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:03,116 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:03,127 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:14:03,128 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:14:03,128 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:03,138 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:14:03,139 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:14:03,139 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:14:03,151 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:03,162 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:03,173 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:03,185 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:03,201 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:03,213 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:03,224 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:14:03,224 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:14:03,224 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:14:03,234 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:03,449 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:14:03,450 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:03,663 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:14:03,663 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:14:03,663 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:14:03,673 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:03,683 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:03,695 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:03,712 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:03,723 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:03,734 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:14:03,734 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:14:03,746 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:03,756 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:03,767 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:03,779 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:03,794 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:03,806 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:03,817 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:14:03,817 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:14:03,817 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:04,036 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:14:04,037 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:14:04,037 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:14:04,053 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:04,064 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:04,075 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:04,088 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:04,105 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:04,115 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:04,127 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:14:04,127 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:14:04,128 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:14:04,140 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:04,355 INFO manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision', 'sync_mom_hookfiles_timeout'] 2021-02-04 20:14:04,356 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision,sync_mom_hookfiles_timeout 2021-02-04 20:14:04,367 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:14:04,386 INFO manager on pbsdev-centos7-mky18-server: delete hook p1234 2021-02-04 20:14:04,386 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c delete hook p1234 2021-02-04 20:14:04,405 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:04,619 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:14:04,619 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:14:04,631 INFO manager on pbsdev-centos7-mky18-server: delete queue workq 2021-02-04 20:14:04,631 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq 2021-02-04 20:14:04,643 INFO manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'} 2021-02-04 20:14:04,643 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True 2021-02-04 20:14:04,655 INFO status on pbsdev-centos7-mky18-server: queue workq 2021-02-04 20:14:04,655 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:04,868 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:14:04,869 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:14:04,878 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:04,888 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:04,901 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:04,919 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:04,932 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:04,943 INFO manager on pbsdev-centos7-mky18-server: delete node @default 2021-02-04 20:14:04,943 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default 2021-02-04 20:14:04,959 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:14:04,980 INFO manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'} 2021-02-04 20:14:04,980 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false 2021-02-04 20:14:05,002 INFO manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'} 2021-02-04 20:14:05,002 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq 2021-02-04 20:14:05,013 INFO status on pbsdev-centos7-mky18-server: resource 2021-02-04 20:14:05,013 INFO manager on pbsdev-centos7-mky18-server: list resource 2021-02-04 20:14:05,013 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource 2021-02-04 20:14:05,024 INFOCLI status on pbsdev-centos7-mky18-server: server license_count 2021-02-04 20:14:05,024 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:05,035 INFO server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed 2021-02-04 20:14:05,035 INFO status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:05,035 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:05,287 ERROR Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value 2021-02-04 20:14:05,292 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:14:05,301 INFO scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:14:05,301 INFO manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm'] 2021-02-04 20:14:05,302 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm 2021-02-04 20:14:05,315 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:05,326 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbscbboyk47 /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:05,337 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:05,348 INFO scheduler pbsdev-centos7-mky18-server: reverting holidays file to default 2021-02-04 20:14:05,348 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:05,358 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2021-02-04 20:14:05,369 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:05,380 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:05,391 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:05,402 INFO Scheduler sent signal -HUP 2021-02-04 20:14:05,403 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:14:05,414 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148 2021-02-04 20:14:05,425 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default 2021-02-04 20:14:05,451 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:14:05,461 INFO status on pbsdev-centos7-mky18-server: sched default 2021-02-04 20:14:05,461 INFO manager on pbsdev-centos7-mky18-server: list sched default 2021-02-04 20:14:05,461 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:05,471 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:05,483 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:05,493 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:05,510 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:05,521 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:05,553 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:14:05,577 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,578 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,587 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:14:05,588 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:14:05,613 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:14:05,637 INFOCLI pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:14:05,725 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:14:05,780 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbs7hkjh6h0 pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmp3hs1a483PtlPbstmpcopy 2021-02-04 20:14:05,804 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmp3hs1a483PtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:14:05,831 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmp3hs1a483PtlPbstmpcopy 2021-02-04 20:14:05,847 INFO MoM sent signal -HUP 2021-02-04 20:14:05,848 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:14:05,873 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 14917 2021-02-04 20:14:05,918 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:14:05,943 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,943 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,953 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:14:05,954 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,954 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,968 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,968 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,979 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:05,989 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 got: state = state-unknown,down 2021-02-04 20:14:06,992 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:07,006 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2 got: state = state-unknown,down 2021-02-04 20:14:08,007 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:08,020 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ... OK 2021-02-04 20:14:08,020 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:14:08,021 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:14:08,051 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:14:08,077 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,077 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,091 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:14:08,092 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:14:08,118 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:14:08,145 INFOCLI pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:14:08,238 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:14:08,293 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbsmi5k5d2p pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpj0lt9orwPtlPbstmpcopy 2021-02-04 20:14:08,319 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpj0lt9orwPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:14:08,347 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpj0lt9orwPtlPbstmpcopy 2021-02-04 20:14:08,363 INFO MoM sent signal -HUP 2021-02-04 20:14:08,364 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:14:08,392 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 14867 2021-02-04 20:14:08,438 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:14:08,463 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,464 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,474 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:14:08,474 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,474 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,489 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,490 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,500 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:08,510 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 got: state = state-unknown,down 2021-02-04 20:14:09,515 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:09,529 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2 got: state = state-unknown,down 2021-02-04 20:14:10,530 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:10,542 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ... OK 2021-02-04 20:14:10,542 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:14:10,542 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:14:10,552 INFO manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'} 2021-02-04 20:14:10,552 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off 2021-02-04 20:14:10,571 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2021-02-04 20:14:10,581 INFO ============================================== 2021-02-04 20:14:10,581 INFO Completed TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:14:10,581 INFO ============================================== 2021-02-04 20:14:10,581 INFO manager on pbsdev-centos7-mky18-server: set server {'log_events': 4095} 2021-02-04 20:14:10,581 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server log_events=4095 2021-02-04 20:14:10,593 INFO status on pbsdev-centos7-mky18-server: hook 2021-02-04 20:14:10,593 INFO manager on pbsdev-centos7-mky18-server: list hook 2021-02-04 20:14:10,593 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:14:10,611 INFO manager on pbsdev-centos7-mky18-server: create hook s1234 2021-02-04 20:14:10,611 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c create hook s1234 2021-02-04 20:14:10,632 INFO status on pbsdev-centos7-mky18-server: hook s1234 2021-02-04 20:14:10,632 INFO manager on pbsdev-centos7-mky18-server: list hook s1234 2021-02-04 20:14:10,632 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook s1234 2021-02-04 20:14:10,650 INFO status on pbsdev-centos7-mky18-server: hook s1234 2021-02-04 20:14:10,650 INFO manager on pbsdev-centos7-mky18-server: list hook s1234 2021-02-04 20:14:10,650 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook s1234 2021-02-04 20:14:10,670 INFO manager on pbsdev-centos7-mky18-server: set hook s1234 {'event': 'modifyvnode', 'enabled': 'True', 'debug': 'True'} 2021-02-04 20:14:10,671 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook s1234 event=modifyvnode,enabled=True,debug=True 2021-02-04 20:14:10,690 INFO status on pbsdev-centos7-mky18-server: server sync_mom_hookfiles_timeout 2021-02-04 20:14:10,690 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:10,701 INFO Setting sync_mom_hookfiles_timeout to 15s 2021-02-04 20:14:10,701 INFO manager on pbsdev-centos7-mky18-server: set server {'sync_mom_hookfiles_timeout': 15} 2021-02-04 20:14:10,701 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server sync_mom_hookfiles_timeout=15 2021-02-04 20:14:10,712 INFO manager on pbsdev-centos7-mky18-server: import hook s1234 {'content-type': 'application/x-python', 'content-encoding': 'default', 'input-file': '/tmp/PtlPbs693ylegj'} 2021-02-04 20:14:10,712 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c import hook s1234 application/x-python default /tmp/PtlPbs693ylegj 2021-02-04 20:14:10,731 INFO server pbsdev-centos7-mky18-server: imported hook body --- import pbs import sys try: e = pbs.event() v = e.vnode v_o = e.vnode_o lsct = v.last_state_change_time lsct_o = v_o.last_state_change_time state_str_buf_v = ",".join(v.extract_state_strs()) state_str_buf_v_o = ",".join(v_o.extract_state_strs()) state_int_buf_v = ','.join([str(_) for _ in v.extract_state_ints()]) state_int_buf_v_o = ','.join( [str(_) for _ in v_o.extract_state_ints()]) # print show_vnode_state record (bi consumable) svs1_data = "v.state_hex=%s v_o.state_hex=%s v.state_strs=%s " "v_o.state_strs=%s" % (hex(v.state), hex(v_o.state), state_str_buf_v, state_str_buf_v_o) svs2_data = "v.state_ints=%s v_o.state_ints=%s v.lsct=%s " "v_o.lsct=%s" % (state_int_buf_v, state_int_buf_v_o, str(lsct), str(lsct_o)) svs_data = "%s %s" % (svs1_data, svs2_data) pbs.logmsg(pbs.LOG_DEBUG, "show_vnode_state;name=%s %s" % (v.name, svs_data)) # print additional hook parameter values pbs.logmsg(pbs.LOG_DEBUG, "name: v=%s, v_o=%s" % (v.name, v_o.name)) pbs.logmsg(pbs.LOG_DEBUG, "state: v=%s, v_o=%s" % (hex(v.state), hex(v_o.state))) pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: v=%s, v_o=%s" % ( str(lsct), str(lsct_o))) pbs.logmsg(pbs.LOG_DEBUG, "comment: v=%s, v_o=%s" % (v.comment, v_o.comment)) pbs.logmsg(pbs.LOG_DEBUG, "aoe: v=%s, v_o=%s" % (v.current_aoe, v_o.current_aoe)) pbs.logmsg(pbs.LOG_DEBUG, "in_mvn_host: v=%s, v_o=%s" % ( v.in_multivnode_host, v_o.in_multivnode_host)) pbs.logmsg(pbs.LOG_DEBUG, "jobs: v=%s, v_o=%s" % (v.jobs, v_o.jobs)) pbs.logmsg(pbs.LOG_DEBUG, "Mom: v=%s, v_o=%s" % (v.Mom, v_o.Mom)) pbs.logmsg(pbs.LOG_DEBUG, "ntype: v=%s, v_o=%s" % (hex(v.ntype), hex(v_o.ntype))) pbs.logmsg(pbs.LOG_DEBUG, "pcpus: v=%s, v_o=%s" % (v.pcpus, v_o.pcpus)) pbs.logmsg(pbs.LOG_DEBUG, "pnames: v=%s, v_o=%s" % (v.pnames, v_o.pnames)) pbs.logmsg(pbs.LOG_DEBUG, "Port: v=%s, v_o=%s" % (v.Port, v_o.Port)) pbs.logmsg(pbs.LOG_DEBUG, "Priority: v=%s, v_o=%s" % (v.Priority, v_o.Priority)) pbs.logmsg(pbs.LOG_DEBUG, "provision_enable: v=%s, v_o=%s" % ( v.provision_enable, v_o.provision_enable)) pbs.logmsg(pbs.LOG_DEBUG, "queue: v=%s, v_o=%s" % (v.queue, v_o.queue)) pbs.logmsg(pbs.LOG_DEBUG, "res_assigned: v=%s, v_o=%s" % ( v.resources_assigned, v_o.resources_assigned)) pbs.logmsg(pbs.LOG_DEBUG, "res_avail: v=%s, v_o=%s" % ( v.resources_available, v_o.resources_available)) pbs.logmsg(pbs.LOG_DEBUG, "resv: v=%s, v_o=%s" % (v.resv, v_o.resv)) pbs.logmsg(pbs.LOG_DEBUG, "resv_enable: v=%s, v_o=%s" % ( v.resv_enable, v_o.resv_enable)) pbs.logmsg(pbs.LOG_DEBUG, "sharing: v=%s, v_o=%s" % (v.sharing, v_o.sharing)) # sanity test some values if (lsct < lsct_o) or (lsct_o <= 0): e.reject("last_state_change_time: bad timestamp value") else: pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: good times") if (v.name != v_o.name) or (not v.name): e.reject( "name: vnode and vnode_o name values are null or mismatched") else: pbs.logmsg(pbs.LOG_DEBUG, "name: good names") if (isinstance(v.state, int)) and (isinstance(v_o.state, int)): pbs.logmsg(pbs.LOG_DEBUG, "state: good states") else: e.reject("state: bad state value") if len(v.extract_state_strs()) == len(v.extract_state_ints()): pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v sets") else: e.reject("state sets: bad v sets") if len(v_o.extract_state_strs()) == len(v_o.extract_state_ints()): pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v_o sets") else: e.reject("state sets: bad v_o sets") e.accept() except SystemExit: pass except: pbs.event().reject("%s hook failed with %s" % ( pbs.event().hook_name, sys.exc_info()[:2])) --- 2021-02-04 20:14:10,734 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:14:10,744 INFO running init script to stop pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:14:10,745 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbsputyz6po Contents of /tmp/PtlPbsputyz6po: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:14:16,355 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbsputyz6po 2021-02-04 20:14:16,361 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:14:16,373 INFO running init script to start pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:14:16,374 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbsxd5csrwg Contents of /tmp/PtlPbsxd5csrwg: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:14:20,372 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbsxd5csrwg 2021-02-04 20:14:20,380 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2021-02-04 20:14:20,390 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:20,403 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match 2021-02-04 20:14:21,602 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2 2021-02-04 20:14:22,793 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:14:22,801 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:14:22,803 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:14:22,806 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:14:22,809 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:14:22,811 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:14:22,814 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:14:22,816 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:14:22,819 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:14:22,823 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:14:22,826 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:14:23,996 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:14:24,003 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:14:24,006 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:14:24,009 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:14:24,009 INFO checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom1 start=1612469650.7319407 end=1612469664.0096166 2021-02-04 20:14:24,049 INFO server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom1" - on all matches - with existence... OK 2021-02-04 20:14:24,053 INFO server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK 2021-02-04 20:14:24,055 INFO server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK 2021-02-04 20:14:24,058 INFO server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK 2021-02-04 20:14:24,061 INFO server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK 2021-02-04 20:14:24,064 INFO server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK 2021-02-04 20:14:24,066 INFO server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK 2021-02-04 20:14:24,069 INFO server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK 2021-02-04 20:14:24,073 INFO server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK 2021-02-04 20:14:24,077 INFO server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK 2021-02-04 20:14:24,081 INFO server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK 2021-02-04 20:14:24,085 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:14:24,088 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK 2021-02-04 20:14:24,092 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK 2021-02-04 20:14:24,097 INFO server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK 2021-02-04 20:14:24,097 INFO checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom2 start=1612469650.7319407 end=1612469664.0977042 2021-02-04 20:14:24,143 INFO server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom2" - on all matches - with existence... OK 2021-02-04 20:14:24,144 INFO ================================================ 2021-02-04 20:14:24,144 INFO Entered TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:14:24,144 INFO ================================================ 2021-02-04 20:14:24,144 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:24,154 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:14:24,154 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:24,369 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:14:24,369 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:14:24,370 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:14:24,381 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:24,392 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:24,404 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:24,421 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:24,432 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:24,444 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:14:24,444 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:14:24,456 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:24,466 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:24,481 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:24,493 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:24,511 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:24,523 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:24,534 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:14:24,535 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:14:24,535 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:14:24,750 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:14:24,751 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:14:24,751 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:14:24,765 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:14:24,776 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:24,789 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:24,802 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:24,819 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:24,831 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:24,843 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:14:24,844 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:14:24,844 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:14:24,854 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:14:24,867 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:14:24,880 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:14:24,896 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:14:24,909 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:14:24,921 INFO ================================================= 2021-02-04 20:14:24,921 INFO Completed TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:14:24,921 INFO ================================================= 2021-02-04 20:14:24,922 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:14:24,979 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:14:25,035 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:14:25,090 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:14:25,148 INFO ok 2021-02-04 20:14:25,149 INFO ================================================================================ run: 5, succeeded: 5, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:03:01.111071 2021-02-04 20:14:25,149 INFO Cleaning up temporary files 2021-02-04 20:14:25,149 INFO Cleaning up /var/tmp dir 2021-02-04 20:14:25,149 INFO Cleaning up /tmp dir real 3m1.711s user 0m57.214s sys 0m13.521s [02/04 20:14:25] pbsdev@pbsdev-centos7-mky18-server:~ 50%