[02/04 20:42:12] pbsdev@pbsdev-centos7-mky18-server:~ 59% time sudo $CV2_PTL_INSTALL_DIR/bin/pbs_benchpress --db-name $CV2_PBS_BUILD_DIR/test/tests/ptl_test_results.json -f $CV2_PTL_INSTALL_DIR/tests/functional/pbs_hook_modifyvnode_state_changes.py -t TestPbsModifyvnodeStateChanges.test_check_node_state_lookup_00 -p "moms=mom1:mom2" 2021-02-04 20:42:31,759 INFO input command: pbs_benchpress --db-name /home/pbsdev/pbs/build/test/tests/ptl_test_results.json -f /opt/ptl/tests/functional/pbs_hook_modifyvnode_state_changes.py -t TestPbsModifyvnodeStateChanges.test_check_node_state_lookup_00 -p moms=mom1:mom2 2021-02-04 20:42:31,759 INFO param: moms=mom1:mom2 2021-02-04 20:42:31,759 INFO ptl version: 19.1.1.1093.g7f30851.modified 2021-02-04 20:42:31,762 INFO platform: Linux pbsdev-centos7-mky18-server 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 x86_64 2021-02-04 20:42:31,762 INFO python version: 3.6.8 2021-02-04 20:42:31,762 INFO user: root 2021-02-04 20:42:31,762 INFO -------------------------------------------------------------------------------- 2021-02-04 20:42:31,763 INFO Cleaning up temporary files 2021-02-04 20:42:31,763 INFO Cleaning up /var/tmp dir 2021-02-04 20:42:31,763 INFO Cleaning up /tmp dir 2021-02-04 20:42:31,773 INFO ====================================================================== 2021-02-04 20:42:31,773 INFO suite name: TestPbsModifyvnodeStateChanges 2021-02-04 20:42:31,773 INFO suite docstring: Test the modifyvnode hook by inducing various vnode state changes and inspecting the pbs log for expected values. 2021-02-04 20:42:31,773 INFO ====================================================================== 2021-02-04 20:42:31,773 INFO ================================================== 2021-02-04 20:42:31,773 INFO Entered TestPbsModifyvnodeStateChanges setUpClass 2021-02-04 20:42:31,774 INFO ================================================== 2021-02-04 20:42:31,774 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser 2021-02-04 20:42:31,776 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser1 2021-02-04 20:42:31,779 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser2 2021-02-04 20:42:31,781 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser3 2021-02-04 20:42:31,783 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser4 2021-02-04 20:42:31,785 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser5 2021-02-04 20:42:31,788 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser6 2021-02-04 20:42:31,790 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser7 2021-02-04 20:42:31,793 INFOCLI2 pbsdev-centos7-mky18-server: id pbsother 2021-02-04 20:42:31,796 INFOCLI2 pbsdev-centos7-mky18-server: id pbstest 2021-02-04 20:42:31,798 INFOCLI2 pbsdev-centos7-mky18-server: id tstusr00 2021-02-04 20:42:31,801 INFOCLI2 pbsdev-centos7-mky18-server: id tstusr01 2021-02-04 20:42:31,804 INFOCLI2 pbsdev-centos7-mky18-server: id pbsoper 2021-02-04 20:42:31,808 INFOCLI2 pbsdev-centos7-mky18-server: id pbsmgr 2021-02-04 20:42:31,811 INFOCLI2 pbsdev-centos7-mky18-server: id pbsadmin 2021-02-04 20:42:31,813 INFOCLI2 pbsdev-centos7-mky18-server: id pbsdata 2021-02-04 20:42:31,815 INFOCLI2 pbsdev-centos7-mky18-server: id pbsroot 2021-02-04 20:42:31,818 INFOCLI2 pbsdev-centos7-mky18-server: id root 2021-02-04 20:42:31,820 INFOCLI2 pbsdev-centos7-mky18-server: id pbsbuild 2021-02-04 20:42:31,823 INFOCLI2 pbsdev-centos7-mky18-server: id root 2021-02-04 20:42:31,825 INFO FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server 2021-02-04 20:42:31,835 INFO server pbsdev-centos7-mky18-server: server operating mode set to cli 2021-02-04 20:42:31,835 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:31,844 INFO server pbsdev-centos7-mky18-server: version 19.1.1.1093.g7f30851.modified 2021-02-04 20:42:31,844 INFO expect action: created new action kicksched 2021-02-04 20:42:31,844 INFO expect action: added action kicksched to server pbsdev-centos7-mky18-server 2021-02-04 20:42:31,845 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:32,058 INFO FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server 2021-02-04 20:42:32,063 INFOCLI2 pbsdev-centos7-mky18-server: which chown 2021-02-04 20:42:32,064 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:32,074 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:32,085 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:32,101 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:32,112 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:32,124 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_logs 2021-02-04 20:42:32,135 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:42:32,150 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:32,160 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:32,171 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:32,188 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:32,198 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:32,208 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_logs 2021-02-04 20:42:32,220 INFO FQDN name pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local differs from name provided mom1 2021-02-04 20:42:32,221 INFOCLI2 mom1: ssh mom1 ls -1 /opt/pbs/python/bin/python 2021-02-04 20:42:32,239 INFOCLI2 mom1: ssh mom1 python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:42:32,324 INFO FQDN name pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local differs from name provided mom2 2021-02-04 20:42:32,326 INFOCLI2 mom2: ssh mom2 ls -1 /opt/pbs/python/bin/python 2021-02-04 20:42:32,343 INFOCLI2 mom2: ssh mom2 python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:42:32,426 INFO FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server 2021-02-04 20:42:32,431 INFO manager on pbsdev-centos7-mky18-server: set server {'pbs_license_min': 2} 2021-02-04 20:42:32,431 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server pbs_license_min=2 2021-02-04 20:42:32,451 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:32,667 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:42:32,679 INFO running init script to stop pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:42:32,679 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbs3y85qjb8 Contents of /tmp/PtlPbs3y85qjb8: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:42:38,271 INFOCLI2 pbsdev-centos7-mky18-server: which rm 2021-02-04 20:42:38,273 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbs3y85qjb8 2021-02-04 20:42:38,277 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:42:38,287 INFO running init script to start pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:42:38,287 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbsl7sbhh9y Contents of /tmp/PtlPbsl7sbhh9y: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:42:42,274 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbsl7sbhh9y 2021-02-04 20:42:42,281 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock 2021-02-04 20:42:42,291 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:42,301 INFO =================================================== 2021-02-04 20:42:42,301 INFO Completed TestPbsModifyvnodeStateChanges setUpClass 2021-02-04 20:42:42,301 INFO =================================================== 2021-02-04 20:42:42,336 INFO mom pbsdev-centos7-mky18-mom1 log match: searching for "alps_client" - with existence - No match 2021-02-04 20:42:42,842 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -l /etc/sgi-compute-node-release 2021-02-04 20:42:42,864 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -l /etc/sgi-known-distributions 2021-02-04 20:42:42,881 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -l /etc/sgi-compute-node-release 2021-02-04 20:42:42,899 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -l /etc/sgi-known-distributions 2021-02-04 20:42:42,917 INFOCLI2 mom2: ssh mom2 which cat 2021-02-04 20:42:42,934 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo 2021-02-04 20:42:42,965 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs 2021-02-04 20:42:42,998 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:42:42,998 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:42:43,016 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:42:43,041 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:42:43,041 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:42:43,059 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:42:43,084 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:42:43,084 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs 2021-02-04 20:42:43,101 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:42:43,125 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:42:43,151 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:42:43,176 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:42:43,200 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:42:43,225 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:42:43,251 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:42:43,277 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:42:43,301 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:42:43,326 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:42:43,352 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:42:43,377 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:42:43,401 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:42:43,426 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:42:43,451 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:42:43,477 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:42:43,501 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:42:43,526 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:42:43,551 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:42:43,576 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:42:43,602 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root 2021-02-04 20:42:43,628 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo 2021-02-04 20:42:43,632 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs 2021-02-04 20:42:43,635 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:42:43,646 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:42:43,657 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:42:43,667 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:42:43,678 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:42:43,689 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:42:43,699 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:42:43,713 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:42:43,724 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:42:43,734 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:42:43,745 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:42:43,756 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:42:43,766 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:42:43,777 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:42:43,788 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:42:43,798 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:42:43,809 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:42:43,820 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:42:43,830 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:42:43,841 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:42:43,852 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:42:43,863 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:42:43,873 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root 2021-02-04 20:42:43,885 INFOCLI2 mom1: ssh mom1 which cat 2021-02-04 20:42:43,902 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo 2021-02-04 20:42:43,936 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs 2021-02-04 20:42:43,969 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs 2021-02-04 20:42:43,969 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:42:43,988 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv 2021-02-04 20:42:44,013 INFOCLI2 grep'ing for server_priv in /var/spool/pbs 2021-02-04 20:42:44,013 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:42:44,032 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv 2021-02-04 20:42:44,057 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs 2021-02-04 20:42:44,058 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs 2021-02-04 20:42:44,075 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv 2021-02-04 20:42:44,101 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser 2021-02-04 20:42:44,127 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1 2021-02-04 20:42:44,155 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2 2021-02-04 20:42:44,182 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3 2021-02-04 20:42:44,209 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4 2021-02-04 20:42:44,235 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5 2021-02-04 20:42:44,261 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6 2021-02-04 20:42:44,289 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7 2021-02-04 20:42:44,316 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother 2021-02-04 20:42:44,344 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest 2021-02-04 20:42:44,368 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00 2021-02-04 20:42:44,396 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01 2021-02-04 20:42:44,423 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper 2021-02-04 20:42:44,449 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr 2021-02-04 20:42:44,476 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin 2021-02-04 20:42:44,502 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata 2021-02-04 20:42:44,529 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot 2021-02-04 20:42:44,556 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:42:44,582 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild 2021-02-04 20:42:44,609 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root 2021-02-04 20:42:44,635 INFO test name: test_check_node_state_lookup_00 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)... 2021-02-04 20:42:44,635 INFO test start time: Thu Feb 4 20:42:44 2021 2021-02-04 20:42:44,636 INFO test docstring: Test: check for the existence and values of the pbs.REVERSE_STATE_CHANGES dictionary run a hook that converts a state change hex into a string, then search for it in the server log. 2021-02-04 20:42:44,636 INFO ============================================= 2021-02-04 20:42:44,636 INFO Entered TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:42:44,636 INFO ============================================= 2021-02-04 20:42:44,636 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:44,647 INFO status on pbsdev-centos7-mky18-server: server 2021-02-04 20:42:44,648 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:44,861 INFO manager on pbsdev-centos7-mky18-server: unset server managers 2021-02-04 20:42:44,862 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers 2021-02-04 20:42:44,881 INFO manager on pbsdev-centos7-mky18-server: unset server operators 2021-02-04 20:42:44,882 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators 2021-02-04 20:42:44,901 INFO status on pbsdev-centos7-mky18-server: server ['managers', 'operators'] 2021-02-04 20:42:44,901 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:45,115 INFO manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')} 2021-02-04 20:42:45,115 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@* 2021-02-04 20:42:45,133 INFO server pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:42:45,134 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:45,349 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:42:45,350 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:45,565 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:42:45,565 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:42:45,565 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:42:45,576 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:45,586 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:45,598 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:45,614 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:45,625 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:45,636 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:42:45,637 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:42:45,649 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:42:45,660 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:45,670 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:45,682 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:45,699 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:45,710 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:45,721 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:42:45,722 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:42:45,722 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:45,939 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:42:45,940 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:42:45,940 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:42:45,953 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:42:45,963 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:45,973 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:45,986 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:46,003 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:46,014 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:46,025 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:42:46,025 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:42:46,025 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:42:46,035 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:46,045 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:42:46,045 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:46,261 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:42:46,261 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:42:46,262 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:42:46,272 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:46,282 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:46,293 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:46,310 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:46,322 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:46,332 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:42:46,333 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:42:46,344 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:42:46,355 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:46,365 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:46,376 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:46,392 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:46,402 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:46,412 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:42:46,413 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:42:46,413 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:46,628 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:42:46,629 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:42:46,629 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:42:46,642 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:42:46,652 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:46,662 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:46,673 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:46,688 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:46,700 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:46,711 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:42:46,711 INFO status on pbsdev-centos7-mky18-server as root: reservation 2021-02-04 20:42:46,711 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f 2021-02-04 20:42:46,721 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:46,935 INFO manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'scheduler_iteration', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision', 'max_job_sequence_id'] 2021-02-04 20:42:46,935 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,scheduler_iteration,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision,max_job_sequence_id 2021-02-04 20:42:46,950 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:42:46,967 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:47,181 INFO status on pbsdev-centos7-mky18-server: node 2021-02-04 20:42:47,182 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a 2021-02-04 20:42:47,193 INFO manager on pbsdev-centos7-mky18-server: delete queue workq 2021-02-04 20:42:47,193 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq 2021-02-04 20:42:47,203 INFO manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'} 2021-02-04 20:42:47,204 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True 2021-02-04 20:42:47,214 INFO status on pbsdev-centos7-mky18-server: queue workq 2021-02-04 20:42:47,214 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:47,430 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:42:47,431 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:42:47,442 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:47,453 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:47,468 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:47,486 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:47,499 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:47,511 INFO manager on pbsdev-centos7-mky18-server: delete node @default 2021-02-04 20:42:47,511 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default 2021-02-04 20:42:47,525 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:42:47,542 INFO manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'} 2021-02-04 20:42:47,542 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false 2021-02-04 20:42:47,562 INFO manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'} 2021-02-04 20:42:47,563 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq 2021-02-04 20:42:47,574 INFO status on pbsdev-centos7-mky18-server: resource 2021-02-04 20:42:47,574 INFO manager on pbsdev-centos7-mky18-server: list resource 2021-02-04 20:42:47,575 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource 2021-02-04 20:42:47,585 INFOCLI status on pbsdev-centos7-mky18-server: server license_count 2021-02-04 20:42:47,585 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:47,799 INFO server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed 2021-02-04 20:42:47,800 INFO status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:47,800 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:48,012 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:42:48,030 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:42:48,077 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:42:48,095 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))" 2021-02-04 20:42:48,145 ERROR Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value 2021-02-04 20:42:48,151 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:42:48,162 INFO scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults 2021-02-04 20:42:48,162 INFO manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm'] 2021-02-04 20:42:48,162 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm 2021-02-04 20:42:48,176 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:48,188 INFOCLI2 pbsdev-centos7-mky18-server: which cp 2021-02-04 20:42:48,192 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbs84stwbmd /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:48,203 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:48,213 INFO scheduler pbsdev-centos7-mky18-server: reverting holidays file to default 2021-02-04 20:42:48,214 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:48,225 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config 2021-02-04 20:42:48,235 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:48,246 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:48,258 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:48,269 INFO Scheduler sent signal -HUP 2021-02-04 20:42:48,269 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:42:48,280 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148 2021-02-04 20:42:48,291 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default 2021-02-04 20:42:48,317 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock 2021-02-04 20:42:48,329 INFO status on pbsdev-centos7-mky18-server: sched default 2021-02-04 20:42:48,329 INFO manager on pbsdev-centos7-mky18-server: list sched default 2021-02-04 20:42:48,329 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:42:48,339 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:42:48,350 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:42:48,362 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:42:48,378 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:42:48,390 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:42:48,452 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:42:48,487 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import sys; print(sys.platform)" 2021-02-04 20:42:48,542 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:48,569 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:48,569 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:48,578 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:42:48,578 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which rm 2021-02-04 20:42:48,597 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:42:48,623 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:42:48,649 INFOCLI pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:42:48,741 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:42:48,794 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which scp 2021-02-04 20:42:48,813 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbsr0qt5gpq pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmp73zsvxe4PtlPbstmpcopy 2021-02-04 20:42:48,840 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which cp 2021-02-04 20:42:48,860 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmp73zsvxe4PtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:42:48,888 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmp73zsvxe4PtlPbstmpcopy 2021-02-04 20:42:48,909 INFO MoM sent signal -HUP 2021-02-04 20:42:48,909 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:48,935 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 16794 2021-02-04 20:42:48,980 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:49,005 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,005 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,015 ERROR err: ['/opt/pbs/bin/pbsnodes: Server has no node list'] 2021-02-04 20:42:49,015 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,016 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,032 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,032 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,042 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:49,053 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 got: state = state-unknown,down 2021-02-04 20:42:50,057 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:50,070 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2 got: state = state-unknown,down 2021-02-04 20:42:51,072 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:51,082 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ... OK 2021-02-04 20:42:51,082 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:51,082 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:51,117 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python 2021-02-04 20:42:51,155 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import sys; print(sys.platform)" 2021-02-04 20:42:51,205 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:51,230 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,230 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,240 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:42:51,240 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which rm 2021-02-04 20:42:51,260 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue 2021-02-04 20:42:51,287 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue 2021-02-04 20:42:51,313 INFOCLI pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list 2021-02-04 20:42:51,405 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])" 2021-02-04 20:42:51,457 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which scp 2021-02-04 20:42:51,475 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbswoxw86f9 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpi9w15wfbPtlPbstmpcopy 2021-02-04 20:42:51,502 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which cp 2021-02-04 20:42:51,522 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpi9w15wfbPtlPbstmpcopy /var/spool/pbs/mom_priv/config 2021-02-04 20:42:51,550 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpi9w15wfbPtlPbstmpcopy 2021-02-04 20:42:51,567 INFO MoM sent signal -HUP 2021-02-04 20:42:51,568 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:51,593 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 16730 2021-02-04 20:42:51,638 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:51,663 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,664 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,674 ERROR err: ['Node: pbsdev-centos7-mky18-mom2, Error: Unknown node '] 2021-02-04 20:42:51,674 INFO manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,674 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,689 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,690 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,699 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:51,710 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 got: state = state-unknown,down 2021-02-04 20:42:52,713 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:52,727 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2 got: state = state-unknown,down 2021-02-04 20:42:53,728 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:53,738 INFO expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ... OK 2021-02-04 20:42:53,738 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:42:53,739 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:42:53,748 INFO manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'} 2021-02-04 20:42:53,748 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off 2021-02-04 20:42:53,765 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock 2021-02-04 20:42:53,775 INFO ============================================== 2021-02-04 20:42:53,776 INFO Completed TestPbsModifyvnodeStateChanges setUp 2021-02-04 20:42:53,776 INFO ============================================== 2021-02-04 20:42:53,790 INFO manager on pbsdev-centos7-mky18-server: set server {'log_events': 4095} 2021-02-04 20:42:53,791 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server log_events=4095 2021-02-04 20:42:53,801 INFO status on pbsdev-centos7-mky18-server: hook 2021-02-04 20:42:53,801 INFO manager on pbsdev-centos7-mky18-server: list hook 2021-02-04 20:42:53,801 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook 2021-02-04 20:42:53,820 INFO manager on pbsdev-centos7-mky18-server: create hook x1234 2021-02-04 20:42:53,820 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c create hook x1234 2021-02-04 20:42:53,839 INFO status on pbsdev-centos7-mky18-server: hook x1234 2021-02-04 20:42:53,840 INFO manager on pbsdev-centos7-mky18-server: list hook x1234 2021-02-04 20:42:53,840 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook x1234 2021-02-04 20:42:53,858 INFO status on pbsdev-centos7-mky18-server: hook x1234 2021-02-04 20:42:53,858 INFO manager on pbsdev-centos7-mky18-server: list hook x1234 2021-02-04 20:42:53,858 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook x1234 2021-02-04 20:42:53,876 INFO manager on pbsdev-centos7-mky18-server: set hook x1234 {'event': 'modifyvnode', 'enabled': 'True', 'debug': 'True'} 2021-02-04 20:42:53,877 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook x1234 event=modifyvnode,enabled=True,debug=True 2021-02-04 20:42:53,896 INFO status on pbsdev-centos7-mky18-server: server sync_mom_hookfiles_timeout 2021-02-04 20:42:53,896 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:42:53,906 INFO Setting sync_mom_hookfiles_timeout to 15s 2021-02-04 20:42:53,906 INFO manager on pbsdev-centos7-mky18-server: set server {'sync_mom_hookfiles_timeout': 15} 2021-02-04 20:42:53,906 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server sync_mom_hookfiles_timeout=15 2021-02-04 20:42:53,917 INFO manager on pbsdev-centos7-mky18-server: import hook x1234 {'content-type': 'application/x-python', 'content-encoding': 'default', 'input-file': '/tmp/PtlPbswc3opkps'} 2021-02-04 20:42:53,918 INFOCLI pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c import hook x1234 application/x-python default /tmp/PtlPbswc3opkps 2021-02-04 20:42:53,936 INFO server pbsdev-centos7-mky18-server: imported hook body --- import pbs e = pbs.event() pbs.logmsg(pbs.LOG_DEBUG, "pbs.__file__:" + pbs.__file__) # this is backwards as it's a reverse lookup. for value, key in pbs.REVERSE_NODE_STATE.items(): pbs.logmsg(pbs.LOG_DEBUG, "key:%s value:%s" % (key, value)) e.accept() --- 2021-02-04 20:42:53,957 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:53,983 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1 2021-02-04 20:42:53,983 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:53,994 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1 2021-02-04 20:42:54,005 INFO expect on server pbsdev-centos7-mky18-server: state ~ free|provisioning|offline|job-busy node pbsdev-centos7-mky18-mom1 ... OK 2021-02-04 20:42:54,025 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:42:54,052 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:42:54,053 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsl2do2b5l pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsl2do2b5l 2021-02-04 20:42:54,077 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsl2do2b5l Contents of /tmp/PtlPbsl2do2b5l: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:42:55,139 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsl2do2b5l 2021-02-04 20:42:55,172 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:42:55,198 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:42:55,200 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsfgefp0bc pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsfgefp0bc 2021-02-04 20:42:55,224 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsfgefp0bc Contents of /tmp/PtlPbsfgefp0bc: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:42:55,328 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsfgefp0bc 2021-02-04 20:42:55,367 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:42:55,398 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:42:56,708 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - attempt 2 2021-02-04 20:42:58,003 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence - attempt 3 2021-02-04 20:42:59,308 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:42:59,318 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence - No match 2021-02-04 20:43:00,611 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:43:00,618 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_FREE value:0" - with existence... OK 2021-02-04 20:43:00,621 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE value:1" - with existence... OK 2021-02-04 20:43:00,624 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DOWN value:2" - with existence... OK 2021-02-04 20:43:00,627 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DELETED value:4" - with existence... OK 2021-02-04 20:43:00,630 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_STALE value:32" - with existence... OK 2021-02-04 20:43:00,635 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOBBUSY value:16" - with existence... OK 2021-02-04 20:43:00,639 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOB_EXCLUSIVE value:64" - with existence... OK 2021-02-04 20:43:00,643 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_RESV_EXCLUSIVE value:8192" - with existence... OK 2021-02-04 20:43:00,647 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_BUSY value:128" - with existence... OK 2021-02-04 20:43:00,650 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNKNOWN value:256" - with existence... OK 2021-02-04 20:43:00,652 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEEDS_HELLOSVR value:512" - with existence... OK 2021-02-04 20:43:00,655 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_INIT value:1024" - with existence... OK 2021-02-04 20:43:00,659 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_PROV value:2048" - with existence... OK 2021-02-04 20:43:00,662 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_WAIT_PROV value:4096" - with existence... OK 2021-02-04 20:43:00,667 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNRESOLVABLE value:8" - with existence... OK 2021-02-04 20:43:00,671 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_SLEEP value:262144" - with existence... OK 2021-02-04 20:43:00,674 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE_BY_MOM value:16384" - with existence... OK 2021-02-04 20:43:00,678 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MARKEDDOWN value:32768" - with existence... OK 2021-02-04 20:43:00,681 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_ADDRS value:65536" - with existence... OK 2021-02-04 20:43:00,684 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MAINTENANCE value:131072" - with existence... OK 2021-02-04 20:43:00,686 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_CREDENTIALS value:524288" - with existence... OK 2021-02-04 20:43:00,689 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_VNODE_UNAVAILABLE value:409903" - with existence... OK 2021-02-04 20:43:00,711 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:43:00,738 INFO status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2 2021-02-04 20:43:00,738 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:43:00,749 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2 2021-02-04 20:43:00,760 INFO expect on server pbsdev-centos7-mky18-server: state ~ free|provisioning|offline|job-busy node pbsdev-centos7-mky18-mom2 ... OK 2021-02-04 20:43:00,778 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:43:00,803 INFO running init script to stop pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop'] 2021-02-04 20:43:00,805 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsjda443oj pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbsjda443oj 2021-02-04 20:43:00,830 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbsjda443oj Contents of /tmp/PtlPbsjda443oj: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop ---------------------------------------- 2021-02-04 20:43:01,894 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsjda443oj 2021-02-04 20:43:01,930 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d 2021-02-04 20:43:01,956 INFO running init script to start pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start'] 2021-02-04 20:43:01,957 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbseyapb780 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbseyapb780 2021-02-04 20:43:01,983 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbseyapb780 Contents of /tmp/PtlPbseyapb780: ---------------------------------------- #!/bin/bash sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start ---------------------------------------- 2021-02-04 20:43:02,080 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbseyapb780 2021-02-04 20:43:02,119 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock 2021-02-04 20:43:02,149 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - No match 2021-02-04 20:43:03,452 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - attempt 2 2021-02-04 20:43:04,744 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence - attempt 3 2021-02-04 20:43:06,042 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK 2021-02-04 20:43:06,051 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence - No match 2021-02-04 20:43:07,338 INFO server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence... OK 2021-02-04 20:43:07,345 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_FREE value:0" - with existence... OK 2021-02-04 20:43:07,348 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE value:1" - with existence... OK 2021-02-04 20:43:07,351 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DOWN value:2" - with existence... OK 2021-02-04 20:43:07,354 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_DELETED value:4" - with existence... OK 2021-02-04 20:43:07,357 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_STALE value:32" - with existence... OK 2021-02-04 20:43:07,360 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOBBUSY value:16" - with existence... OK 2021-02-04 20:43:07,364 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_JOB_EXCLUSIVE value:64" - with existence... OK 2021-02-04 20:43:07,368 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_RESV_EXCLUSIVE value:8192" - with existence... OK 2021-02-04 20:43:07,373 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_BUSY value:128" - with existence... OK 2021-02-04 20:43:07,377 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNKNOWN value:256" - with existence... OK 2021-02-04 20:43:07,380 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEEDS_HELLOSVR value:512" - with existence... OK 2021-02-04 20:43:07,383 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_INIT value:1024" - with existence... OK 2021-02-04 20:43:07,386 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_PROV value:2048" - with existence... OK 2021-02-04 20:43:07,389 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_WAIT_PROV value:4096" - with existence... OK 2021-02-04 20:43:07,392 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_UNRESOLVABLE value:8" - with existence... OK 2021-02-04 20:43:07,395 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_SLEEP value:262144" - with existence... OK 2021-02-04 20:43:07,399 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_OFFLINE_BY_MOM value:16384" - with existence... OK 2021-02-04 20:43:07,402 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MARKEDDOWN value:32768" - with existence... OK 2021-02-04 20:43:07,406 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_ADDRS value:65536" - with existence... OK 2021-02-04 20:43:07,410 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_MAINTENANCE value:131072" - with existence... OK 2021-02-04 20:43:07,413 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_NEED_CREDENTIALS value:524288" - with existence... OK 2021-02-04 20:43:07,416 INFO server pbsdev-centos7-mky18-server log match: searching for "key:ND_STATE_VNODE_UNAVAILABLE value:409903" - with existence... OK 2021-02-04 20:43:07,417 INFO ================================================ 2021-02-04 20:43:07,417 INFO Entered TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:43:07,417 INFO ================================================ 2021-02-04 20:43:07,417 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:43:07,426 INFO status on pbsdev-centos7-mky18-server: job 2021-02-04 20:43:07,427 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:43:07,643 INFO status on pbsdev-centos7-mky18-server: sched 2021-02-04 20:43:07,643 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:43:07,643 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:43:07,654 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:43:07,664 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:43:07,676 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:43:07,694 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:43:07,706 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:43:07,717 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'} 2021-02-04 20:43:07,717 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False 2021-02-04 20:43:07,731 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:43:07,741 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:43:07,753 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:43:07,765 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:43:07,782 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:43:07,793 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:43:07,805 INFO expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ... OK 2021-02-04 20:43:07,805 INFO delete job on pbsdev-centos7-mky18-server as root: 2021-02-04 20:43:07,805 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local 2021-02-04 20:43:08,021 INFO expect on server pbsdev-centos7-mky18-server: job_state set 0 job ... OK 2021-02-04 20:43:08,022 INFO manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'} 2021-02-04 20:43:08,022 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True 2021-02-04 20:43:08,036 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default 2021-02-04 20:43:08,048 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:43:08,060 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:43:08,073 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:43:08,091 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:43:08,104 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:43:08,118 INFO expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ... OK 2021-02-04 20:43:08,118 INFO manager on pbsdev-centos7-mky18-server: list sched 2021-02-04 20:43:08,118 INFOCLI pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched 2021-02-04 20:43:08,128 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv 2021-02-04 20:43:08,139 INFO scheduler pbsdev-centos7-mky18-server: setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time 2021-02-04 20:43:08,153 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs 2021-02-04 20:43:08,168 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group 2021-02-04 20:43:08,182 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays 2021-02-04 20:43:08,194 INFO ================================================= 2021-02-04 20:43:08,194 INFO Completed TestPbsModifyvnodeStateChanges tearDown 2021-02-04 20:43:08,194 INFO ================================================= 2021-02-04 20:43:08,196 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:43:08,253 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:43:08,310 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))" 2021-02-04 20:43:08,367 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())" 2021-02-04 20:43:08,426 INFO ok 2021-02-04 20:43:08,427 INFO ================================================================================ run: 1, succeeded: 1, failed: 0, errors: 0, skipped: 0, timedout: 0 Tests run in 0:00:36.654245 2021-02-04 20:43:08,427 INFO Cleaning up temporary files 2021-02-04 20:43:08,428 INFO Cleaning up /var/tmp dir 2021-02-04 20:43:08,428 INFO Cleaning up /tmp dir real 0m36.969s user 0m9.289s sys 0m3.029s [02/04 20:43:08] pbsdev@pbsdev-centos7-mky18-server:~ 60%