[02/04 20:26:38] pbsdev@pbsdev-centos7-mky18-server:~ 52% time sudo $CV2_PTL_INSTALL_DIR/bin/pbs_benchpress --db-name $CV2_PBS_BUILD_DIR/test/tests/ptl_test_results.json -f $CV2_PTL_INSTALL_DIR/tests/functional/pbs_hook_modifyvnode_state_changes.py -t TestPbsModifyvnodeStateChanges.test_hook_state_changes_01 -p "moms=mom1:mom2"
2021-02-04 20:29:39,516 INFO     input command: pbs_benchpress --db-name /home/pbsdev/pbs/build/test/tests/ptl_test_results.json -f /opt/ptl/tests/functional/pbs_hook_modifyvnode_state_changes.py -t TestPbsModifyvnodeStateChanges.test_hook_state_changes_01 -p moms=mom1:mom2
2021-02-04 20:29:39,517 INFO     param: moms=mom1:mom2
2021-02-04 20:29:39,517 INFO     ptl version: 19.1.1.1093.g7f30851.modified
2021-02-04 20:29:39,520 INFO     platform: Linux pbsdev-centos7-mky18-server 4.19.121-linuxkit #1 SMP Tue Dec 1 17:50:32 UTC 2020 x86_64 x86_64
2021-02-04 20:29:39,520 INFO     python version: 3.6.8
2021-02-04 20:29:39,520 INFO     user: root
2021-02-04 20:29:39,520 INFO     --------------------------------------------------------------------------------
2021-02-04 20:29:39,520 INFO     Cleaning up temporary files
2021-02-04 20:29:39,521 INFO     Cleaning up /tmp dir
2021-02-04 20:29:39,521 INFO     Cleaning up /var/tmp dir
2021-02-04 20:29:39,530 INFO     ======================================================================
2021-02-04 20:29:39,530 INFO     suite name: TestPbsModifyvnodeStateChanges
2021-02-04 20:29:39,530 INFO     suite docstring:

    Test the modifyvnode hook by inducing various vnode state changes and
    inspecting the pbs log for expected values.


2021-02-04 20:29:39,530 INFO     ======================================================================
2021-02-04 20:29:39,531 INFO     ==================================================
2021-02-04 20:29:39,531 INFO      Entered TestPbsModifyvnodeStateChanges setUpClass
2021-02-04 20:29:39,531 INFO     ==================================================
2021-02-04 20:29:39,531 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser
2021-02-04 20:29:39,534 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser1
2021-02-04 20:29:39,537 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser2
2021-02-04 20:29:39,540 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser3
2021-02-04 20:29:39,543 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser4
2021-02-04 20:29:39,546 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser5
2021-02-04 20:29:39,549 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser6
2021-02-04 20:29:39,551 INFOCLI2 pbsdev-centos7-mky18-server: id pbsuser7
2021-02-04 20:29:39,554 INFOCLI2 pbsdev-centos7-mky18-server: id pbsother
2021-02-04 20:29:39,556 INFOCLI2 pbsdev-centos7-mky18-server: id pbstest
2021-02-04 20:29:39,558 INFOCLI2 pbsdev-centos7-mky18-server: id tstusr00
2021-02-04 20:29:39,561 INFOCLI2 pbsdev-centos7-mky18-server: id tstusr01
2021-02-04 20:29:39,563 INFOCLI2 pbsdev-centos7-mky18-server: id pbsoper
2021-02-04 20:29:39,565 INFOCLI2 pbsdev-centos7-mky18-server: id pbsmgr
2021-02-04 20:29:39,568 INFOCLI2 pbsdev-centos7-mky18-server: id pbsadmin
2021-02-04 20:29:39,571 INFOCLI2 pbsdev-centos7-mky18-server: id pbsdata
2021-02-04 20:29:39,574 INFOCLI2 pbsdev-centos7-mky18-server: id pbsroot
2021-02-04 20:29:39,577 INFOCLI2 pbsdev-centos7-mky18-server: id root
2021-02-04 20:29:39,580 INFOCLI2 pbsdev-centos7-mky18-server: id pbsbuild
2021-02-04 20:29:39,583 INFOCLI2 pbsdev-centos7-mky18-server: id root
2021-02-04 20:29:39,586 INFO     FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server
2021-02-04 20:29:39,594 INFO     server pbsdev-centos7-mky18-server: server operating mode set to cli
2021-02-04 20:29:39,594 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:39,603 INFO     server pbsdev-centos7-mky18-server: version 19.1.1.1093.g7f30851.modified
2021-02-04 20:29:39,604 INFO     expect action: created new action kicksched
2021-02-04 20:29:39,604 INFO     expect action: added action kicksched to server pbsdev-centos7-mky18-server
2021-02-04 20:29:39,604 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:39,817 INFO     FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server
2021-02-04 20:29:39,822 INFOCLI2 pbsdev-centos7-mky18-server: which chown
2021-02-04 20:29:39,825 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:39,834 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:39,845 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:39,862 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:39,872 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:39,883 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_logs
2021-02-04 20:29:39,893 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:29:39,908 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:39,918 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:39,929 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:39,944 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:39,954 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:39,965 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_logs
2021-02-04 20:29:39,977 INFO     FQDN name pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local differs from name provided mom1
2021-02-04 20:29:39,979 INFOCLI2 mom1: ssh mom1 ls -1 /opt/pbs/python/bin/python
2021-02-04 20:29:39,998 INFOCLI2 mom1: ssh mom1 python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))"
2021-02-04 20:29:40,079 INFO     FQDN name pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local differs from name provided mom2
2021-02-04 20:29:40,081 INFOCLI2 mom2: ssh mom2 ls -1 /opt/pbs/python/bin/python
2021-02-04 20:29:40,099 INFOCLI2 mom2: ssh mom2 python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))"
2021-02-04 20:29:40,181 INFO     FQDN name pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local differs from name provided pbsdev-centos7-mky18-server
2021-02-04 20:29:40,187 INFO     manager on pbsdev-centos7-mky18-server: set server {'pbs_license_min': 2}
2021-02-04 20:29:40,187 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server pbs_license_min=2
2021-02-04 20:29:40,206 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:40,219 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2021-02-04 20:29:40,231 INFO     running init script to stop pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'stop']
2021-02-04 20:29:40,232 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbskqb31zpz
Contents of /tmp/PtlPbskqb31zpz:
----------------------------------------
#!/bin/bash
sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d stop
----------------------------------------
2021-02-04 20:29:45,840 INFOCLI2 pbsdev-centos7-mky18-server: which rm
2021-02-04 20:29:45,843 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbskqb31zpz
2021-02-04 20:29:45,848 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2021-02-04 20:29:45,857 INFO     running init script to start pbs server on pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=1', 'PBS_START_MOM=0', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start']
2021-02-04 20:29:45,858 INFOCLI2 pbsdev-centos7-mky18-server: /tmp/PtlPbs37a9a99v
Contents of /tmp/PtlPbs37a9a99v:
----------------------------------------
#!/bin/bash
sudo -H PBS_START_SERVER=1 PBS_START_MOM=0 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start
----------------------------------------
2021-02-04 20:29:49,864 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/rm /tmp/PtlPbs37a9a99v
2021-02-04 20:29:49,872 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/server.lock
2021-02-04 20:29:49,881 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:49,892 INFO     ===================================================
2021-02-04 20:29:49,892 INFO     Completed TestPbsModifyvnodeStateChanges setUpClass
2021-02-04 20:29:49,892 INFO     ===================================================
2021-02-04 20:29:49,926 INFO     mom pbsdev-centos7-mky18-mom1 log match: searching for "alps_client" - with existence - No match
2021-02-04 20:29:50,427 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -l /etc/sgi-compute-node-release
2021-02-04 20:29:50,445 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -l /etc/sgi-known-distributions
2021-02-04 20:29:50,463 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -l /etc/sgi-compute-node-release
2021-02-04 20:29:50,481 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -l /etc/sgi-known-distributions
2021-02-04 20:29:50,498 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/cat /proc/meminfo
2021-02-04 20:29:50,504 INFOCLI2 pbsdev-centos7-mky18-server: df -k /var/spool/pbs
2021-02-04 20:29:50,508 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/mom_priv
2021-02-04 20:29:50,520 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/server_priv
2021-02-04 20:29:50,530 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H ls /var/spool/pbs/sched_priv
2021-02-04 20:29:50,544 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser ls /home/pbsuser
2021-02-04 20:29:50,554 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser1 ls /home/pbsuser1
2021-02-04 20:29:50,564 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser2 ls /home/pbsuser2
2021-02-04 20:29:50,576 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser3 ls /home/pbsuser3
2021-02-04 20:29:50,587 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser4 ls /home/pbsuser4
2021-02-04 20:29:50,598 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser5 ls /home/pbsuser5
2021-02-04 20:29:50,609 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser6 ls /home/pbsuser6
2021-02-04 20:29:50,620 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsuser7 ls /home/pbsuser7
2021-02-04 20:29:50,630 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsother ls /home/pbsother
2021-02-04 20:29:50,642 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbstest ls /home/pbstest
2021-02-04 20:29:50,654 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr00 ls /home/tstusr00
2021-02-04 20:29:50,665 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u tstusr01 ls /home/tstusr01
2021-02-04 20:29:50,676 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsoper ls /home/pbsoper
2021-02-04 20:29:50,686 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsmgr ls /home/pbsmgr
2021-02-04 20:29:50,698 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsadmin ls /home/pbsadmin
2021-02-04 20:29:50,709 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsdata ls /home/pbsdata
2021-02-04 20:29:50,720 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsroot ls /home/pbsroot
2021-02-04 20:29:50,732 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root
2021-02-04 20:29:50,743 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u pbsbuild ls /home/pbsbuild
2021-02-04 20:29:50,755 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H -u root ls /root
2021-02-04 20:29:50,767 INFOCLI2 mom2: ssh mom2 which cat
2021-02-04 20:29:50,784 INFOCLI2 mom2: ssh mom2 /usr/bin/cat /proc/meminfo
2021-02-04 20:29:50,819 INFOCLI2 mom2: ssh mom2 df -k /var/spool/pbs
2021-02-04 20:29:50,853 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs
2021-02-04 20:29:50,854 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs
2021-02-04 20:29:50,872 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/mom_priv
2021-02-04 20:29:50,898 INFOCLI2 grep'ing for server_priv in /var/spool/pbs
2021-02-04 20:29:50,898 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs
2021-02-04 20:29:50,917 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/server_priv
2021-02-04 20:29:50,944 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs
2021-02-04 20:29:50,944 INFOCLI2 mom2: ssh mom2 ls -l /var/spool/pbs
2021-02-04 20:29:50,962 INFOCLI2 mom2: ssh mom2 sudo -H ls /var/spool/pbs/sched_priv
2021-02-04 20:29:50,987 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser ls /home/pbsuser
2021-02-04 20:29:51,012 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser1 ls /home/pbsuser1
2021-02-04 20:29:51,037 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser2 ls /home/pbsuser2
2021-02-04 20:29:51,063 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser3 ls /home/pbsuser3
2021-02-04 20:29:51,089 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser4 ls /home/pbsuser4
2021-02-04 20:29:51,114 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser5 ls /home/pbsuser5
2021-02-04 20:29:51,141 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser6 ls /home/pbsuser6
2021-02-04 20:29:51,166 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsuser7 ls /home/pbsuser7
2021-02-04 20:29:51,192 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsother ls /home/pbsother
2021-02-04 20:29:51,217 INFOCLI2 mom2: ssh mom2 sudo -H -u pbstest ls /home/pbstest
2021-02-04 20:29:51,243 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr00 ls /home/tstusr00
2021-02-04 20:29:51,267 INFOCLI2 mom2: ssh mom2 sudo -H -u tstusr01 ls /home/tstusr01
2021-02-04 20:29:51,293 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsoper ls /home/pbsoper
2021-02-04 20:29:51,318 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsmgr ls /home/pbsmgr
2021-02-04 20:29:51,345 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsadmin ls /home/pbsadmin
2021-02-04 20:29:51,370 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsdata ls /home/pbsdata
2021-02-04 20:29:51,396 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsroot ls /home/pbsroot
2021-02-04 20:29:51,421 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root
2021-02-04 20:29:51,446 INFOCLI2 mom2: ssh mom2 sudo -H -u pbsbuild ls /home/pbsbuild
2021-02-04 20:29:51,471 INFOCLI2 mom2: ssh mom2 sudo -H -u root ls /root
2021-02-04 20:29:51,496 INFOCLI2 mom1: ssh mom1 which cat
2021-02-04 20:29:51,513 INFOCLI2 mom1: ssh mom1 /usr/bin/cat /proc/meminfo
2021-02-04 20:29:51,545 INFOCLI2 mom1: ssh mom1 df -k /var/spool/pbs
2021-02-04 20:29:51,579 INFOCLI2 grep'ing for mom_priv in /var/spool/pbs
2021-02-04 20:29:51,579 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs
2021-02-04 20:29:51,597 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/mom_priv
2021-02-04 20:29:51,625 INFOCLI2 grep'ing for server_priv in /var/spool/pbs
2021-02-04 20:29:51,625 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs
2021-02-04 20:29:51,645 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/server_priv
2021-02-04 20:29:51,674 INFOCLI2 grep'ing for sched_priv in /var/spool/pbs
2021-02-04 20:29:51,674 INFOCLI2 mom1: ssh mom1 ls -l /var/spool/pbs
2021-02-04 20:29:51,695 INFOCLI2 mom1: ssh mom1 sudo -H ls /var/spool/pbs/sched_priv
2021-02-04 20:29:51,719 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser ls /home/pbsuser
2021-02-04 20:29:51,747 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser1 ls /home/pbsuser1
2021-02-04 20:29:51,775 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser2 ls /home/pbsuser2
2021-02-04 20:29:51,802 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser3 ls /home/pbsuser3
2021-02-04 20:29:51,829 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser4 ls /home/pbsuser4
2021-02-04 20:29:51,855 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser5 ls /home/pbsuser5
2021-02-04 20:29:51,882 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser6 ls /home/pbsuser6
2021-02-04 20:29:51,875 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsuser7 ls /home/pbsuser7
2021-02-04 20:29:51,902 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsother ls /home/pbsother
2021-02-04 20:29:51,929 INFOCLI2 mom1: ssh mom1 sudo -H -u pbstest ls /home/pbstest
2021-02-04 20:29:51,956 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr00 ls /home/tstusr00
2021-02-04 20:29:51,982 INFOCLI2 mom1: ssh mom1 sudo -H -u tstusr01 ls /home/tstusr01
2021-02-04 20:29:52,010 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsoper ls /home/pbsoper
2021-02-04 20:29:52,038 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsmgr ls /home/pbsmgr
2021-02-04 20:29:52,066 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsadmin ls /home/pbsadmin
2021-02-04 20:29:52,093 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsdata ls /home/pbsdata
2021-02-04 20:29:52,117 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsroot ls /home/pbsroot
2021-02-04 20:29:52,146 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root
2021-02-04 20:29:52,174 INFOCLI2 mom1: ssh mom1 sudo -H -u pbsbuild ls /home/pbsbuild
2021-02-04 20:29:52,201 INFOCLI2 mom1: ssh mom1 sudo -H -u root ls /root
2021-02-04 20:29:52,228 INFO     test name: test_hook_state_changes_01 (tests.functional.pbs_hook_modifyvnode_state_changes.TestPbsModifyvnodeStateChanges)...
2021-02-04 20:29:52,228 INFO     test start time: Thu Feb  4 20:29:52 2021
2021-02-04 20:29:52,228 INFO     test docstring:

        Test each mom for state changes:
        1.  sigkill the mom; check server log for expected messages
        2.  start the mom; check server log for expected messages
        3.  verify the chain of reported current/previous states is unbroken

2021-02-04 20:29:52,228 INFO     =============================================
2021-02-04 20:29:52,228 INFO      Entered TestPbsModifyvnodeStateChanges setUp
2021-02-04 20:29:52,228 INFO     =============================================
2021-02-04 20:29:52,229 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:52,238 INFO     status on pbsdev-centos7-mky18-server: server
2021-02-04 20:29:52,238 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:52,453 INFO     manager on pbsdev-centos7-mky18-server: unset server managers
2021-02-04 20:29:52,453 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server managers
2021-02-04 20:29:52,474 INFO     manager on pbsdev-centos7-mky18-server: unset server operators
2021-02-04 20:29:52,474 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c unset server operators
2021-02-04 20:29:52,493 INFO     status on pbsdev-centos7-mky18-server: server ['managers', 'operators']
2021-02-04 20:29:52,494 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:52,708 INFO     manager on pbsdev-centos7-mky18-server: set server {'managers': (2, 'root@*,pbsmgr@*'), 'operators': (2, 'pbsoper@*')}
2021-02-04 20:29:52,708 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set server managers+="root@*,pbsmgr@*",operators+=pbsoper@*
2021-02-04 20:29:52,727 INFO     server pbsdev-centos7-mky18-server: reverting configuration to defaults
2021-02-04 20:29:52,728 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:52,944 INFO     status on pbsdev-centos7-mky18-server: job
2021-02-04 20:29:52,944 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:53,161 INFO     status on pbsdev-centos7-mky18-server: sched
2021-02-04 20:29:53,161 INFO     manager on pbsdev-centos7-mky18-server: list sched
2021-02-04 20:29:53,162 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched
2021-02-04 20:29:53,173 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:53,182 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:53,193 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:53,209 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:53,221 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:53,231 INFO     manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'}
2021-02-04 20:29:53,232 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False
2021-02-04 20:29:53,244 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:29:53,254 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:53,264 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:53,276 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:53,292 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:53,302 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:53,313 INFO     expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ...  OK
2021-02-04 20:29:53,314 INFO     delete job on pbsdev-centos7-mky18-server as root:
2021-02-04 20:29:53,314 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:53,531 INFO     expect on server pbsdev-centos7-mky18-server: job_state set 0 job ...  OK
2021-02-04 20:29:53,531 INFO     manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'}
2021-02-04 20:29:53,531 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True
2021-02-04 20:29:53,544 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:29:53,553 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:53,564 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:53,575 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:53,591 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:53,601 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:53,612 INFO     expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ...  OK
2021-02-04 20:29:53,613 INFO     status on pbsdev-centos7-mky18-server as root: reservation
2021-02-04 20:29:53,613 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f
2021-02-04 20:29:53,623 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:53,838 INFO     status on pbsdev-centos7-mky18-server: job
2021-02-04 20:29:53,839 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:54,055 INFO     status on pbsdev-centos7-mky18-server: sched
2021-02-04 20:29:54,055 INFO     manager on pbsdev-centos7-mky18-server: list sched
2021-02-04 20:29:54,056 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched
2021-02-04 20:29:54,067 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:54,078 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:54,089 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:54,105 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:54,116 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:54,126 INFO     manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'}
2021-02-04 20:29:54,126 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False
2021-02-04 20:29:54,138 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:29:54,148 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:54,159 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:54,169 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:54,186 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:54,196 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:54,206 INFO     expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ...  OK
2021-02-04 20:29:54,206 INFO     delete job on pbsdev-centos7-mky18-server as root:
2021-02-04 20:29:54,206 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:54,420 INFO     expect on server pbsdev-centos7-mky18-server: job_state set 0 job ...  OK
2021-02-04 20:29:54,421 INFO     manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'}
2021-02-04 20:29:54,421 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True
2021-02-04 20:29:54,432 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:29:54,442 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:54,451 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:54,461 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:54,477 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:54,487 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:54,498 INFO     expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ...  OK
2021-02-04 20:29:54,498 INFO     status on pbsdev-centos7-mky18-server as root: reservation
2021-02-04 20:29:54,498 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbs_rstat -f
2021-02-04 20:29:54,508 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:54,723 INFO     manager on pbsdev-centos7-mky18-server: unset server ['scheduling', 'log_events', 'mailer', 'mail_from', 'query_other_jobs', 'resources_default.ncpus', 'scheduler_iteration', 'resv_enable', 'max_array_size', 'pbs_license_max', 'pbs_license_linger_time', 'eligible_time_enable', 'max_concurrent_provision', 'sync_mom_hookfiles_timeout', 'max_job_sequence_id']
2021-02-04 20:29:54,723 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset server scheduling,log_events,mailer,mail_from,query_other_jobs,resources_default.ncpus,scheduler_iteration,resv_enable,max_array_size,pbs_license_max,pbs_license_linger_time,eligible_time_enable,max_concurrent_provision,sync_mom_hookfiles_timeout,max_job_sequence_id
2021-02-04 20:29:54,738 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook
2021-02-04 20:29:54,756 INFO     manager on pbsdev-centos7-mky18-server: delete hook m1234
2021-02-04 20:29:54,756 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c delete hook m1234
2021-02-04 20:29:54,775 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:54,989 INFO     status on pbsdev-centos7-mky18-server: node
2021-02-04 20:29:54,989 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v -a
2021-02-04 20:29:55,002 INFO     manager on pbsdev-centos7-mky18-server: delete queue workq
2021-02-04 20:29:55,002 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete queue workq
2021-02-04 20:29:55,016 INFO     manager on pbsdev-centos7-mky18-server: create queue workq {'queue_type': 'Execution', 'enabled': 'True', 'started': 'True'}
2021-02-04 20:29:55,016 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create queue workq queue_type=Execution,enabled=True,started=True
2021-02-04 20:29:55,029 INFO     status on pbsdev-centos7-mky18-server: queue workq
2021-02-04 20:29:55,030 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Qf workq@pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:55,039 INFO     manager on pbsdev-centos7-mky18-server: list sched
2021-02-04 20:29:55,040 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched
2021-02-04 20:29:55,049 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:55,063 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,074 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:55,090 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:55,101 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:55,112 INFO     manager on pbsdev-centos7-mky18-server: delete node @default
2021-02-04 20:29:55,112 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c delete node @default
2021-02-04 20:29:55,124 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook
2021-02-04 20:29:55,143 INFO     manager on pbsdev-centos7-mky18-server: set hook pbs_cgroups {'enabled': 'false'}
2021-02-04 20:29:55,143 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook pbs_cgroups enabled=false
2021-02-04 20:29:55,163 INFO     manager on pbsdev-centos7-mky18-server: set server {'default_queue': 'workq'}
2021-02-04 20:29:55,163 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server default_queue=workq
2021-02-04 20:29:55,175 INFO     status on pbsdev-centos7-mky18-server: resource
2021-02-04 20:29:55,175 INFO     manager on pbsdev-centos7-mky18-server: list resource
2021-02-04 20:29:55,175 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list resource
2021-02-04 20:29:55,185 INFOCLI  status on pbsdev-centos7-mky18-server: server license_count
2021-02-04 20:29:55,185 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:55,399 INFO     server: pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local licensed
2021-02-04 20:29:55,399 INFO     status on pbsdev-centos7-mky18-server: server pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:55,399 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:29:55,617 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python
2021-02-04 20:29:55,636 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))"
2021-02-04 20:29:55,686 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python
2021-02-04 20:29:55,703 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local python3 -c "import os;print(os.environ.get("PBS_CONF_FILE", False))"
2021-02-04 20:29:55,753 ERROR    Couldn't revert PBS_DAEMON_SERVICE_USER in pbs.conf to its default value
2021-02-04 20:29:55,758 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock
2021-02-04 20:29:55,768 INFO     scheduler pbsdev-centos7-mky18-server: reverting configuration to defaults
2021-02-04 20:29:55,768 INFO     manager on pbsdev-centos7-mky18-server: unset sched ['sched_cycle_length', 'throughput_mode', 'job_run_wait', 'opt_backfill_fuzzy', 'partition', 'sched_priv', 'sched_log', 'scheduling', 'scheduler_iteration', 'preempt_queue_prio', 'preempt_prio', 'preempt_order', 'preempt_sort', 'log_events', 'server_dyn_res_alarm']
2021-02-04 20:29:55,768 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c unset sched sched_cycle_length,throughput_mode,job_run_wait,opt_backfill_fuzzy,partition,sched_priv,sched_log,scheduling,scheduler_iteration,preempt_queue_prio,preempt_prio,preempt_order,preempt_sort,log_events,server_dyn_res_alarm
2021-02-04 20:29:55,785 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,796 INFOCLI2 pbsdev-centos7-mky18-server: which cp
2021-02-04 20:29:55,799 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /tmp/PtlPbs5799s0np /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,810 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_resource_group /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:55,821 INFO     scheduler pbsdev-centos7-mky18-server: reverting holidays file to default
2021-02-04 20:29:55,821 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_holidays /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:55,831 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_sched_config /var/spool/pbs/sched_priv/sched_config
2021-02-04 20:29:55,842 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H cmp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,853 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cp /opt/pbs/etc/pbs_dedicated /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,864 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown root /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,875 INFO     Scheduler sent signal -HUP
2021-02-04 20:29:55,875 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock
2021-02-04 20:29:55,886 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H kill -HUP 46148
2021-02-04 20:29:55,897 INFOCLI2 pbsdev-centos7-mky18-server: /opt/pbs/sbin/pbsfs -e -I default
2021-02-04 20:29:55,920 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/sched.lock
2021-02-04 20:29:55,930 INFO     status on pbsdev-centos7-mky18-server: sched default
2021-02-04 20:29:55,931 INFO     manager on pbsdev-centos7-mky18-server: list sched default
2021-02-04 20:29:55,931 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:29:55,941 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:29:55,952 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:29:55,963 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:29:55,980 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:29:55,990 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:29:56,052 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python
2021-02-04 20:29:56,087 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import sys; print(sys.platform)"
2021-02-04 20:29:56,137 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:29:56,161 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,162 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,171 ERROR    <DshUtils.run_cmd>err: ['/opt/pbs/bin/pbsnodes: Server has no node list']
2021-02-04 20:29:56,171 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which rm
2021-02-04 20:29:56,189 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue
2021-02-04 20:29:56,214 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue
2021-02-04 20:29:56,239 INFOCLI  pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list
2021-02-04 20:29:56,329 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])"
2021-02-04 20:29:56,380 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which scp
2021-02-04 20:29:56,398 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbs7u9rf63s pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/tmpaz9_mi5_PtlPbstmpcopy
2021-02-04 20:29:56,424 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local which cp
2021-02-04 20:29:56,442 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpaz9_mi5_PtlPbstmpcopy /var/spool/pbs/mom_priv/config
2021-02-04 20:29:56,468 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpaz9_mi5_PtlPbstmpcopy
2021-02-04 20:29:56,485 INFO     MoM sent signal -HUP
2021-02-04 20:29:56,485 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:29:56,511 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -HUP 16162
2021-02-04 20:29:56,556 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:29:56,582 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,582 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,592 ERROR    <DshUtils.run_cmd>err: ['/opt/pbs/bin/pbsnodes: Server has no node list']
2021-02-04 20:29:56,592 INFO     manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,592 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,606 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,607 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,617 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:56,627 INFO     expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1  got: state = state-unknown,down
2021-02-04 20:29:57,629 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:57,641 INFO     expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 2  got: state = state-unknown,down
2021-02-04 20:29:58,644 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:58,657 INFO     expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom1 attempt: 3 ...  OK
2021-02-04 20:29:58,657 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom1
2021-02-04 20:29:58,658 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom1
2021-02-04 20:29:58,716 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local ls -1 /opt/pbs/python/bin/python
2021-02-04 20:29:58,750 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import sys; print(sys.platform)"
2021-02-04 20:29:58,801 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:29:58,826 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2
2021-02-04 20:29:58,826 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:29:58,838 ERROR    <DshUtils.run_cmd>err: ['Node: pbsdev-centos7-mky18-mom2,  Error: Unknown node ']
2021-02-04 20:29:58,838 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which rm
2021-02-04 20:29:58,856 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/epilogue
2021-02-04 20:29:58,883 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/rm -f /var/spool/pbs/mom_priv/prologue
2021-02-04 20:29:58,909 INFOCLI  pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /opt/pbs/sbin/pbs_mom -s list
2021-02-04 20:29:58,999 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import tempfile;print(tempfile.mkstemp('PtlPbstmpcopy')[1])"
2021-02-04 20:29:59,052 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which scp
2021-02-04 20:29:59,071 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp /tmp/PtlPbsu0k3mol8 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/tmpavm768ioPtlPbstmpcopy
2021-02-04 20:29:59,095 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local which cp
2021-02-04 20:29:59,116 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cp /tmp/tmpavm768ioPtlPbstmpcopy /var/spool/pbs/mom_priv/config
2021-02-04 20:29:59,142 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/tmpavm768ioPtlPbstmpcopy
2021-02-04 20:29:59,160 INFO     MoM sent signal -HUP
2021-02-04 20:29:59,160 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:29:59,187 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -HUP 16105
2021-02-04 20:29:59,232 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:29:59,257 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,257 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,266 ERROR    <DshUtils.run_cmd>err: ['Node: pbsdev-centos7-mky18-mom2,  Error: Unknown node ']
2021-02-04 20:29:59,266 INFO     manager on pbsdev-centos7-mky18-server: create node pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,267 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c create node pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,281 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,281 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,290 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:29:59,300 INFO     expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2  got: state = state-unknown,down
2021-02-04 20:30:00,305 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:30:00,319 INFO     expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 2  got: state = state-unknown,down
2021-02-04 20:30:01,321 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:30:01,332 INFO     expect on server pbsdev-centos7-mky18-server: state = free node pbsdev-centos7-mky18-mom2 attempt: 3 ...  OK
2021-02-04 20:30:01,332 INFO     status on pbsdev-centos7-mky18-server: node pbsdev-centos7-mky18-mom2
2021-02-04 20:30:01,332 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/pbsnodes -s pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local -v pbsdev-centos7-mky18-mom2
2021-02-04 20:30:01,341 INFO     manager on pbsdev-centos7-mky18-server: set sched default {'opt_backfill_fuzzy': 'off'}
2021-02-04 20:30:01,341 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default opt_backfill_fuzzy=off
2021-02-04 20:30:01,357 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/server_priv/comm.lock
2021-02-04 20:30:01,366 INFO     ==============================================
2021-02-04 20:30:01,366 INFO     Completed TestPbsModifyvnodeStateChanges setUp
2021-02-04 20:30:01,366 INFO     ==============================================
2021-02-04 20:30:01,367 INFO     manager on pbsdev-centos7-mky18-server: set server {'log_events': 4095}
2021-02-04 20:30:01,367 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server log_events=4095
2021-02-04 20:30:01,378 INFO     status on pbsdev-centos7-mky18-server: hook
2021-02-04 20:30:01,378 INFO     manager on pbsdev-centos7-mky18-server: list hook
2021-02-04 20:30:01,378 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook
2021-02-04 20:30:01,397 INFO     manager on pbsdev-centos7-mky18-server: create hook p1234
2021-02-04 20:30:01,397 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c create hook p1234
2021-02-04 20:30:01,416 INFO     status on pbsdev-centos7-mky18-server: hook p1234
2021-02-04 20:30:01,417 INFO     manager on pbsdev-centos7-mky18-server: list hook p1234
2021-02-04 20:30:01,417 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook p1234
2021-02-04 20:30:01,434 INFO     status on pbsdev-centos7-mky18-server: hook p1234
2021-02-04 20:30:01,435 INFO     manager on pbsdev-centos7-mky18-server: list hook p1234
2021-02-04 20:30:01,435 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c list hook p1234
2021-02-04 20:30:01,453 INFO     manager on pbsdev-centos7-mky18-server: set hook p1234 {'event': 'modifyvnode', 'enabled': 'True', 'debug': 'True'}
2021-02-04 20:30:01,454 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c set hook p1234 event=modifyvnode,enabled=True,debug=True
2021-02-04 20:30:01,472 INFO     status on pbsdev-centos7-mky18-server: server sync_mom_hookfiles_timeout
2021-02-04 20:30:01,472 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:30:01,482 INFO     Setting sync_mom_hookfiles_timeout to 15s
2021-02-04 20:30:01,482 INFO     manager on pbsdev-centos7-mky18-server: set server {'sync_mom_hookfiles_timeout': 15}
2021-02-04 20:30:01,483 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set server sync_mom_hookfiles_timeout=15
2021-02-04 20:30:01,494 INFO     manager on pbsdev-centos7-mky18-server: import hook p1234 {'content-type': 'application/x-python', 'content-encoding': 'default', 'input-file': '/tmp/PtlPbs7veqmw6u'}
2021-02-04 20:30:01,494 INFOCLI  pbsdev-centos7-mky18-server: sudo -H /opt/pbs/bin/qmgr -c import hook p1234 application/x-python default /tmp/PtlPbs7veqmw6u
2021-02-04 20:30:01,513 INFO     server pbsdev-centos7-mky18-server: imported hook body
---

import pbs
import sys
try:
    e = pbs.event()
    v = e.vnode
    v_o = e.vnode_o
    lsct = v.last_state_change_time
    lsct_o = v_o.last_state_change_time
    state_str_buf_v = ",".join(v.extract_state_strs())
    state_str_buf_v_o = ",".join(v_o.extract_state_strs())
    state_int_buf_v = ','.join([str(_) for _ in v.extract_state_ints()])
    state_int_buf_v_o = ','.join(
        [str(_) for _ in v_o.extract_state_ints()])
    # print show_vnode_state record (bi consumable)
    svs1_data = "v.state_hex=%s v_o.state_hex=%s v.state_strs=%s "                     "v_o.state_strs=%s" %                     (hex(v.state), hex(v_o.state), state_str_buf_v,
                 state_str_buf_v_o)
    svs2_data = "v.state_ints=%s v_o.state_ints=%s v.lsct=%s "                     "v_o.lsct=%s" %                     (state_int_buf_v, state_int_buf_v_o, str(lsct),
                 str(lsct_o))
    svs_data = "%s %s" % (svs1_data, svs2_data)
    pbs.logmsg(pbs.LOG_DEBUG,
               "show_vnode_state;name=%s %s" % (v.name, svs_data))
    # print additional hook parameter values
    pbs.logmsg(pbs.LOG_DEBUG, "name: v=%s, v_o=%s" % (v.name, v_o.name))
    pbs.logmsg(pbs.LOG_DEBUG,
               "state: v=%s, v_o=%s" % (hex(v.state), hex(v_o.state)))
    pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: v=%s, v_o=%s" % (
               str(lsct), str(lsct_o)))
    pbs.logmsg(pbs.LOG_DEBUG,
               "comment: v=%s, v_o=%s" % (v.comment, v_o.comment))
    pbs.logmsg(pbs.LOG_DEBUG,
               "aoe: v=%s, v_o=%s" % (v.current_aoe, v_o.current_aoe))
    pbs.logmsg(pbs.LOG_DEBUG, "in_mvn_host: v=%s, v_o=%s" % (
               v.in_multivnode_host, v_o.in_multivnode_host))
    pbs.logmsg(pbs.LOG_DEBUG, "jobs: v=%s, v_o=%s" % (v.jobs, v_o.jobs))
    pbs.logmsg(pbs.LOG_DEBUG, "Mom: v=%s, v_o=%s" % (v.Mom, v_o.Mom))
    pbs.logmsg(pbs.LOG_DEBUG,
               "ntype: v=%s, v_o=%s" % (hex(v.ntype), hex(v_o.ntype)))
    pbs.logmsg(pbs.LOG_DEBUG, "pcpus: v=%s, v_o=%s" % (v.pcpus, v_o.pcpus))
    pbs.logmsg(pbs.LOG_DEBUG,
               "pnames: v=%s, v_o=%s" % (v.pnames, v_o.pnames))
    pbs.logmsg(pbs.LOG_DEBUG, "Port: v=%s, v_o=%s" % (v.Port, v_o.Port))
    pbs.logmsg(pbs.LOG_DEBUG,
               "Priority: v=%s, v_o=%s" % (v.Priority, v_o.Priority))
    pbs.logmsg(pbs.LOG_DEBUG, "provision_enable: v=%s, v_o=%s" % (
               v.provision_enable, v_o.provision_enable))
    pbs.logmsg(pbs.LOG_DEBUG, "queue: v=%s, v_o=%s" % (v.queue, v_o.queue))
    pbs.logmsg(pbs.LOG_DEBUG, "res_assigned: v=%s, v_o=%s" % (
               v.resources_assigned, v_o.resources_assigned))
    pbs.logmsg(pbs.LOG_DEBUG, "res_avail: v=%s, v_o=%s" % (
               v.resources_available, v_o.resources_available))
    pbs.logmsg(pbs.LOG_DEBUG, "resv: v=%s, v_o=%s" % (v.resv, v_o.resv))
    pbs.logmsg(pbs.LOG_DEBUG, "resv_enable: v=%s, v_o=%s" % (
               v.resv_enable, v_o.resv_enable))
    pbs.logmsg(pbs.LOG_DEBUG,
               "sharing: v=%s, v_o=%s" % (v.sharing, v_o.sharing))
    # sanity test some values
    if (lsct < lsct_o) or (lsct_o <= 0):
        e.reject("last_state_change_time: bad timestamp value")
    else:
        pbs.logmsg(pbs.LOG_DEBUG, "last_state_change_time: good times")
    if (v.name != v_o.name) or (not v.name):
        e.reject(
            "name: vnode and vnode_o name values are null or mismatched")
    else:
        pbs.logmsg(pbs.LOG_DEBUG, "name: good names")
    if (isinstance(v.state, int)) and (isinstance(v_o.state, int)):
        pbs.logmsg(pbs.LOG_DEBUG, "state: good states")
    else:
        e.reject("state: bad state value")
    if len(v.extract_state_strs()) == len(v.extract_state_ints()):
        pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v sets")
    else:
        e.reject("state sets: bad v sets")
    if len(v_o.extract_state_strs()) == len(v_o.extract_state_ints()):
        pbs.logmsg(pbs.LOG_DEBUG, "state sets: good v_o sets")
    else:
        e.reject("state sets: bad v_o sets")
    e.accept()
except SystemExit:
    pass
except:
    pbs.event().reject("%s hook failed with %s" % (
                       pbs.event().hook_name, sys.exc_info()[:2]))
---
2021-02-04 20:30:01,513 INFO     MoM sent signal -KILL
2021-02-04 20:30:01,513 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:30:01,538 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H kill -KILL 16162
2021-02-04 20:30:01,568 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK
2021-02-04 20:30:01,571 INFO     server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK
2021-02-04 20:30:01,573 INFO     server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK
2021-02-04 20:30:01,576 INFO     server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK
2021-02-04 20:30:01,580 INFO     server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK
2021-02-04 20:30:01,583 INFO     server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK
2021-02-04 20:30:01,586 INFO     server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK
2021-02-04 20:30:01,590 INFO     server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK
2021-02-04 20:30:01,592 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK
2021-02-04 20:30:01,595 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK
2021-02-04 20:30:01,598 INFO     server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node down" - with existence... OK
2021-02-04 20:30:01,600 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK
2021-02-04 20:30:01,602 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK
2021-02-04 20:30:01,604 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK
2021-02-04 20:30:01,621 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2021-02-04 20:30:01,647 INFO     running init script to start pbs mom on pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start']
2021-02-04 20:30:01,648 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbsj6j34vzu pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local:/tmp/PtlPbsj6j34vzu
2021-02-04 20:30:01,671 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /tmp/PtlPbsj6j34vzu
Contents of /tmp/PtlPbsj6j34vzu:
----------------------------------------
#!/bin/bash
sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start
----------------------------------------
2021-02-04 20:30:01,777 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbsj6j34vzu
2021-02-04 20:30:01,811 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:30:01,840 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match
2021-02-04 20:30:03,088 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2
2021-02-04 20:30:04,343 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3
2021-02-04 20:30:05,596 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK
2021-02-04 20:30:05,604 INFO     server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK
2021-02-04 20:30:05,607 INFO     server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK
2021-02-04 20:30:05,609 INFO     server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK
2021-02-04 20:30:05,612 INFO     server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK
2021-02-04 20:30:05,615 INFO     server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK
2021-02-04 20:30:05,617 INFO     server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK
2021-02-04 20:30:05,620 INFO     server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK
2021-02-04 20:30:05,623 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK
2021-02-04 20:30:05,626 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK
2021-02-04 20:30:05,630 INFO     server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local;node up" - with existence... OK
2021-02-04 20:30:05,633 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK
2021-02-04 20:30:05,636 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK
2021-02-04 20:30:05,639 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK
2021-02-04 20:30:05,639 INFO     checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom1 start=1612470601.513361 end=1612470605.639778
2021-02-04 20:30:05,673 INFO     server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom1" - on all matches  - with existence... OK
2021-02-04 20:30:05,674 INFO     MoM sent signal -KILL
2021-02-04 20:30:05,674 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:30:05,700 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H kill -KILL 16105
2021-02-04 20:30:05,728 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK
2021-02-04 20:30:05,730 INFO     server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK
2021-02-04 20:30:05,733 INFO     server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK
2021-02-04 20:30:05,736 INFO     server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK
2021-02-04 20:30:05,739 INFO     server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK
2021-02-04 20:30:05,742 INFO     server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK
2021-02-04 20:30:05,745 INFO     server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK
2021-02-04 20:30:05,748 INFO     server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK
2021-02-04 20:30:05,751 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK
2021-02-04 20:30:05,754 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK
2021-02-04 20:30:05,757 INFO     server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node down" - with existence... OK
2021-02-04 20:30:05,759 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x2" - with existence... OK
2021-02-04 20:30:05,762 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_DOWN,ND_STATE_VNODE_UNAVAILABLE" - with existence... OK
2021-02-04 20:30:05,765 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=2,409903" - with existence... OK
2021-02-04 20:30:05,782 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H ls -l /opt/pbs/libexec/pbs_init.d
2021-02-04 20:30:05,806 INFO     running init script to start pbs mom on pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local using /etc/pbs.conf init_cmd=['sudo', '-H', 'PBS_START_SERVER=0', 'PBS_START_MOM=1', 'PBS_START_SCHED=0', 'PBS_START_COMM=0', '/opt/pbs/libexec/pbs_init.d', 'start']
2021-02-04 20:30:05,807 INFOCLI2 pbsdev-centos7-mky18-server: /usr/bin/scp -p /tmp/PtlPbs6q4ih7h3 pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local:/tmp/PtlPbs6q4ih7h3
2021-02-04 20:30:05,832 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /tmp/PtlPbs6q4ih7h3
Contents of /tmp/PtlPbs6q4ih7h3:
----------------------------------------
#!/bin/bash
sudo -H PBS_START_SERVER=0 PBS_START_MOM=1 PBS_START_SCHED=0 PBS_START_COMM=0 /opt/pbs/libexec/pbs_init.d start
----------------------------------------
2021-02-04 20:30:05,942 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/rm /tmp/PtlPbs6q4ih7h3
2021-02-04 20:30:05,981 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local sudo -H /usr/bin/cat /var/spool/pbs/mom_priv/mom.lock
2021-02-04 20:30:06,010 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - No match
2021-02-04 20:30:07,244 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 2
2021-02-04 20:30:08,491 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence - attempt 3
2021-02-04 20:30:09,733 INFO     server pbsdev-centos7-mky18-server log match: searching for "set_vnode_state;vnode.state=" - with existence... OK
2021-02-04 20:30:09,742 INFO     server pbsdev-centos7-mky18-server log match: searching for "show_vnode_state;name=" - with existence... OK
2021-02-04 20:30:09,745 INFO     server pbsdev-centos7-mky18-server log match: searching for "name: v=" - with existence... OK
2021-02-04 20:30:09,748 INFO     server pbsdev-centos7-mky18-server log match: searching for "state: v=" - with existence... OK
2021-02-04 20:30:09,750 INFO     server pbsdev-centos7-mky18-server log match: searching for "last_state_change_time: v=" - with existence... OK
2021-02-04 20:30:09,753 INFO     server pbsdev-centos7-mky18-server log match: searching for "good times" - with existence... OK
2021-02-04 20:30:09,755 INFO     server pbsdev-centos7-mky18-server log match: searching for "good names" - with existence... OK
2021-02-04 20:30:09,758 INFO     server pbsdev-centos7-mky18-server log match: searching for "good states" - with existence... OK
2021-02-04 20:30:09,760 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v sets" - with existence... OK
2021-02-04 20:30:09,763 INFO     server pbsdev-centos7-mky18-server log match: searching for "good v_o sets" - with existence... OK
2021-02-04 20:30:09,766 INFO     server pbsdev-centos7-mky18-server log match: searching for "Node;pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local;node up" - with existence... OK
2021-02-04 20:30:09,769 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_hex=0x0" - with existence... OK
2021-02-04 20:30:09,772 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_strs=ND_STATE_FREE" - with existence... OK
2021-02-04 20:30:09,776 INFO     server pbsdev-centos7-mky18-server log match: searching for "v.state_ints=0" - with existence... OK
2021-02-04 20:30:09,776 INFO     checkpreviousStateChain search_string=;show_vnode_state;name=pbsdev-centos7-mky18-mom2 start=1612470605.6741247 end=1612470609.7761815
2021-02-04 20:30:09,808 INFO     server pbsdev-centos7-mky18-server log match: searching for ";show_vnode_state;name=pbsdev-centos7-mky18-mom2" - on all matches  - with existence... OK
2021-02-04 20:30:09,809 INFO     ================================================
2021-02-04 20:30:09,810 INFO      Entered TestPbsModifyvnodeStateChanges tearDown
2021-02-04 20:30:09,810 INFO     ================================================
2021-02-04 20:30:09,810 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -Bf pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:30:09,821 INFO     status on pbsdev-centos7-mky18-server: job
2021-02-04 20:30:09,821 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:30:10,033 INFO     status on pbsdev-centos7-mky18-server: sched
2021-02-04 20:30:10,033 INFO     manager on pbsdev-centos7-mky18-server: list sched
2021-02-04 20:30:10,034 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched
2021-02-04 20:30:10,044 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:30:10,054 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:30:10,066 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:30:10,082 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:30:10,093 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:30:10,105 INFO     manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'False'}
2021-02-04 20:30:10,105 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=False
2021-02-04 20:30:10,117 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:30:10,127 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:30:10,139 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:30:10,150 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:30:10,166 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:30:10,177 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:30:10,188 INFO     expect on server pbsdev-centos7-mky18-server: state != scheduling sched default ...  OK
2021-02-04 20:30:10,188 INFO     delete job on pbsdev-centos7-mky18-server as root:
2021-02-04 20:30:10,188 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qstat -f @pbsdev-centos7-mky18-server.pbsdev-centos7-mky18.local
2021-02-04 20:30:10,402 INFO     expect on server pbsdev-centos7-mky18-server: job_state set 0 job ...  OK
2021-02-04 20:30:10,403 INFO     manager on pbsdev-centos7-mky18-server as root: set sched default {'scheduling': 'True'}
2021-02-04 20:30:10,403 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c set sched default scheduling=True
2021-02-04 20:30:10,416 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched default
2021-02-04 20:30:10,426 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:30:10,436 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:30:10,448 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:30:10,465 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:30:10,475 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:30:10,486 INFO     expect on server pbsdev-centos7-mky18-server: scheduling = True sched default ...  OK
2021-02-04 20:30:10,487 INFO     manager on pbsdev-centos7-mky18-server: list sched
2021-02-04 20:30:10,487 INFOCLI  pbsdev-centos7-mky18-server: /opt/pbs/bin/qmgr -c list sched
2021-02-04 20:30:10,498 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/chown -R root /var/spool/pbs/sched_priv
2021-02-04 20:30:10,508 INFO     scheduler pbsdev-centos7-mky18-server:  setting dedicated time file to /var/spool/pbs/sched_priv/dedicated_time
2021-02-04 20:30:10,520 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /opt/pbs/sbin/pbsfs
2021-02-04 20:30:10,537 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/resource_group
2021-02-04 20:30:10,548 INFOCLI2 pbsdev-centos7-mky18-server: sudo -H /usr/bin/cat /var/spool/pbs/sched_priv/holidays
2021-02-04 20:30:10,559 INFO     =================================================
2021-02-04 20:30:10,559 INFO     Completed TestPbsModifyvnodeStateChanges tearDown
2021-02-04 20:30:10,559 INFO     =================================================
2021-02-04 20:30:10,561 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))"
2021-02-04 20:30:10,616 INFOCLI2 pbsdev-centos7-mky18-mom1: ssh pbsdev-centos7-mky18-mom1.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())"
2021-02-04 20:30:10,671 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform;print(' '.join(platform.uname()))"
2021-02-04 20:30:10,726 INFOCLI2 pbsdev-centos7-mky18-mom2: ssh pbsdev-centos7-mky18-mom2.pbsdev-centos7-mky18.local /usr/bin/python3 -c "import platform; print(platform.platform())"
2021-02-04 20:30:10,783 INFO     ok

2021-02-04 20:30:10,783 INFO     ================================================================================
run: 1, succeeded: 1, failed: 0, errors: 0, skipped: 0, timedout: 0
Tests run in 0:00:31.253363
2021-02-04 20:30:10,783 INFO     Cleaning up temporary files
2021-02-04 20:30:10,784 INFO     Cleaning up /tmp dir
2021-02-04 20:30:10,784 INFO     Cleaning up /var/tmp dir

real	0m31.545s
user	0m7.514s
sys	0m2.826s
[02/04 20:30:10] pbsdev@pbsdev-centos7-mky18-server:~ 53%