SSH connection to data container does not work #340
Description
🗣️ Foreword
Thank for taking the time to fill this bug report fully. Without it we may not be able to fix the bug, and the issue may be closed without resolution.
👻 Brief Description
PAM prevents SSH login as root to data container (alamlinux:9)
Version
kitchen-dokken (2.20.7)
Environment
cinc version
Cinc Workstation version: 24.12.1073
Cookstyle version: 7.32.8
Cinc Client version: 18.6.2
Cinc Auditor version: 5.22.65
Cinc CLI version: 5.6.16
Biome version: 1.6.821
Test Kitchen version: 3.6.0
Remote docker: docker:28.1.1-dind
Scenario
kitchen converge for empty cookbook fails due to rsync error
kitchen converge -l debug
-----> Starting Test Kitchen (v3.6.0)
D Policyfile found at /wk/cb4/Policyfile.rb, using Policyfile to resolve cookbook dependencies
-----> Creating <default-ubuntu-2204>...
D driver - pulling dokken/ubuntu-22.04:latest
D ~/.docker/config.json does not exist
/opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/lockfile-2.1.3/lib/lockfile.rb:308: warning: finalizer references object to be finalized
D driver - pulling cincproject/cinc:18.7.3
/opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/lockfile-2.1.3/lib/lockfile.rb:308: warning: finalizer references object to be finalized
D Chef container does not exist, creating a new Chef container
D driver - creating volume container chef-18.7.3 from cincproject/cinc:18.7.3
Creating container chef-18.7.3
D driver - create_container args {"name"=>"chef-18.7.3", "Cmd"=>"true", "Image"=>"cincproject/cinc:18.7.3", "HostConfig"=>{"NetworkMode"=>"dokken"}, "Env"=>["TEST_KITCHEN=1"], "Platform"=>""}
Creating kitchen sandbox at /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204
Creating verifier sandbox at /root/.dokken/verifier_sandbox/0e4720e34f-default-ubuntu-2204
D driver - calling create_data_image
D driver - creating 0e4720e34f-default-ubuntu-2204-data
Creating container 0e4720e34f-default-ubuntu-2204-data
D driver - create_container args {"name"=>"0e4720e34f-default-ubuntu-2204-data", "Image"=>"dokken/kitchen-cache:latest", "HostConfig"=>{"PortBindings"=>nil, "PublishAllPorts"=>true, "NetworkMode"=>"bridge"}, "NetworkingConfig"=>{"EndpointsConfig"=>{"dokken"=>{"Aliases"=>["dokken"]}}}, "Env"=>["TEST_KITCHEN=1"], "Platform"=>""}
Building work image..
D driver - Building work image from dokken/ubuntu-22.04:latest
D driver - starting 0e4720e34f-default-ubuntu-2204
D driver - privileged mode is not supported with user namespaces enabled
D driver - changing UsernsMode from '' to 'host'
Creating container 0e4720e34f-default-ubuntu-2204
D driver - create_container args {"name"=>"0e4720e34f-default-ubuntu-2204", "Cmd"=>["/bin/systemd"], "Image"=>"0e4720e34f-default-ubuntu-2204:latest", "Hostname"=>"dokken", "Env"=>["TEST_KITCHEN=1"], "ExposedPorts"=>nil, "Volumes"=>{}, "HostConfig"=>{"Privileged"=>true, "VolumesFrom"=>["chef-18.7.3", "0e4720e34f-default-ubuntu-2204-data"], "Binds"=>[], "Dns"=>nil, "DnsSearch"=>nil, "Links"=>[], "CapAdd"=>[], "CapDrop"=>[], "SecurityOpt"=>[], "NetworkMode"=>"dokken", "PortBindings"=>nil, "Tmpfs"=>{}, "Memory"=>0, "UsernsMode"=>"host"}, "NetworkingConfig"=>{"EndpointsConfig"=>{"dokken"=>{"Aliases"=>["dokken"]}}}, "Platform"=>""}
Finished creating <default-ubuntu-2204> (0m57.75s).
-----> Converging <default-ubuntu-2204>...
D Checking if we need to prompt for license acceptance on product: chef-workstation version: 18.7.3.
D Reading products and relationships...
D Successfully read products and relationships
D License acceptance required for chef-workstation version: 18.7.3. Prompting
D Chef License accepted with no persistence
Creating kitchen sandbox in /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204
Policy lock file doesn't exist, running `/opt/cinc-workstation/bin/chef-cli install` for Policyfile /wk/cb4/Policyfile.rb...
D [local command] BEGIN (/opt/cinc-workstation/bin/chef-cli install /wk/cb4/Policyfile.rb --chef-license accept-no-persist)
Redirecting to cinc-cli
Building policy cb4
Expanded run list: recipe[cb4::default]
Caching Cookbooks...
Installing cb4 >= 0.0.0 from path
Lockfile written to /wk/cb4/Policyfile.lock.json
Policy revision id: 8c321e21bfc6a0cbc48b4e5e9e8ce740df655d912f88c93b4a80b7826758c7ee
D [local command] END (0m3.25s)
Updating policy lock using `/opt/cinc-workstation/bin/chef-cli update`
D [local command] BEGIN (/opt/cinc-workstation/bin/chef-cli update /wk/cb4/Policyfile.rb --chef-license accept-no-persist)
Redirecting to cinc-cli
Building policy cb4
Expanded run list: recipe[cb4::default]
Caching Cookbooks...
Installing cb4 >= 0.0.0 from path
Lockfile written to /wk/cb4/Policyfile.lock.json
Policy revision id: 8c321e21bfc6a0cbc48b4e5e9e8ce740df655d912f88c93b4a80b7826758c7ee
D [local command] END (0m2.75s)
Preparing dna.json
D Creating dna.json from {:policy_name=>"cb4", :policy_group=>"local"}
Exporting cookbook dependencies from Policyfile /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204 using `/opt/cinc-workstation/bin/chef-cli export`...
D [local command] BEGIN (/opt/cinc-workstation/bin/chef-cli export /wk/cb4/Policyfile.rb /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204 --force --chef-license accept-no-persist)
Redirecting to cinc-cli
Exported policy 'cb4' to /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204
To converge this system with the exported policy, run:
cd /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204
cinc-client -z
D [local command] END (0m1.02s)
Removing non-cookbook files before transfer
Preparing validation.pem
D Using a dummy validation.pem
Preparing client.rb
D Creating client.rb from {:node_name=>"default-ubuntu-2204", :checksum_path=>"/opt/kitchen/checksums", :file_cache_path=>"/opt/kitchen/cache", :file_backup_path=>"/opt/kitchen/backup", :cookbook_path=>["/opt/kitchen/cookbooks", "/opt/kitchen/site-cookbooks"], :data_bag_path=>"/opt/kitchen/data_bags", :environment_path=>"/opt/kitchen/environments", :node_path=>"/opt/kitchen/nodes", :role_path=>"/opt/kitchen/roles", :client_path=>"/opt/kitchen/clients", :user_path=>"/opt/kitchen/users", :validation_key=>"/opt/kitchen/validation.pem", :client_key=>"/opt/kitchen/client.pem", :chef_server_url=>"http://127.0.0.1:8889", :encrypted_data_bag_secret=>"/opt/kitchen/encrypted_data_bag_secret", :treat_deprecation_warnings_as_errors=>false, :chef_license=>"accept-no-persist", :named_run_list=>{}}
Transferring files to <default-ubuntu-2204>
D candidate_ip - 172.18.0.2
D candidate_ssh_port - 32768
D ssh_ip : dind
D ssh_port : 32768
D rsync_cmd :/usr/bin/rsync -a -e 'ssh -2 -i /tmp/dokken/0/id_rsa -o CheckHostIP=no -o Compression=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -p 32768' /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/Policyfile.lock.json /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/README.md /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/cache /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/client.rb /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/cookbook_artifacts /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/dna.json /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/policies /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/policy_groups /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/run_command /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204/validation.pem root@dind:/opt/kitchen:
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(231) [sender=3.2.7]
D Attempting to execute command - try 1 of 1.
sh: 0: cannot open /opt/kitchen/run_command: No such file
D Cleaning up local sandbox in /root/.dokken/kitchen_sandbox/0e4720e34f-default-ubuntu-2204
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Converge failed on instance <default-ubuntu-2204>. Please see .kitchen/logs/default-ubuntu-2204.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
D ------Exception-------
D Class: Kitchen::ActionFailed
D Message: 1 actions failed.
>>>>>> Converge failed on instance <default-ubuntu-2204>. Please see .kitchen/logs/default-ubuntu-2204.log for more details
D ----------------------
D ------Backtrace-------
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:181:in `report_errors'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:172:in `run_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command/action.rb:35:in `block in call'
D /opt/cinc-workstation/embedded/lib/ruby/3.1.0/benchmark.rb:296:in `measure'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command/action.rb:33:in `call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/cli.rb:52:in `perform'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/cli.rb:198:in `block (2 levels) in <class:CLI>'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/thor-1.2.2/lib/thor/command.rb:27:in `run'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in `invoke_command'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/thor-1.2.2/lib/thor.rb:392:in `dispatch'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/thor-1.2.2/lib/thor/base.rb:485:in `start'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/bin/kitchen:11:in `block in <top (required)>'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/errors.rb:183:in `with_friendly_errors'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/bin/kitchen:11:in `<top (required)>'
D /opt/cinc-workstation/bin/kitchen:439:in `load'
D /opt/cinc-workstation/bin/kitchen:439:in `<main>'
D ----End Backtrace-----
D -Composite Exception--
D Class: Kitchen::InstanceFailure
D Message: Converge failed on instance <default-ubuntu-2204>. Please see .kitchen/logs/default-ubuntu-2204.log for more details
D ----------------------
D ------Backtrace-------
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/kitchen-dokken-2.20.7/lib/kitchen/provisioner/dokken.rb:75:in `rescue in call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/kitchen-dokken-2.20.7/lib/kitchen/provisioner/dokken.rb:74:in `call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:419:in `block in converge_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:563:in `synchronize_or_call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:524:in `block in action'
D /opt/cinc-workstation/embedded/lib/ruby/3.1.0/benchmark.rb:296:in `measure'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:523:in `action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:414:in `converge_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:392:in `block (2 levels) in transition_to'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/lifecycle_hooks.rb:47:in `run_with_hooks'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:391:in `block in transition_to'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:390:in `each'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:390:in `transition_to'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:139:in `converge'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:195:in `public_send'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:195:in `run_action_in_thread'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:166:in `block (2 levels) in run_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
D ----End Backtrace-----
D ---Nested Exception---
D Class: Kitchen::ActionFailed
D Message: Docker Exec (2) for command: [sh /opt/kitchen/run_command]
D ----------------------
D ------Backtrace-------
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/kitchen-dokken-2.20.7/lib/kitchen/provisioner/dokken.rb:75:in `rescue in call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/kitchen-dokken-2.20.7/lib/kitchen/provisioner/dokken.rb:74:in `call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:419:in `block in converge_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:563:in `synchronize_or_call'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:524:in `block in action'
D /opt/cinc-workstation/embedded/lib/ruby/3.1.0/benchmark.rb:296:in `measure'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:523:in `action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:414:in `converge_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:392:in `block (2 levels) in transition_to'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/lifecycle_hooks.rb:47:in `run_with_hooks'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:391:in `block in transition_to'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:390:in `each'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:390:in `transition_to'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/instance.rb:139:in `converge'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:195:in `public_send'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:195:in `run_action_in_thread'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/test-kitchen-3.6.0/lib/kitchen/command.rb:166:in `block (2 levels) in run_action'
D /opt/cinc-workstation/embedded/lib/ruby/gems/3.1.0/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
D ----End Backtrace-----
Steps to Reproduce
Testing SSH connection (after converge error)
ssh -2 -i /tmp/dokken/0/id_rsa -o CheckHostIP=no -o Compression=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -p 32768 root@dind cat /etc/os-release ; echo $?
255
Creating a test data container with the UsePAM=no option
docker run -d -P dokken/kitchen-cache:latest /usr/sbin/sshd -D -p 22 -o UseDNS=no -o UsePrivilegeSeparation=no -o MaxAuthTries=60 -o UsePAM=no
docker -H dind ps --no-trunc
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d2c042f6eb3963a596439268c84cdbd9365eb7d771b14222cc10042f98fd53d7 dokken/kitchen-cache:latest "/usr/sbin/sshd -D -p 22 -o UseDNS=no -o UsePrivilegeSeparation=no -o MaxAuthTries=60 -o UsePAM=no" 9 seconds ago Up 9 seconds 0.0.0.0:32769->22/tcp, [::]:32769->22/tcp amazing_lamport
07f0ec21387b74d822988629160171262a4d0d9c2ff9598abc7c22daf95a450a 0e4720e34f-default-ubuntu-2204:latest "/bin/systemd" 5 minutes ago Up 5 minutes 0e4720e34f-default-ubuntu-2204
19cca9173ce3906c2a0b63c4eccf74fa355513e9106ce0f40b7d279df1d523eb dokken/kitchen-cache:latest "/usr/sbin/sshd -D -p 22 -o UseDNS=no -o UsePrivilegeSeparation=no -o MaxAuthTries=60" 5 minutes ago Up 5 minutes 0.0.0.0:32768->22/tcp, [::]:32768->22/tcp 0e4720e34f-default-ubuntu-2204-data
SSH connection works
ssh -2 -i /tmp/dokken/0/id_rsa -o CheckHostIP=no -o Compression=no -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -p 32769 root@dind cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.5 (Teal Serval)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.5"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.5 (Teal Serval)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"
ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.5"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.5"
SUPPORT_END=2032-06-01
Expected Result
Dockerfile generated by the function data_dockerfile from lib/kitchen/helpers.rb should allow SSH login as root.
Actual Result
cat /tmp/dokken/Dockerfile
FROM almalinux:9
MAINTAINER Sean OMeara "[email protected]"
ENV LANG en_US.UTF-8
RUN dnf -y install tar rsync openssh-server passwd git
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''
# uncomment to debug cert issues
# RUN echo 'root:dokkendokkendokken' | chpasswd
# RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
# RUN sed 's@session *required *pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
RUN mkdir -p /root/.ssh/
COPY authorized_keys /root/.ssh/authorized_keys
RUN chmod 700 /root/.ssh/
RUN chmod 600 /root/.ssh/authorized_keys
EXPOSE 22
CMD [ "/usr/sbin/sshd", "-D", "-p", "22", "-o", "UseDNS=no", "-o", "UsePrivilegeSeparation=no", "-o", "MaxAuthTries=60" ]
VOLUME /opt/kitchen
VOLUME /opt/verifier
➕ Additional context
Related issues #44