This article is an AI-assisted translation of the original French content.

This post presents an experiment in step-by-step debug execution of a Java application running on a remote server using an SSH tunnel.

«Debugging a Remote JVM»: What are we talking about? Link to heading

When an application misbehaves (explicit error or unexpected result), several tools or methods can be used to understand what is going wrong and modify the code accordingly:

  • reading logs
  • running / writing specific tests
  • consulting observability reports or performing a JDK Flight Recorder recording
  • checking data in databases, files, …
  • running in debug mode (step by step)

Debug execution is well known on the developer’s workstation for running code step by step and understanding what is going wrong. If the problem is not reproducible on the developer’s workstation, let us explore here how to perform the same step-by-step execution for an application deployed on a remote server. The step-by-step execution is controlled from the developer’s IDE. This is summarized by the following diagram:

graph LR subgraph "Developer workstation" subgraph IDE Debugger end end subgraph "Remote server" subgraph "Remote JVM" Application end end Debugger --"controls step-by-step execution"--> Application

Why an SSH Tunnel? Link to heading

The communication between the Debugger in the IDE and the remote JVM hosting the application takes place via TCP network exchanges. The JVM’s listening port is configurable in the application’s launch command (see below). The port used by the IDE’s debugger is chosen randomly by the latter.

If the IDE’s debugger can directly reach the remote VM on the configured port (by following the JetBrains tutorial in the references below), then the required ports are open and there is no need to follow this tutorial further, unless the network context is not secure (see Precautions).

If, on the other hand, as in many companies, only certain ports (already in use) are open on machines, remote debugging will not work directly and it will be necessary to use an SSH tunnel for communications between the debugger and the remote JVM. That is the subject of what follows.

Architecture with the SSH Tunnel Link to heading

Here an SSH tunnel with local port forwarding will be used. This tunnel will encapsulate the TCP network traffic between the IDE’s debugger and the remote JVM within the SSH communications between the two machines (since these are assumed to be allowed here). In practice, the port on which the JVM listens for debugger instructions will appear as a local port on the machine and will be used as such by the IDE’s debugger.

Implementation on an Example Link to heading

In the following example, user fabrice wants to remotely debug from his IntelliJ IDE a Java application running on a remote machine identified by its IP 192.168.0.82. The JVM will listen for debugger instructions on port 5005, which will be “linked by the SSH tunnel” to port 50005 on the local machine.

Debugger architecture with SSH tunnel

Prerequisites Link to heading

  • user fabrice must be able to establish an SSH connection to machine 192.168.0.82: ssh fabrice@192.168.0.82
  • on the remote machine, the user must be able to restart the Java test application by modifying its command line

Test Application Link to heading

The test application is a simple Java web application based on Spring Boot with a single class:

@SpringBootApplication
public class RemoteDebugApplication {

	public static void main(String[] args) {
		SpringApplication.run(RemoteDebugApplication.class, args);
	}

	@Controller
    record DebugedController(SimpleAsyncTaskExecutor executor) {

		public DebugedController() {
			this(new SimpleAsyncTaskExecutor());
			executor.setVirtualThreads(true);
		}

		@GetMapping("/test")
		ResponseBodyEmitter test(){
			ResponseBodyEmitter emitter = new ResponseBodyEmitter();
			executor.execute(new RunnableEmmiter(emitter));
			return emitter;
		}
	}

	static class RunnableEmmiter implements Runnable{

		public static final Duration PAUSE_TIME = Duration.ofMillis(50);
		private final ResponseBodyEmitter emitter;
		private boolean stop = false;

        RunnableEmmiter(ResponseBodyEmitter emitter) {
            this.emitter = emitter;
        }

        @Override
		public void run() {
			while (!stop) {
                try {
					emitter.send(LocalDateTime.now());
					emitter.send("\n");
                    Thread.sleep(PAUSE_TIME);
                } catch (InterruptedException | IOException e) {
                    stop = true;
                }
            }
			emitter.complete();
		}
	}
}

The application inherits from the Spring Boot parent pom (org.springframework.boot:spring-boot-starter-parent) and contains a single dependency on org.springframework.boot:spring-boot-starter-web.

It exposes a single endpoint GET /test accessible by everyone that indefinitely serves timestamps to the client until the application is stopped or the process is interrupted (allows verifying that the boolean value can be modified via the remote debugger).

Local Verification Link to heading

This consists of launching the application locally (it will behave the same locally as remotely) in debug mode in the IDE and setting a breakpoint at line 51 on emitter.send(LocalDateTime.now());. By default the application listens on port 8080, so timestamps can be triggered by calling GET http://localhost:8080/test with a curl-type client. The debug view is set up and execution stops at the breakpoint.

Local debug view

Verify that step-by-step execution works and that the loop runs indefinitely by using Step over while on the client side a new timestamp is received at each iteration. You can also verify that by changing the value of the boolean stop to true via the debugger and then resuming execution, the program exits the loop and the HTTP response is closed.

Deploying the Application on the Remote Server Link to heading

The application will be run as an executable fat jar: this jar is automatically produced by Maven or Gradle during the build when the package phase is executed (or the bootJar task respectively). This is the jar that must be placed on the remote VM 192.168.0.82 to be executed there.

Creating the Debug Configuration in IntelliJ Link to heading

From the IntelliJ debugger’s perspective, the remote application will run on localhost and the JVM will listen for debugger instructions on port 50005. Configure the “Remote JVM Debug” run configuration for this purpose:

  • In IntelliJ, open the window with run configurations (Menu -> Run -> Edit configurations…)
  • Create a new configuration of type Remote JVM Debug
  • Fill in the fields as follows:
    • Debugger mode: Attach to remote VM
    • Host: localhost
    • Port: 50005
    • if the “Transport” field is present: enter the value Socket
  • Copy the content of the Command line argument for remote JVM field to add to the application launch command after modifications: it will be in the form -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:50005 — you will need to replace the port number at the end with 5005.

Launching the Application on the Remote Server Link to heading

Launch the Java test application on the remote machine by adding the debug option to the command line with port number 50005 replaced by 5005:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar /path/to/remote-debug.jar

Creating the SSH Tunnel Link to heading

Creating an SSH tunnel with local port forwarding between port 5005 on the remote machine and port 50005 on the workstation is done by running the following command on the workstation:

ssh -L 50005:127.0.0.1:5005 fabrice@192.168.0.82

An SSH session opens and the tunnel is created. The SSH session can be used and closed independently of the tunnel, which will remain open as long as there are exchanges.

Launching the Debugger in IntelliJ Link to heading

Launch the debug configuration previously created in IntelliJ by selecting it among the run configurations and clicking on the bug-shaped icon: The IDE connects to the remote JVM and the debug view is set up: Debug view

Note that in the debug view, it says Connected to the target VM, address: 'localhost:50005', transport: 'socket'

It Works! Link to heading

  • Verify that the breakpoint set at line 51 is still present
  • Make a GET http://localhost:8080/test request with a curl-type client
  • Execution of the remote application stops at the breakpoint and debug information is displayed
  • Step-by-step execution is possible and timestamps sent progressively by the server can be observed, just like in the local case
  • Change the value of the boolean stop to true (right-click -> Set value… or F2) and resume execution: the application exits the loop and the HTTP response is closed: boolean stop set to true
  • At the end of the debugging session:
    1. Stop the debugger in the IDE by disconnecting it (close the tab with the ongoing debugging in the debug view)
    2. Close the SSH session if it is still open (exit)
    3. If necessary, stop the remote application

Precautions Link to heading

  • Network exchanges between the debugger and the remote JVM are in plaintext: without using an SSH tunnel, confidential information can transit in clear text — the SSH tunnel addresses this.
  • Some best practices for securing an SSH tunnel

On Kubernetes Link to heading

If the application is deployed in a Kubernetes cluster, port forwarding should be used rather than SSH tunneling. If the telepresence tool is present on the cluster, IntelliJ also integrates with it: I have never tried it, so I have no idea how it works.

References Link to heading