Master Shell's timing with Delphinscheduler and the processehn

This is where you will learn how to effectively start the Shell scripts and monitor sudo support by managing the catalogs and full logping.

In this article we dive deeply how Dolphinscheduler uses ProcessBuilder To execute Shell's commands. By default, Delphinscheduler surrounds the shell scripts using BashShellInterceptorBuilderGeneration of commands that support both standard and sudo modes.

We also walk through a Spring boot An example of the configuration of the working catalogs, the connection of output flows, monitoring processes, and output logs – helping you achieve effective Shell task management and schedule.

1. How Dolphinscheduler uses a processbuilder

1.1. Packing the bark commands

Dolphinscheduleris is done in command packing:

org.apache.dolphinscheduler.plugin.task.api.shell.ShellInterceptorBuilderFactory

public class ShellInterceptorBuilderFactory {

    private final static String INTERCEPTOR_TYPE = PropertyUtils.getString("shell.interceptor.type", "bash");

    @SuppressWarnings("unchecked")
    public static IShellInterceptorBuilder newBuilder() {
        // Default logic
        if (INTERCEPTOR_TYPE.equalsIgnoreCase("bash")) {
            return new BashShellInterceptorBuilder();
        }
        if (INTERCEPTOR_TYPE.equalsIgnoreCase("sh")) {
            return new ShShellInterceptorBuilder();
        }
        if (INTERCEPTOR_TYPE.equalsIgnoreCase("cmd")) {
            return new CmdShellInterceptorBuilder();
        }
        throw new IllegalArgumentException("Unsupported shell type: " + INTERCEPTOR_TYPE);
    }
}

✅ Default, BashShellInterceptorBuilder used.

Here is an appropriate part:

org.apache.dolphinscheduler.plugin.task.api.shell.bash.BashShellInterceptorBuilder

public class BashShellInterceptorBuilder
        extends BaseLinuxShellInterceptorBuilder {

    @Override
    public BashShellInterceptorBuilder newBuilder() {
        return new BashShellInterceptorBuilder();
    }

    @Override
    public BashShellInterceptor build() throws FileOperateException, IOException {
        // Core part: Generate shell script
        generateShellScript();
        List bootstrapCommand = generateBootstrapCommand();
        // Instantiate BashShellInterceptor
        return new BashShellInterceptor(bootstrapCommand, shellDirectory);
    }

    @Override
    protected String shellInterpreter() {
        return "bash";
    }

    @Override
    protected String shellExtension() {
        return ".sh";
    }

    @Override
    protected String shellHeader() {
        return "#!/bin/bash";
    }
}

The commands are generated here:

org.apache.dolphinscheduler.plugin.task.api.shell.BaseLinuxShellInterceptorBuilder#generateBootstrapCommand

protected List generateBootstrapCommand() {
    if (sudoEnable) {
        // sudo -u tenant -i /opt/xx.sh
        return bootstrapCommandInSudoMode();
    }
    // bash /opt/xx.sh
    return bootstrapCommandInNormalMode();
}

Two roads:

• Normal mode (bash /path/to/script.sh)
• Sudo mode (sudo -u tenant -i /path/to/script.sh)

Example of sudo mode:

private List bootstrapCommandInSudoMode() {
    List bootstrapCommand = new ArrayList<>();
    bootstrapCommand.add("sudo");
    if (StringUtils.isNotBlank(runUser)) {
        bootstrapCommand.add("-u");
        bootstrapCommand.add(runUser);
    }
    bootstrapCommand.add("-i");
    bootstrapCommand.add(shellAbsolutePath().toString());
    return bootstrapCommand;
}

1.2. Execution of Shell's command

The actual implementation is discussed here:

org.apache.dolphinscheduler.plugin.task.api.shell.BaseShellInterceptor

public abstract class BaseShellInterceptor implements IShellInterceptor {

    protected final String workingDirectory;
    protected final List executeCommands;

    protected BaseShellInterceptor(List executeCommands, String workingDirectory) {
        this.executeCommands = executeCommands;
        this.workingDirectory = workingDirectory;
    }

    @Override
    public Process execute() throws IOException {
        ProcessBuilder processBuilder = new ProcessBuilder();
        processBuilder.directory(new File(workingDirectory));  // Set working directory (important for loading JARs etc.)
        processBuilder.redirectErrorStream(true);              // Merge error stream into output stream
        processBuilder.command(executeCommands);
        log.info("Executing shell command: {}", String.join(" ", executeCommands));
        return processBuilder.start();
    }
}

2. Practical Example: The timing of shells is easy

Let's build a simple spring boot application that launches the cream script!

2.1. pom.xml addiction

  org.springframework.boot
  spring-boot-starter
  2.6.1

2.2. Spring boot code

@SpringBootApplication
public class Application {

    public static void main(String[] args) throws Exception {
        SpringApplication.run(Application.class, args);

        List executeCommands = new ArrayList<>();
        executeCommands.add("sudo");
        executeCommands.add("-u");
        executeCommands.add("qiaozhanwei");
        executeCommands.add("-i");
        executeCommands.add("/opt/test/my.sh");

        ProcessBuilder processBuilder = new ProcessBuilder();
        processBuilder.directory(new File("/opt/test"));  // Set working directory
        processBuilder.redirectErrorStream(true);         // Merge error output into standard output
        processBuilder.command(executeCommands);

        Process process = processBuilder.start();

        try (BufferedReader inReader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
            String line;
            while ((line = inReader.readLine()) != null) {
                // Print each line of output
                System.out.println(line);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

        // Wait for 10 minutes maximum
        boolean status = process.waitFor(10, TimeUnit.MINUTES);
        System.out.println("Execution status -> " + status);
    }
}

2.3. Example Log output

Logs show:

• Shell script was successfully carried out
• Standard and error outputs were captured
• The process ended within 10 minutes

Salmon:

Executing shell command : sudo -u qiaozhanwei -i /opt/test/my.sh
...
Job job_1694766249884_0931 completed successfully
status ->true

Process finished with exit code 0

The final thoughts

Use Delphinscheduler + ProcessBuilderYou can flexibly handle the full control of the task:

• Work catalogs
• Error/Standard Output Flow
• Sudo promises
• Monitoring

This approach is easyTo do, strongand Easy to expand -Mandatory skill when you manage shell -based workflows big data or Devops Scenarios!