Skip to main content

Getting Started with Datorios

Welcome to Datorios, the observability platform tailored specifically for Apache Flink®. Our mission is to empower developers and data engineers with unparalleled visibility into their real-time Flink data streams, ensuring operational excellence and data quality.

What is Datorios?​

Datorios is a powerful observability solution designed to enhance your Apache Flink operations. By unifying metrics, logs, and data traces, Datorios enables you to detect, isolate, and resolve issues before they impact your business.

Key Features​

  • Data Lineage: Understand the flow of your data with upstream and downstream lineage tracking, allowing you to trace records back to their source or through processing stages.
  • State Analysis: Monitor state evolution throughout operator event processing, providing critical insights into operator behavior to refine your data flows with precision.

Getting Started​

  1. Sign Up: Visit our sign-up page to create your free account.
  2. Deployment: Integrate Datorios with your Apache Flink environment. We support installation via Docker and Kubernetes.
  3. Explore Features: Utilize our comprehensive observability tools, including job insights, state viewing, performance analysis, and data lineage, to monitor and optimize your Flink jobs.

For more information, visit our website or contact our support team.

Note: Datorios supports Flink version 1.6.x and above, and languages including Java, Python, and Scala.

Installation and Quick Setup

Getting started with Datorios is straightforward. Follow these steps to quickly deploy the observability layer and integrate it with your Apache Flink environment:

Components of Datorios​

Datorios Client:​

  • Runs on Docker Compose.
  • Installs the Apache Flink engine on your local or cloud machine to deploy your jobs.

Observability Service:​

  • Provides deep investigation and debugging capabilities.

Installation Steps​

1. Download and Extract the Client:​

  1. Download the datorios.tar.gz file for your version.
  2. Extract the archive: 
    tar zxvf datorios.tar.gz
  3. Set the script as executable: 
    chmod +x datorios.sh

2. Configure Environment Variables:​

  1. Open the .env file and configure:
    • MOUNT_SRC_PATH: Local directory containing your Flink jar files.
    • MOUNT_DST_PATH: Mount point inside the Docker container.
  2. Example: 
    MOUNT_SRC_PATH=/home/jars
    MOUNT_DST_PATH=/flink_jobs

3. Fetch the Datorios Client Images:​

Run the following command:

./datorios.sh fetch

4. Start a Flink Cluster:​

Use the command:

./datorios.sh [cluster-name] start

Example:

./datorios.sh my-cluster start

5. Run Flink Jobs:​

  1. Specify the job path using the configured environment variables.
  2. Use the command: 
    ./datorios.sh [cluster-name] flink run /MOUNT_DST_PATH/job_name.jar

6. Stop a Flink Cluster:​

Use the command:

./datorios.sh [cluster-name] stop

Verification​

Check the cluster status with:

./datorios.sh list