3 - Getting started with OpenSyncro

Having completed OpenSyncro installation process, point your browser to http://localhost:8080/opensyncro/Login and log in to your OpenSyncro database. Make sure that Javascript support is enabled in your browser. After login the user is presented with "Pipes / All Pipes" view displaying a list of complete integration processes, i.e. Pipes, stored in OpenSyncro's database. Before proceeding to creating, modifying and executing the Pipes (see chapters 3.3, 3.4, 3.5 and 3.6), it is recommended that the default user's password is changed for security reasons.

This chapter first explains how to add and remove users and edit the user data. Next we have look at Pipe Component management, and then OpenSyncro's Pipe and Component editors will be introduced. After we have familiarized ourselves with OpenSyncro's user interface, a couple of example integration processes are introduced.

3.1 - User management

User management interface ("Users") can be found from the "General" pulldown menu in the left corner of top menu bar. The Users table view shows all users that have access to the current OpenSyncro database (via OpenSyncro). If you wish to add a new user, click on "Create" button in the down-left corner. Users' login name ("Login") and real name ("Name") are displayed with the option to change password, or delete user ("Delete" button). User passwords are always hidden. Use the "OK" button to save, or "Cancel" button to abandon changes.

Picture 3.1 : User management

Note: currently any OpenSyncro user may change other users' data, including passwords.

3.2 - Installing and removing Pipe Components

Component manager ("Components") is located in the "General" pulldown menu, the left corner of top menu bar. It allows to view and (re)load Components, which are the Java based building blocks of OpenSyncro Pipes.

OpenSyncro's Component selection consists of built-in Default Components and external Dynamic Components (.jar files), which are loaded from a directory path specified by "component.dir" property in the opensyncro.properties file. While the Components are initially loaded at OpenSyncro web application start-up, Component manager provides a "Reload all components" button which allows to install/remove Dynamic Components without restarting OpenSyncro.

Component manager displays a table of names, descriptions and Component ID's of all Source, Converter and Destination Components currently available to OpenSyncro. External Dynamic Components' information is highlighted with a green background color.

Picture 3.2 : Component manager

*Please note that the SBODISSource, SBODISRawCommand, SBODISConverter and SBODISDestination components in the above screenshot are not included in the free OpenSyncro distribution packages, but are available as an extension called the SAP Business One Connector Pack.

3.3 - Pipe list: Adding and removing integration processes

After logging in to OpenSyncro's administration interface, the user is taken to the Pipe list ("All pipes") page which displays the names of all integration processes stored in the current OpenSyncro database. It is possible to create a new Pipe by clicking on the "Add" button, edit an existing Pipe ("Edit" button next to the Pipe name), start a Pipe ("Start" button), clone a Pipe ("Clone" button) and remove a Pipe ("Delete" button). Cloning a Pipe creates a new Pipe which has exactly the same Components and settings as the original Pipe. Deleting a Pipe also deletes Transfer log messages concerning the Pipe. Pipe names can be edited in the list and the changes saved by hitting the "OK" button. The "Cancel" button abandons all changes.

Currently running Pipe's rows are highlighted with a light green background color. Information about the last execution of a Pipe is also visible on this page. You will see last execution start and end times, duration, status and the name the OpenSyncro user who started the Pipe. If a Pipe execution was initiated by remote invocation (HttpStart servlet) request, "Remote" is displayed in place of the user name. If a Pipe has never been executed these columns will be empty.

Picture 3.3 : Pipe list

The Pipe list page is always accessible from the Pipes pulldown menu by selecting "All pipes" menu item.

3.4 - Pipe editor: Creating, maintaining and testing integration processes

Pipe editor is entered by clicking on an "Edit" button in Pipe list. Alternatively, the "Pipes" pulldown menu in the top menu bar contains a "Pipe editor" item which opens up the Pipe editor for the last Pipe edited during the current session.

Pipe editor displays a Pipe configuration in three columns: Source, Converters and Destination. Source column on the left specifies a single Source Component used to retrieve data to be processed by the Pipe from the source system. Converters column in the middle is list of optional Converter Components. First Converter on the top takes its input from the Source Component and after processing the data passes it to the next Converter on the list. The Destination Component on the right column takes the output from the last Converter - or directly from the Source Component if there are no Converters - and delivers it to the target system.

Picture 3.4 : Pipe Editor

Components can be added to the Pipe by selecting the Component name from a pulldown menu and clicking the "Create" button next to the menu. For removing a Component from the Pipe, there is a "Delete" button for each Component. Converter Components' order in the middle column can be changed by clicking the small up/down arrows next to Component's name.

Every OpenSyncro Pipe Component can take parameters. Component parameters are set and modified in Component Editor, which is entered by clicking "Edit" button next to the Component.

OpenSyncro Pipes can be started by pressing the "Start" button. When the execution of the Pipe task is finished, the Pipe Editor page reloads and execution information (start/end times, duration and status) will be displayed. Status information of the Pipe execution is written to "Transfer log", available at the top menu bar. Chapter 3.7 explains the "Transfer log" and its settings.

3.5 - Pipe settings: Logging, email notification and remote invocation

Pipe settings is entered by clicking on a "Pipe settings" tab or via the "Pipes" pulldown menu in the top menu bar.

Pipe settings tab allows you to change the name of the pipe, enable remote HTTP invocation and set transfer log verbosity level. "Transfer log verbosity level" determines the types of messages recorded in the log. The possible choices are: 'Errors', 'Warnings', 'All' (debug) and 'Dynamic'. Whereas the first three modes log messages equal to and above the set verbosity level (i.e. 'warnings' includes also errors and 'all' also covers warnings and errors), 'dynamic' mode logs all messages only if the Pipe execution fails with an error. If the Pipe is executed successfully, warning messages are written to the Transfer log but the debug messages from 'all' level are deleted.

Email notification requires the "Outgoing email server" (SMTP hostname or IP address) and "Receiver of notification email" (one or more email addresses separated by a comma or semi-colon character) fields to be filled in. Email subject line will contain the Pipe name, completion status and execution time. The email body consists of Transfer log message lines.

The "Notification level" determines the threshold for sending the notification mail, choices being 'Errors', 'Warnings', 'All' and 'None'. If the pipe's log does not contain messages at or below the set notification level, no mail is sent. For example, when Notification level is set to 'Warnings', a mail is sent if the log contains warning or error messages. The types of messages that are present in the notification mail is determined by the "Transfer log verbosity level" value. Setting notification level to 'None' will result in no message being sent, regardless of log contents.

OpenSyncro can be told to send a notification mail when a pipe aborts but there are no errors in the log. For this, check the "Send notification mail when pipe aborts" checkbox. Also, make sure that 'Notification level' is not set to 'None'

Picture 3.5 : Pipe Settings

When "Enable HTTP start" is activated, the Pipe may be started with a single HTTP GET request, loading a URL containing the Pipe name and it's "Start password". With this feature, an OpenSyncro Pipe execution can be easily initiated by a command line tool such as 'wget' or 'curl', scheduled to run for example every 30 minutes. The OpenSyncro's HTTP Start URL format is as follows:

http://localhost:8080/opensyncro/HttpStart?database=DatabaseName&pipe=PipeName&password=PipeStartPassword&mode=Mode

The mode parameter determines whether the HttpStart servlet waits until the Pipe has finished executing (this requires the mode parameter to be set to sync) or, if another instance of the Pipe is already executing, adds the request to the Pipe Execution Queue (mode parameter set to async). HttpStart returns an error message if the HTTP request parameters are incorrect or HTTP start has not been enabled for the specified Pipe. If Pipe execution is started successfully, "OK" message is returned if sync mode is used. In async mode the message returned is "QUEUED (queue length: n)", where n is the total number of requests for this Pipe waiting to be executed. Note: In sync mode HttpStart responds with "OK" even though the Pipe execution may have resulted in an error.

3.6 - Component editor: Setting Pipe Component parameters

Component editor is started from the Pipe editor by clicking the "Edit" button next to any Source, Converter or Destination Component in the Pipe. Component editor displays all the changeable parameters of the Component and allows user to modify the parameter values. Use the "OK" button to save, or "Cancel" button to abandon changes.

Picture 3.6 : Component Editor view of CSVtoXMLConverter

3.7 - Transfer log: Monitoring and debugging integration processes

When a Pipe is executed, status and debug information is written to the "Transfer log", depending on the "Transfer log verbosity level" setting in Pipe editor. The setting has 3 choices: "Errors" logs only error messages that cause the Pipe execution fail, "Warnings" includes both error and warning messages and "All" writes to log every debug message generated by Pipe components - in addition to the warning and error messages.

Picture 3.7 : Transfer Log

The overall status of the finished Pipe execution is displayed in the "Status" column. Possible status messages are "OK", "Source error", "Conversion error", "Destination error" and "Aborted". The "Aborted" status means that the Pipe execution was stopped, but no error occurred. For example if a Source Component has been set up with a search query that returns an empty result set, the Pipe is aborted as there is no data with which to continue the Pipe process.

At the bottom left corner on a Transfer log page you'll find a "Clear Transfer Log" button, which clears the entire log - all log messages originating from any Pipe execution.

3.8 - Pipe Execution Queue: Monitoring Pipe execution request queues

Pipe execution requests OpenSyncro receives via remote invocation (HttpRequest servlet) are queued so that two instances of the same Pipe will not execute simultaneously. This page allows you to view the current statuses of these execution requests. The columns displayed are "Pipe", "Request queued" and "Pipe started". "Request received" shows the time and date when the request for Pipe execution was received. "Pipe" shows the name of the Pipe to be executed. "Pipe started" shows the time and date when Pipe execution started or "Queued" if the request is waiting for another instance of the Pipe to finish execution.

Picture 3.8 : Pipe execution queue

The "Clear execution queue" button removes all pipe execution requests from the queue. Note however, that Pipes already executing are not stopped.

When Tomcat is started, OpenSyncro checks the database for unprocessed Pipe execution requests. What it does with found requests depends on a per-database setting. It is possible to set up OpenSyncro to ignore all Pipe execution requests in the database and remove them, or execute all found requests at startup. To specify which route to take you need to modify the opensyncro.properties file. For each OpenSyncro database two lines need to be added. First, "init1.database=database name", second "init1.resumemode=resume mode". Resume mode should be set to either 'CLEAR', which removes all Pipe execution requests from the specified database, or 'RUN', which executes found requests. The number in the property name (in this case '1' in init1.database) should be incremented with each added database (for example, the second OpenSyncro database in the file should start with "init2" prefix). When using more than one database, make sure that all numbers in the property names are sequential (i.e. no gaps in the numbering).

If you do not use OpenSyncro's remote execution capabilities you can skip adding your database(s) to the opensyncro.properties file. In this case OpenSyncro will not delete requests from the database nor will it execute them.

3.9 - Example Pipes of OpenSyncro Demo Database

If you have initialized OpenSyncro's database using the database dump with demonstration material ('opensyncro_demodb.sql', see: "Initializing/updating an OpenSyncro database"), the Pipe list view displays five example Pipes which you can experiment with. Before starting any of the example Pipes from Pipe editor, you need to visit the Component editor of Source and Destination components of each Pipe and enter the correct arguments, for example directory paths, URLs and database connection parameters.

List of OpenSyncro's Example Pipes
Example 1: CSV table to flat XML	Reads a CSV table ("Installation/examples/example_1_input.csv") from filesystem, converts it to flat XML format and writes the result to a file.
Example 2: Custom ASCII to XML	Reads a custom ASCII based data file ("Installation/examples/example_2_input.txt") from filesystem, converts it to XML and writes the result to a file.
Example 3: Export all orders from Workspace	Connects to a Workspace webshop via Workspace OpenInterface (Web Services), exports all available sales orders in the native OpenInterface XML format and writes the result to a file.
Example 4: Import product data to Workspace	Reads an XML file ("Installation/examples/example_4_input.xml") in Workspace OpenInterface product data format and sends the product record to a Workspace webshop.
Example 5: Import customers from CSV table to Workspace	Reads a CSV table ("Installation/examples/example_5_input.csv") containing simple customer account data from filesystem and converts the CSV data to flat XML format. The flat XML is then transformed to Workspace OpenInterface customer data format and sent to a Workspace webshop.

For example Pipes 1, 2, 4 and 5 you need to enter the full path of the example input data files provided in the OpenSyncro install ZIP archive, to the LocalFileSource component parameters. For example Pipes 3 - 5 the database connection parameters for a Workspace webshop need to be set to the Source/Destination components. Additionally, for OpenSyncro to perform data import/export functions with Workspace you need to create a Workspace user with OpenInterface access enabled. Example Pipes 1 - 3 write their output data to local filesystem, so you have to specify a valid LocalFileDestination path for each pipe.

Having entered the Source and Destination component parameters for an example Pipe, you can test the Pipe by pressing the "Start" button in the Pipe editor. The button press will result in OpenSyncro executing the current Pipe and reloading the Pipe editor page, after which you can check the Transfer Log for status information. If there were no errors or warnings, the output data should be either a file created in the directory you specified in the LocalFileDestination's Component editor (example Pipes 1 - 3) or a product/customer record visible in the Workspace webshop's Administration Interface (example Pipes 4 - 5).