{show-if:display=pdf}
{toc}
{show-if}
{section}
{column}

{warning}This version must be installed on Petals ESB 5.0.1+{warning}

h1. Features

This component allows to expose Talend jobs as services into Petals and to execute them inside the bus.
It only acts as service provider, not as a service consumer.

\\
This component supports jobs created with both Talend Open Studio and Talend Integration Suite (see [http://www.talend.com] for more details).

It provides several mechanisms to pass information and data to a job.
Il also supports several mechanisms to retrieve information and data from a job.

\\
Before going further, it must be clear that no configuration for this component is intended to be created by hand. Neither by the Petals Studio.
In fact, only Talend Open Studio and Talend Integration Suite have the ability to generate a correct configuration for this component.
{column}
{hide-if:display=printable}
{column:width=350px}
{panel:title=Table of contents}{toc}{panel}
{panel:title=Contributors}{contributors:order=name|mode=list}{panel}
{column}
{hide-if}
{section}

h1. Recommended usage

The Petals-SE-Talend component is intended to execute jobs created with Talend Open Studio and Talend Integration Suite.
Whenever you want to work with both Talend and Petals ESB, it is the component to use.

\\
There are three ways of seeing the jobs that will be executed in Petals ESB.

First, they can be seen as *an extension of the platform capabilities*.
Indeed, a job can easily interact with almost any database, any kind of file, or other systems (SAP, Alfresco, etc...). Rather than developing a specific component, it can be interesting to create a job to cover this task, and expose it as a service through Petals. *The job here acts as a mediation way with data stores or systems.*

\\
Second, a job can be seen as *a set of transformation means*.
Indeed, Talend products provide _Talend components_ that are very efficient and convenient for transformations (e.g. the tMap component). However, it must be clear that these transformations only cover flat structures, like schemas of relational databases. Object or XML schemas are not covered. From this point of view, *the Petals-SE-Talend component cannot replace the Petals-SE-XSLT component*. But it can be an alternative in few cases. Hence, transformations will either rely on attachment files (the content of the attached files is transformed by the job) or on Talend components for Petals (known as _tPetalsInput_ and _tPetalsOutput_). The later solution provides a means to place the content to transform inside the XML message, rather than as an attachment. But still with constraints on the XML shape.

\\
Eventually, the last way to apprehend a Talend job inside Petals, is to see it as a *an application to execute inside Petals*.
This application may work with data stores (files, databases...), may involve data transformations, but may also use other (if not redundant with Petals) features. As an example, it is possible to send mails inside a job, connect to FTPs, etc... Obviously, these features are also available inside Petals. But in some cases, it can be more interesting and more simple to integrate them directly in the job than use the Petals'ones (where you will have to use EIPs or BPELs to link the calls). In some other cases though, the exact opposite may be the best option, i.e. externalize some parts to Petals components. *It all depends on the expected granularity and reusability*.

\\
The wide variety of possibilities (allowed by the non-less important variety of _Talend components_, and by the features of the Petals-SE-Talend component) makes this solution a very flexible one. However, as a Swiss-knife component, *the Petals-SE-Talend component should mainly be seen as a functional service-engine*. Performances, without being bad, cannot be the best ones offered. People looking for a very specific and high-performance usage will prefer develop their own Petals component, or use Petals-SE-Pojo or Petals-SE-Jsr181 components.


h1. Talend Open Studio vs. Talend Integration Suite

Two products can be used to create Talend jobs and export them into Petals: Talend Open Studio, which is an open source product, and Talend Integration Suite, which is the upgraded version of Talend Open Studio (but not free).
There are slight differences between what can be made with these two products.

h2. Talend Open Studio

Talend Open Studio supports the following features for the Petals-SE-Component:
* Expose a context as a parameter into the service's WSDL.
* Pass attachment files to a job.
* Pass native parameters and options to a job.
* Get the job's execution result.
* Get output attachments from a job.

Other features are only supported by Talend Integration Suite.
In particular, it does not support any specific component for Petals.
However, it is possible to use alternative ways, depending on the desired behavior.


h2. Talend Integration Suite

In Talend Integration Suite, all the features of Talend Open Studio are supported.
It also embeds additional capabilities to work with Petals ESB.


* tPetalsInput and tPetalsOutput components can be used in a job.
** These components respectively allow you to pass data from Petals to the job and from the job to Petals.
** These two components do not have any specific parameter. They just need their schema to be defined.

* The Petals behavior can be simulated for these two components. It means a job created in Talend Integration Suite which would contain a tPetalsInput or a tPetalsOutput component, can be executed inside Talend Integration Suite.
** Thus, for a tPetalsInput, the Petals input message can be mocked by (loaded from) an XML file during the simulation.
** For the tPetalsOutput, the returned document will be printed into the console.

\\
Here is a sample job using tPetalsInput and tPetalsOutput components.
This job simply takes an input message from Petals and transforms it before returning it in the bus.

!tPetalsInputOutput.jpg!


h2. Limitations

There are some restrictions in the Petals export and in the use of the Talend components for Petals.

\\
About the export:
* Only one job can be exported at once for Petals.
** One job = One deployment archive for Petals.

* Some data types are not supported in the export for Petals:
** Lists, Objects, passwords, List of values are not supported.
** It also applies to exported contexts.

\\
About the tPetalsInput and tPetalsOutput components:
* A job can contain several instances of any of these components.

* All the tPetalsInput must have the same schema.
** This is checked during the export.

* All the tPetalsOutput must have the same schema.
** This is checked during the export.

* All the tPetalsInput instances read the same input message.

* All the tPetalsOutput instances write in the same output message.
** If two instances of tPetalsOutput are on the same processing flow, the second will erase what is written by the first instance.
** tPetalsOutput instances should be placed on different _branches_ (onSubjobOK, onSubjobError and so on...).


h2. The Talend export for Petals

The export is the same in both Talend Open Studio and Talend Integration Suite.
The generated artifacts are the same in both of them. And the content of the jbi.xml is (almost) the same in both of them.
The only differences are the job's content (which Talend components have been added into the job) and the generated WSDL (which depends on the job content and the export options).

The service-unit structure and the jbi.xml are described farther.
This section only introduces the export options and their impact on the result.
The generated WSDL is not discussed in this documentation.
However, you will find samples in the simple use cases for this component.


Here is a snapshot of the Talend export for Petals.

{center}
!FinalTalendExportForPetals_1.jpg!
{center}

The target file is the location of the service-assembly to generate.
The job version meaning is explicit.

The export options are the following:

|| Export Option || Description ||
| Singleton job | True to make the job singleton. A singleton job can have only one instance running at once on a given Petals-SE-Talend component. |
| Generate the end-point | True to let Petals generate the end-point at deployment time. If false, the end-point name is the job name with the suffix "Endpoint". |
| Validate Petals messages | True to validate all the messages / requests against the WSDL. \\
Be careful, enabling this option reduces the performances (disk access). \\ |
| User routines | Embed the user routines in the service-unit. |
| Source files | True to embed the source files in the generated service-unit. |
| Jobs contexts | Select the context that will be used by default by the job. |

\\
Eventually, there is the edition link to specify how contexts should be exposed in the generated WSDL.

{center}
!FinalTalendExportForPetals_2.jpg!
{center}

When this link is clicked, a dialog shows up.
It lists all the context variables from the job, with their name, the type they will be associated with if exported, and the way they are exported.
By default, no context is exported. Said differently, the export mode for all the contexts is *Not exported*.

\\
Here is a small description of the export modes.

|| Export Mode || Description ||
| Not exported | The context is not exported (not visible as a parameter). But the context can still be overridden using the native parameters (options) of the job. |
| Parameter | The context is exported as a parameter in the WSDL's operation. |
| In-Attachment | The context will be passed the location of a temporary file whose content was attached in the input message. |
| Out-Attachment | The context will be read after the job was executed.
* This context must point to a file.
* The file content will be read by the Petals-SE-Talend component and put as an attachment into the response. 
* The context name will be used as the attachment name.
* The file will be deleted by the component right after its content was loaded. |
| Parameter and Out-Attachment | A mix between the Parameter and the Out-Attachment modes.
* The context is exposed as a parameter.
* It will also be read after the job execution.
* The file will be deleted anyway.
* The advantage of this export mode is to define the output file destination dynamically. |



h1. Exposing a Talend job as a Petals service (Provides mode)

The most important thing to understand is that this component is not intended to be used without Talend products.
It is useless to create JBI descriptor and a service-unit by hand for this component.
In fact, you should even care about the operations this component supports, and only rely on the WSDL generated at the export.

The operations contained by the generated WSDL only contain the operations that this component supports.

\\
This component supports two operations:
* *executeJob:* this operation creates a new job instance, passes it the received parameters, executes it and gets the result back before sending it in the response.
* *executeJobOnly:* this operation creates a new job instance, passes it the received parameters and executes it. This operation does not send back a response.


h2. The "executeJob" operation

The fully qualified name of this operation is:
* Name space URI: any URI, provided it is not null (e.g. *{html}http://petals.ow2.org/talend/{html}*)
* Local part: *executeJob*

\\
{warning}
In the version 1.0 (or 1.0.0), the only name space URI that is accepted for this operation is *{html}http://petals.ow2.org/talend/{html}*.
From the version 1.0.1, you can use any name space URI, as long as it matches a WSDL operation.
Once again, rely on the WSDL generated by Talend products.
{warning}

\\
This operation only supports the *InOut* message exchange pattern (MEP).
When invoking this operation, you must call it using its fully qualified name.

\\
The input and output depend on the job itself and is fully described in the generated WSDL definitions.
Once again, rely on the generated WSDL.


h2. The "executeJobOnly" operation

The fully qualified name of this operation is:
* Name space URI: any URI, provided it is not null (e.g. *{html}http://petals.ow2.org/talend/{html}*)
* Local part: *executeJobOnly*

\\
{warning}
In the version 1.0 (or 1.0.0), the only name space URI that is accepted for this operation is *{html}http://petals.ow2.org/talend/{html}*.
From the version 1.0.1, you can use any name space URI, as long as it matches a WSDL operation.
Once again, rely on the WSDL generated by Talend products.
{warning}

\\
This operation only supports the *InOnly* message exchange pattern (MEP).
When invoking this operation, you must call it using its fully qualified name.

\\
The input depends on the job itself and is fully described in the generated WSDL definitions.
This operation does not send any response. Once again, rely on the generated WSDL.


h2. JBI Descriptor

The service-unit descriptor file ( jbi.xml ) looks like this:
{code:lang=xml}<!-- Remember, this file is intended to be generated by Talend Open Studio or Talend Integration Suite -->
<?xml version="1.0" encoding="UTF-8"?>
<jbi:jbi version="1.0"
xmlns:jbi="http://java.sun.com/xml/ns/jbi"
xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
xmlns:talend="http://petals.ow2.org/components/talend/version-1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<jbi:services binding-component="false">
<jbi:provides
interface-name="generatedNs:AttachmentInOutServicePortType"
service-name="generatedNs:AttachmentInOutService"
endpoint-name="AttachmentInOutEndpoint"
xmlns:generatedNs="http://petals.ow2.org/talend/">

<!-- CDK parameters -->
<petalsCDK:wsdl>AttachmentInOut.wsdl</petalsCDK:wsdl>
<petalsCDK:validate-wsdl>true</petalsCDK:validate-wsdl>

<!-- Component parameters -->
<talend:name>AttachmentInOut</talend:name>
<talend:class-name>
talenddemosjava.attachmentinout_0_1.AttachmentInOut
</talend:class-name>
<talend:context>Default</talend:context>
<talend:singleton>true</talend:singleton>
<talend:validate-exchange-by-wsdl>false</talend:validate-exchange-by-wsdl>

<!--
Define all the expected output attachements.
There can be as many "output-attachment" as required.
-->
<talend:output-attachment>outputFile</talend:output-attachment>
</jbi:provides>
</jbi:services>
</jbi:jbi>

{code}

\\
*Configuration of a Service-Unit to expose a Talend job as a service into Petals ESB :*

|| Parameter || Description || Default || Required ||
| name | The job's name \\ | \- | Yes |
| class-name | The job's class name \\ | \- | Yes |
| context | The context to use (if invalid, the job will use the default context) \\ | \- | Yes |
| singleton | The singleton property \\ | \- | Yes |
| validate-exchange-by-wsdl | Validate the messages against the WSDLs' schemas | False | No \\ |
| output-attachment \\ | The name of a context variable that points to a file that must be attached to the returned message. \\
The cardinality for this element is 0-*. \\ | \- | No \\ |

\\
{include:0 CDK SU Provide Configuration}


h2. Service-Unit content

The service-unit must contain the JAR archives of the job and its dependencies, plus the context files.
It is also highly recommended to provide a WSDL description of your job. This WSDL is not mandatory, but not providing it will prevent your service from interacting with other Petals services and components.
By default, a WSDL is generated during the Talend export for Petals.

\\
The directory structure of a SU for the Petals-SE-Talend looks like this:

{noformat}
su-talend-JobName-provide.zip
+ META-INF
- jbi.xml
+ src
- { source files }
+ { Contexts directory }
- *.properties
- JobName.wsdl
- systemRoutines.jar
- userRoutines.jar
- JobName_JobVersion.jar
- JobDependencies.jar (there can be several jars)
{noformat}


h1. Configuring the component


The component can be configured through its JBI descriptor file, as shown below.
{code:lang=xml}<?xml version="1.0" encoding="UTF-8"?>
<jbi:jbi
version="1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:petalsCDK="http://petals.ow2.org/components/extensions/version-5"
xmlns:jbi="http://java.sun.com/xml/ns/jbi"
xmlns:talend="http://petals.ow2.org/components/talend/version-1">

<jbi:component type="service-engine">

<jbi:identification>
<jbi:name>petals-se-talend</jbi:name>
<jbi:description>
A service engine to expose and run Talend jobs as services in Petals
</jbi:description>
</jbi:identification>

<jbi:component-class-name>
org.ow2.petals.se.talend.TalendSe
</jbi:component-class-name>
<jbi:component-class-path>
<jbi:path-element/>
</jbi:component-class-path>

<jbi:bootstrap-class-name>
org.ow2.petals.component.framework.DefaultBootstrap
</jbi:bootstrap-class-name>
<jbi:bootstrap-class-path>
<jbi:path-element />
</jbi:bootstrap-class-path>

<!-- CDK specific fields -->
<petalsCDK:acceptor-pool-size>5</petalsCDK:acceptor-pool-size>
<petalsCDK:processor-pool-size>10</petalsCDK:processor-pool-size>
<petalsCDK:ignored-status>DONE_AND_ERROR_IGNORED</petalsCDK:ignored-status>
<petalsCDK:jbi-listener-class-name>
org.ow2.petals.se.talend.TalendJBIListener
</petalsCDK:jbi-listener-class-name>

<!-- Component specific configuration -->
<!--
The WSDL-based validation for exchanges checks that the called operation,
the MEP and the input message are valid with respect to the WSDL.

This property is also available for service-units.
Enabling this property in the component enables it for all the service-units
deployed on this component and overrides their configurations.

When set to false, the service-unit property is used.

Set this property to true to enable it, false to disable it.
This property is optional. Default is false.
Beware, performances are impacted if this property is enabled.
-->
<talend:validate-exchange-by-wsdl>false</talend:validate-exchange-by-wsdl>

</jbi:component>
</jbi:jbi>{code}

This component also has one specific configuration parameter.
|| Parameter || Description || Default || Required ||
| validate-exchange-by-wsdl | True to validate the received messages with respect to the WSDL of the target service. | false | false |

This parameter is also available in the configuration of the service-units.
\\
Setting it in the component enables it for all the service-units deployed on this component. It also overrides the service-unit configuration for this parameter.
\\
\\
Beware, for the moment, WSDL-based validation does not work with messages having attachments.



h1. Understanding the way messages are processed

This section deals with the way messages (or requests) are processed by the Petals-SE-Talend component.
As a user, it is important to understand the logic of the component to use it efficiently.

A request received by this component may have only one goal: execute the target Talend job.
The request processing is made up of the different steps involved between the message reception and the response.
There are five steps in the processing of a request.

h2. Validating the request

When a request is received and started to be processed in the Petals-SE-Talend component, it is validated before being really processed.

Here are the different steps involved in this validation process.

The first step is the WSDL-based validation of the request's XML payload.
If the *validate-exchange-by-wsdl* parameter is set to *true*, either in the component or in the service-unit, then the XML payload is validated against the WSDL of the service-unit.
If the validation fails, an exception is thrown (which becomes either fault or an error, depending on the Message Exchange Pattern). Otherwise, the validation goes on.

{warning}
Be careful, WSDL-based validation does not work when the input message contains attachments.
The Talend export for Petals does prevent that from happening.
Just remember it if you modify the jbi.xml by hand.
{warning}

\\
The WSDL-based validation checks three elements:
# The called operation is defined in the service's WSDL.
# The called operation is associated with the called Message Exchange Pattern (MEP).
# The XML payload is validated against the WSDL's XML schemas.

{warning}
Be careful, the current implementation of this feature makes disk access, thus reducing the performances.
{warning}

\\
This component supports *InOut*, *InOnly* and *RobustInOnly* patterns.


\\
The second and last step in the validation is a check about the singleton property of a job.
If a job is singleton, it means that only one instance of this job can be executed at once.

{info}


One typical example of a singleton job is a job which moves data from one database to another one.
It would make no sense for two instances of this job to run at the same time, especially if they work on the same databases.
{info}


If the job is singleton and already running, then an exception (fault or error) is raised.
Otherwise, a new job instance is created. If the job is singleton, then the running state of this job is set to true and locked until it is this state is released (i.e. the job is executed).

\\
{info}

The job creation strategy is a lazy strategy. A job instance is created on every received and validated message.
The consequence for singleton jobs is that all the messages sent to a singleton job while it is running will be rejected.
{info}


Once accepted, the request can now be parsed to prepare the job's input.


h2. Preparing the job's input

Once the request has been accepted, it is parsed to get the different possible parameters for the job.
The message input contains up to 4 parts, that are described in the serivce's WSDL.
# The first parameters are the context parameters, child elements of the *contexts* element from the input message. These parameters will be passed to the job in its main method.
# Then, the data flow to be passed to a tPetalsIOnput instance is retrieved from the request. This data flow may not be present.
# The third kind of parameters is the input attachments.
# Eventually, the component processes the native options to be passed to the job.



{info}
If a job does not support to be passed data flow (for a tPetalsInput), an entry is logged, but no fault is raised. The execution goes on normally.
{info}

\\
Input attachments must respect some constraints:

* Each input attachments is serialized as a temporary file.
* Its location will be passed to the job through a context variable. This is why attachments are associated with context variables.
* Be careful, attachments are expected to be passed in MTOM mode. That is to say the attachment element has a grand-child element "xop:include" whose href attribute references an attachment.
* Besides, the name of the attachment element is the name of the context variable that will be associated with the temporary file location.

{info}
As a user, you do not have to worry about this appearing complexity.
The configuration and the WSDL creation are made by the tools, during the export.
And hopefully, clients to call such a service can be generated automatically from the WSDL.{info}


\\
As you can see, from one JBI message (an XML payload and attachments), the Petals-SE-Talend component gets at most 4 kinds of parameter to pass to the jobs.
Three of them are merged together, since they are passed as contexts to the job. The remaining one concerns the tPetalsInput data.
Notice that the input message may not define any of these parameters. In this case, the component will pass nothing to the job.

{info}
In fact, the WSDL content and the expected parameters depend on the job's content and on the defined options during the export operation.
{info}


h2. Executing the job

At this point, the Petals-SE-Talend has built the job instance and prepared the parameters.
If the job contains a tPetalsInput component, the data for this component is passed to the job.
The Talend contexts and options are then passed to the job, right before its execution is launched.



h2. Getting the job's output

The native job's output is an array of array of String, that is to say: {code:lang=java}String[][]{code}
This result may contain only an integer, in which case {code:lang=java}String[ 0 ][ 0 ]{code} is an integer and indicates the result of the job execution. Otherwise, this array contains raw data (which is the case if the job contains a tBufferOutput).

Checking it can be a solution to determine whether the job execution succedded or not. The Petals-SE-Talend does not do it. It is the responsibility of the client to make this check (since in fact, it depends on the job itself).

\\
If the job contained a tPetalsOutput, then the output data flow is retreived from the job.

{info}
If a job does not support to be asked data flow (for a tPetalsOutput), an entry is logged, but no fault is raised. The execution goes on normally.
{info}

\\
Eventually, if it was specified during the job export that output attachments are to be expected after the job was executed, then they are taken back from the job.
These attachments must be passed from the job to the component through files. These files are loaded by the component in memory and then, deleted from the disk.
The deletion of these files is not an option. Letting them on the disk could represent important risks. Indeed, a malicious client could override the context on each call, thus creating an infinite number of files on the disk. Unfinite until the disk crashes, obviously.

Like input attachments, output attachments are returned in MTOM mode.
{info}
If a component expects output attachments to be returned by the job, and that this job does not support it, then an exception (fault or error)is thrown.
This can typically happen if you created your job with Talend Open Studio and exported a context as an "OUT-Attachment".
{info}


h2. Building the response

Now that everything has been gathered from the job, the response can be built and returned.
Hence, the response can count up to 3 parts:
# The job's result (remember, the array of array of String). This part is always returned.
# The output data beans, if the job contained a tPetalsOutput.
# The output attachments.

Like the input message, the structure of the output message is determined by the job content and the options which were checked during the export of the job for Petals.

h1. Monitoring the component

h2. Using metrics

Several probes providing metrics are included in the component, and are available through the JMX MBean '{{org.ow2.petals:type=custom,name=monitoring_*<component-id>*}}', where {{*<component-id>*}} is the unique JBI identifier of the component.

h3. Common metrics

{include:0 CDK Component Monitoring Metrics 5.6.0}

h3. Dedicated metrics

No dedicated metric is available.

h2. Receiving alerts

Several alerts are notified by the component through notification of the JMX MBean '{{org.ow2.petals:type=custom,name=monitoring_*<component-id>*}}', where {{*<component-id>*}} is the unique JBI identifier of the component.

{tip}To integrate these alerts with Nagios, see [petalsesbsnapshot:Receiving Petals ESB defects in Nagios].{tip}

No dedicated alert is available.